Integration of Sikuli with Selenium WebDriver

Sikuli is an image-based open source tool to automate the GUI and can be used on any platform like Windows/Linux/Mac/Mobile. Sikuli uses a technique called image recognition to identify and control GUI components. At times when to access a GUI object, there is no access to its internal or external code; Sikuli comes in handy. It simply automates anything you see on the screen.

Need of Sikuli Integration with Selenium

Although as easy as it is to use Sikuli, there is one overhead since there is no built-in method to open a website/web-application in Sikuli. So if you’re executing a Sikuli script, you need to make sure that the website is already open in a browser.
To overcome this, Sikuli can be paired with Selenium WebDriver. Selenium offers a method to open a website in a browser. Using that method will help us prevent the effort wasted every time to open the website manually in the browser. Once the application is open, we can easily run Sikuli scripts on that website.

How it’s done:

1. Crop the image of the object which could not be identified by Selenium and put it into data pool folder.

2. Pass the path of that image into selenium script.

3. At the run time, Sikuli searches the image on the screen and matches it to the image in the data pool. If found then it performs the desirable action on that object.

Sikuli Setup for Selenium

1. In order to integrate Sikuli with Selenium, we need some jars such as sikuli-script.jar, selenium-server-standalone-3.10.0.jar, selenium-java-3.10.0.jar
 (or you can download any version of jars)

2. Create a java project in eclipse and add the listed external jars to your project using Build Path.

3. Create a java class inside the project, say Sikuli_Test.java.

4. To work on images using Selenium, create an object of Screen/Pattern classes.

Sample Program

In the above program, we tried to open the homepage of ‘google.com’ followed by creating objects for the screen and pattern classes. In the pattern class, we have passed the reference of any image on which we have to perform any operation(like click/double click/type, etc.) While executing this program, Selenium searches the image on the screen which we have passed as reference and performs the desired action on that if it exists

Nowadays Selenium is widely used tool for automating web-application. In real life, products are enterprise application, so we can expect application would be an integration of Silverlight, Java Applets, Raphael Java Script, Canvas Objects, Java Scripts, JQuery, etc. However, we cannot expect everything gets identified with Selenium and can face challenges in automating the few of the above mentioned UI with Selenium. So this time you get stuck, add a flavor of Sikuli in your script.