SikuliX

Using a Screen other than the whole desktop

Asked by Dan

I switched over from trying sikuliFifirefoxDriver to using the sikuliX branch. I am able to call up a webpage in Selenium and click certain images on the page using sikuli. But sikuli is using the visible desktop as its screen. Is there a way to make the selenium browser view the screen and select and click images from there. I want to be able to run multiple selenium browsers at the same time, which may or may not be visible on the desktop at any given moment and be able click images on them independently. I want to:

- go to a webpage
- get a Screen using the selenium snapshot of the browser window
- wait for an image to appear
- click that image

Can you point me to some example code that will do step 2?

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
Dan
Solved:
Last query:
Last reply:
Revision history for this message
Roman Podolyan (podolyan-roman) said : 		#1

Sikuli can restrict image search to some Region, less than whole screen — http://doc.sikuli.org/region.html , that is you may search in several independent regions, so you may want to go into that topic.

But I don't get how you want to display multiple browsers on the screen and switch between them. Once I wrote scripts that switched between, say, XCode and SVN client, clicking on tasks icons in Mac OS X taskbar — that was working with different applications, only one of which may be present on screen in the any given moment (also Sikuli has some tools to bring application front and back http://doc.sikuli.org/globals.html#controlling-applications-and-their-windows , but I didn't use them at that time).

So. If you want to restrict search to some less than whole screen part, use Regions.
If you want to switch between several applications, you may write custom "switch by taskbar icons" part.

As for saving images for later use, there is Capture — http://doc.sikuli.org/screen.html#capturing
I never used it so far, but you can search for examples here on the Q & A forum.

Revision history for this message
Dan (mendels99) said : 		#2

I want to have several overlapping browsers running. When something changes in one, I want sikuli to bring it to the front and click some images in it. The problem is, if a browser is not visible on the desktop when it changes, sikuli will not sense the element being present.

A region seems to be just a subset of the desktop view. I want to use selenium to grab a screenshot of the browser view, regardless whether or not it is in front and visible on the desktop, and use sikuli to detect if an image is present in it. I can get the screenshot of the browser view in a BufferedImage format from selenium. Can I create a screen or a region using that that sikuli can then check?

Revision history for this message
RaiMan (raimund-hocke) said : 		#3

if you want to work with browser sessions, you should combine Sikuli with Selenium.

For 1.1.0 there is an easy to use option, to activate the Selenium API for the usage inside of Sikuli scripts.

see http://sikulix.com/support

Revision history for this message
Dan (mendels99) said : 		#4

I think I figured out a way to what I want. I can grab the screenshot and put it into an Image object. Then I can OCR it to figure out if the proper screen has come up. But I was trying to use and Image.find to find the image I want to click on the screen and gives me an error that it is not implemented?

[error] Image: find: not implemented yet