Constrain search for visual data

Asked by insignis on 2010-01-25

Is there a feature, or will there be a feature, to constrain Sikuli to only look for visual data in certain region(s) of the screen?

For instance, if I am automating something in Firefox, would I be able to tell it "only look for this picture on Firefox windows"? Perhaps something to find windows with specific text in their titlebars, or constrain to windows launched by a given process?

I realize this increases the complexity, and you're going for a minimalistic approach, but the one thing I noticed about the demo on YouTube that I watched, is that it's fairly slow at finding the picture you asked it to Click/Wait/etc. I thought being able to do something like this might make it faster....

Question information

English Edit question
Sikuli Edit question
No assignee Edit question
Last query:
Last reply:
insignis (insignis) said : #1

... or is that what switchApp does? Regardless, I can't get switchApp to work on Windows (but I think that's been reported elsewhere). (aperti) said : #2

I have also had this thought. Specifying X,Y axis starting points on your screen instead of starting from the top left and working it's way across, down, across, down, etc.

Tom Yeh (doubleshow) said : #3

Currently, Sikuli supports region-based search based on absolute positions on the screen. You can define a region (the third toolbar icon from the left) by drawing a box on the screen. In you case, you simply draw the box around the Firefox window. Then, an icon will represent the region you have selected. You can add .inside(find(IMAGE)) to look for IMAGE within the region instead of the entire desktop.

Below is a screenshot showing the piece of code that will do the region-based operation as described above.

Project Sikuli

RaiMan (raimund-hocke) said : #4

This should only be relevant if you have a big screen and relatively small windows. But if you want, there are some possibilities, to get near to what you want.

Since everything starts with images that should be somewhere on the screen, its good to identify some invariables for each relevant app, and save them with a readable filename somewhere else. So you can use them again by using "insert an image". These images are duplicated into your present Sikuli.

I guess, Sikuli works as fast as possible, if you at least have the background of the screen only in one colour (no picture or even animations). If you want to concentrate on one app, just hide all windows from all other applications (there are shortcuts for that - use type()).

Then look for the upper left corner of your app and save this match in a variable:
upperLeft = find(... image upper left ...)

upperLeft knows its position on the screen and you can get it:
cornerUpperLeftX = upperLeft.getX()
cornerUpperLeftY = upperLeft.getY()

same goes for the lower right corner (it definitely works, if there is only the relevant window visible)
lowerRight = find(... image lower right ...)
cornerLowerRightX = lowerRight.getX() + lowerRight.getW()
cornerLowerRightY = lowerRight.getY() + lowerRight.getH()

Here you have to add W(idth) and H(eight) since the images/matches have there origin in the upper left corner.

Now you can define your own region:
myRegion = Subregion( cornerUpperLeftX, cornerUpperLeftY, cornerLowerRightX - cornerUpperLeftX, cornerLowerRightY - cornerUpperLeftY)

This is what Sikuli IDE creates inside, if you grab a region.

From now on, every find looks like this (as mentioned by Tom above)
myMatch = myRegion.inside().find(... picture your looking for ...)

now you can test myMatch, you could click on it or do some further manipulation like above.

Since you now focus on your region, you can even restore all hidden windows.

Can you help with this problem?

Provide an answer of your own, or ask insignis for more information if necessary.

To post a message you must log in.