How to use Sikuli in multi monitor environments?

Asked by Olof Bjarnason

OS: Win7 64-bit
Sikuli: x1.0 rc3

I ran a sikuli script which failed with "org.sikuli.FindFailed", and it is quite obvious to me why: I have dual screens (extended desktop). The script worked yesterday, when the Chrome-window it opens happened to open on the same desktop/screen on which I double-clicked the .skl file.

Are there any plans on adding support for extended desktop / multi screens in Sikuli? It is kind of a show stopper for using Sikuli as an end-to-end testing tool here at work (all work stations have dual screens).

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
Olof Bjarnason (objarni) said :
#1

Note: where the .skl file is located (on which screen) does not matter. What does matter, is where the Chrome window opens, and that seems to be guided by where I closed Chrome last time. Sikuli seem to be doing image recognition only on the "primary screen" of the O/S, not the secondary.

Revision history for this message
Best RaiMan (raimund-hocke) said :
#2

Sikuli supports multi screen environments
(see docs: http://sikuli.org/docx/screen.html#multi-monitor-environments)

But currently you have to tell Sikuli, on which of the screens it should search.

If you do not specify a search region (e.g. find(some-image)) the primary screen will be searched.

So, if you know, something happens on Screen(1), you have to specify:
(best is to assign Screen(1) once: SCR1 = Screen(1))
SCR1.find(some-image-on-screen1)

If you have an app (like chrome), you might use the App class functions, to find out, where your app window is located:

aChrome = App("Chrome") # to be adapted for Windows
aChrome.focus()
SCR1 = aChrome.window().getScreen()

or you decide to work directly with the app window:

aChrome = App("Chrome") # to be adapted for Windows
aChrome.focus()
aChromeWin = aChrome.window()
with aChromeWin:
    find(some-image-in-chrome-window)
    # more actions inside Chrome window

Revision history for this message
Olof Bjarnason (objarni) said :
#3

Thanks RaiMan!! That's great news!

I love Sikuli project, but it still a little "rough at the edges", at least in a Windows environment where I have played with it. Some examples of roughness:

1. the installer does not associate the sikuli-icon to .skl files (had to do it manually which was a bit of googling/hacking). i think this might also be related to the .skl files being run by a .bat-file, which makes it harder (or rules out?) correct icon association
2. type() does not work with backslash (and some other ubiqous characters) on a non-american keyboard layout (Swedish in my case), I use paste() as a workaround
3. openApp and friends kind of require a full-path to work (I think you stated this, kind-of, above in your comment # to be adapted for Windows)

I'd really like to help out, but the build-procedure is so complicated my eyes cross ;) Instead, I'll keep asking questions & filing bugs.

Revision history for this message
Olof Bjarnason (objarni) said :
#4

prevComment.replace('ubiqous', 'ubiquitous')

Revision history for this message
RaiMan (raimund-hocke) said :
#5

--- build-procedure
agreed, that is not something you should do normally. You only have to do that, if you really want to add/improve/test something on the native C++ level.
Any other ideas you have may simply be added by using the standard Python/Jython or Java import features. Even the possibility to pack a complete Jython/Java solution into an extension is rather easy.

--- 1. installer - .skl
In my opinion, the .skl is something that should no longer be used by anyone using Sikuli X.
It does not add any advantages but only oddities.
The only thing of value might be, that you might send a .sikuli as one file over the net. But since a .skl is only a zipped version of the corresponding .sikuli, you might zip it yourself in these cases and unzip on the far end.
BTW: biggest disadvantage: .skl cannot be imported or added to the image path.
Recommendation: just forget .skl

--- 2. type()
known issue, see faq 933

--- 3. openApp() - use App class instead
but yes, on windows, it is a bit more complicated, since you have to distinguish between apps to be started and running apps (identified by their windows title).
Apps found on system path can be opened without using absolute path (which in fact is not the case for Firefox, Chrome and many other apps on Windows).

because I have to switch the machine, see next comment for an example.

Revision history for this message
RaiMan (raimund-hocke) said :
#6

This works, if Chrome is not started yet:

ac = App(r"C:\Users\Raimund Hocke\AppData\Local\Google\Chrome\Application\chrome.exe")
ac.focus()
while not ac.window():
    wait(1)
win = None
for i in range(100):
    w = ac.window(i)
    if not w:
        break
    if w.h<300:
        continue
    win = w
    break
win.highlight(2)

Chrome places more than 10 hidden windows on the desktop at startup. the above code filters these windows (Sikuli currently cannot distinguish between hidden and visible windows) by some criteria (in this case the window height). The first one not filtered seems to be the app window.
The region of this window is now in win.

Revision history for this message
Olof Bjarnason (objarni) said :
#7

Thanks for all the info RaiMan! I think some of the comments you make, like that about .skl files (which I'm going to forget about from now on) should be added to the GUI for Sikuli A.S.A.P. to stop people using it. To begin with, it could be left there, but with an added [Deprecated] text in the Menu of Sikuli, or maybe an explamation/warning icon in the File menu.

I will try out your algorithm for finding the Chrome window later today.

About finding applications not-on-path in Windows, one idea might be to scan the Start-menu for the names there. For example, Chrome is title "Google Chrome" which I think is an OK abbreviation of the full path. And you're right - it is not so common for applications to modify the system PATH environment variable (adding their own path basically), so some kind of hack is needed to make openApp and friends more comfy under Windows.

If I can ask you one more question while we're at it: why does Sikuli API-calls use camelCase and not the ordinary PEP8 style? It just feels awqwuard, as a Python hobbyist, to write switchApp instead of switch_app. :O)

Revision history for this message
Olof Bjarnason (objarni) said :
#8

prevComment.replace('awqwuard', 'awkward') # how typical to misspel something when commenting on spelling

Revision history for this message
RaiMan (raimund-hocke) said :
#9

--- PEP8 vs. camelCase
Did never think about that before, since I have been a camelCase fan all the time during my IT-life (BTW: I am not one of the developers ;-).

on the other hand, the naming is not very consequent throughout the API. But since in the background some restructuring takes place, your idea might be getting reality in the future.

Revision history for this message
RaiMan (raimund-hocke) said :
#10
Revision history for this message
Olof Bjarnason (objarni) said :
#11

I've signed up for the mailing list, hope to be a contributor in one way or the other.

Many thanks for your urgent and thorough support RaiMan!

Revision history for this message
RaiMan (raimund-hocke) said :
#12

Always welcome :-)