Can't get Multimonitor Environment to work

Asked by Gromadusi

Hi,

i started playing around with sikuli appr. half a year ago, and picked it up again with 0.10. I'm confrontend with the following issue:

running the built in windows screen identifier, i get [1][2] which presumably should correspond to the sikuli mapping [0][1].

however, running 'selectRegion()' and analysing the resulting region objects, i found out, that sikuli maps the screens as [1][0].

so, based on that, i spent several hours to create stable workarounds, including recalculating x coordinates of ROIs (basically substracting the width of my second/right monitor, to no avail so far. Another workaround was to dragdrop windows containing the current ROI to Screen(0) and start from there, but again, it wasn't working as expected.

in all cases, every new find() results in a FindFailed exception, and also if i use setROI, it does not work reliably.

checking mouse location on either screen always shows me coordinates based on [0,0] on both screens. reading the docu i had the idea, that on one screen it should be [-x,0]?

the documentation section about multimonitor environments (http://sikuli.org/trac/wiki/reference-0.10#MultiMonitorEnvironments) does not reveal how to do it right...or I'm missunderstanding some important piece here :)

how can getNumberScreens(), getBounds() and scr.getBounds() help me to reach my goal?

can somebody provide me with a working example?

or is multiMon evn coverage still work in progress?

thank you so much in advance for sharing your knowledge!

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
Gromadusi
Solved:
Last query:
Last reply:
Revision history for this message
Gromadusi (gromadusi+fwd) said :
#1

my environment is:

Windows 7 64bit
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) Client VM (build 17.0-b16, mixed mode, sharing)

Revision history for this message
RaiMan (raimund-hocke) said :
#2

--- the pixel coordinates:
cite from doc - complete guide - multi monitor section:
The base is the coordinate system (picture above), that positions the primary monitor with its upper left corner at (0,0) extending the x-direction to the right and the y-direction towards the lower boundary of the screen. The position of additional monitors can be configured in the operating system to be on either side of the primary monitor, with different positions and sizes. So monitors left of the primary will have pixels with negative x-values and monitors above will have negative y-values (left and above both x and y are negative).

--- the Screen objects:
at startup of the IDE or when starting an .skl, the monitor status is evaluated and saved in the object Screen (a readonly object!).
try the following:

for i in range(0, getnumberScreens()):
   print Screen(i)
   print Screen(i).getBounds() # you get the coordinates

this should show the relevant information, to identify your screens in your configuration based on the pixel mapping.
normally the so called primary monitor is Screen(0).

--- finding and acting on screens:
a plain find(sthg) searches on Screen(0). if you want to look on other screens, you have to name them:
scr1 = Screen(1) # once assigned, since every Screen(n) creates a new instance
scr1.find(sthg)

or use the with sc1: block.

This means in the current version you have to know on which screen to search or check on every available screen (no region that puts together all screens as one region).

--- When you are through, just give me a hint, where I can improve the doc.

Revision history for this message
Gromadusi (gromadusi+fwd) said :
#3

Hi RaiMan,

Thanks a lot for your answer!

running your script, i get the following log output:

Screen(0)[1280,0 1280x1024] E:Y, T:3.0
java.awt.Rectangle[x=1280,y=0,width=1280,height=1024]
Screen(1)[0,0 1280x1024] E:Y, T:3.0
java.awt.Rectangle[x=0,y=0,width=1280,height=1024]

this reveals my assumption that sikuli somehow defines my secondary monitor on my right ([2] in windows) as default screen(0) and vice versa, my primary monitor([1] in windows) as secondary screen(1)
i could indeed wrap my mind around this (although it causes headaches :) ), but a small extension to your script leads me to my next issue of mouse actions (mouseMove())
it appears that Env.getMouseLocation is not aware of multiple monitors.

the script (with right mouse clicks and a wait to make the current position more visible) looks like:

for i in range(0, getNumberScreens()):
        print Screen(i)
        print Screen(i).getBounds() # you get the coordinates
        mouseMove(Location(Screen(i).x,Screen(i).y))
        wait(2)
        mouseDown(Button.RIGHT)
        mouseUp(Button.RIGHT)
        print "current Mouse Location is: ",Env.getMouseLocation()

and produces:

Screen(0)[1280,0 1280x1024] E:Y, T:3.0
java.awt.Rectangle[x=1280,y=0,width=1280,height=1024]
current Mouse Location is: (2559,0)
Screen(1)[0,0 1280x1024] E:Y, T:3.0
java.awt.Rectangle[x=0,y=0,width=1280,height=1024]
current Mouse Location is: (1280,0)

so, the pointer moves to the wrong position. instead of top left corners, it ends up in the top right corners. again, i'm afraid i'm still not getting the concept of screens and coordinates :)

your question on how you could improve the doc: i'm a 'learn-by-seeing-examples'-guy, and the sikuli reference, apart from being very(!) concise and explanatory (kudos for that, i'm aware that your involvement in sikuli is on a pure interest basis) , is lacking code examples.

Revision history for this message
Gromadusi (gromadusi+fwd) said :
#4

please allow me to narrow my comment about lacking examples:
of course not the whole documentation misses examples, but the sections i'm currently stuck at!

Revision history for this message
Gromadusi (gromadusi+fwd) said :
#5

am i wrong assuming that my current issue is the same as:
https://answers.launchpad.net/sikuli/+question/113326

if i'm wrong: sorry for the duplicate post

Revision history for this message
RaiMan (raimund-hocke) said :
#6

I just tested with my Win7:
I commit, that Sikuli gets something wrong internally, if Screen(0) does not have its upper left corner at (0,0), (this at least applies to getMouseLocation(), hover() for example works correct, but click() does not)

You are right (sorry, I forgot this case):
https://answers.launchpad.net/sikuli/+question/113326
was reporting the same issues.

I will test on my Mac later (its a bit easier ;-) and report a bug.

For now:
arrange your configuration, so that Screen(0) has (0,0) as its upper right corner and things should work as expected.

I will put my test case as example into the doc.

Revision history for this message
Gromadusi (gromadusi+fwd) said :
#7

thanks for confirming, RaiMan. and no need to be sorry, you were of great help.
my quick hack (although i dislike it): turn off second monitor while sikuli'ing and waiting for a fix