Need to hover before click

Asked by fani

Hello,

In reality, I click on an image, the screen turns white, so I cannot find the next image to click on.

My plan is, to simulate it and to get a popup window with 'default' as soon as sikuli cannot find the second image:

try:
click(image 1)
click(image 2)

exept:
popup('default')

But in sikuli, however, image 2 can be found even the screen is white. If I change the code into:

try:
click(image 1)
hover(image 2)
click(image 2)

exept:
popup('default')

The popup window shows up, so it cannot be clicked on image 2

My question: How can this happen? Does Sikuli always save the postion of the following image? I just want to understand how Sikuli is working.

Thank you!

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
fani
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

this is a timing problem I guess:

after having clicked image1 you have to wait a short time, since Sikuli at the start of a search, which might last up to 1.5 seconds on the whole screen, takes a screenshot internally, that is searched for the image. So in your case this happens directly after the click(image 1) and at this time, the screen is not yet white and hence image 2 is still visible.

adding the hover(image 2) in the second case, inserts such a "wait", so that the following click fails, since it now searches image2 again on the white screen.

So I suggest the following (I hate try/except ;-)

click(image1)
wait(1)
if exists(image2, 0): # only one search, come back immediately, do not wait 3 seconds
    click(getLastMatch()) # this should not happen
else:
    popup("default")

Revision history for this message
RaiMan (raimund-hocke) said :
#2

*** since you asked, to understand how Sikuli works, here a text I wrote this morning:

some comments on how Sikuli image search works (to understand this is vital for more complex solutions):

- in the standard, Sikuli waits 3 seconds for an image to come up (this can be adjusted either globally or in functions like wait(img, time) or exists(img, time) )
- during this waiting time, Sikuli repeatedly searches for that image (one search means taking a screenshot of the specified region and searching the image in that screenshot)
- how often Sikuli searches can be adjusted as well globally

and you should know something about search timing:
- on a whole screen (e.g. exists(image)) one search takes between 0.5 and 1.5 seconds depending on screen size
So searching on the whole screen means constantly searching, since the standard search frequency is 3 per second, which only leads to not needing 100% cpu for Sikuli, that a search must last significantly less than 0.3 seconds.
- to get minimal search times, you have to restrict the search region as much as possible (e.g. an area of about 300x300 gives you a chance to have search times around 0.1 seconds)
Of course search time depends massively on your cpu power, since it is pure number crunching and bit processing of very large arrays.

So back to the challenge:
- generally use exists(image,0) with a restricted region, to have only one search trial and get minimal search times
this is how you use exists() already in your script
- insert needed pauses to not heat up cpu as needed with wait(seconds), where seconds
- the solution is very different, depending on wether all images, that need to be checked, are there BEFORE the first search is started or wether they might popup at different times and in different sequences during your search.

So to start with the easier one let's suppose, all images are on the screen already:

while exists (img1, 0):
    m2 = reg2.exists (img2, 0)
    m3 = reg3.exists (img3, 0)
    m4 = reg4.exists (img4, 0)
    # now we know all 3 matches, which gives us the needed flexibility
    # example:
    # we filter the valid combinations
    if m2 and not m4: click(m2) # prio 1: click if img2 is there but not img4
    elif m2 and m3: click(m3) # prio 2
    elif m3 and m4: click(m4) # prio 3
    # here we come if combination is valid
    else: continue # search again

# or another possibility:
    if m2:
        if m3: click(m3) # prio 1
        elif m4: click(m4) #prio 2
        else: continue # neither m3 nor m4 is not valid, so search again.

the same approach can be taken in the case of "appearing at different times/different sequence", but then you might first have to wait with a while loop until the first valid combination is there and then even check again before taking any actions

Revision history for this message
fani (sagerer) said :
#3

I also tried to insert a wait() even for 10 seconds. Sikuli still found image 2. Different to reality the screen turns white as soon as the mouse is moving and not as soon as you click on the image 1 ...

I found a way how to solve the problem, I am just interested in the "Why?" But never mind - thanks!

Revision history for this message
RaiMan (raimund-hocke) said :
#4

is it some transparent white, so the image is still somehow "visible"?

then
click(Pattern(image 2).similar(0.99))

should fail