SikuliX

Problem with mouse clicks on images

Asked by Anders W

Hi,
I have problem with mouse clicks on found images.
After verifying that a image exists, sometimes the mouse pointer is not moved to the correct area of the region before the click.

Example:
if region.exists(button):
    region.click()

This sometimes result in a click to the left of the button

SikuliX version 2.0.4
Java version 1.8.0_271

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said : 		#1

Is the button visible before and moving to its place as a result of some action before (some animation)?

try with a short but appropriate wait before:

wait(0.3)
if region.exists(button):
    region.click()

Revision history for this message
Anders W (avanceanders) said : 		#2

I'm using SikuliX to test an web application so the action just before is switch to new web page where this button should appear an be clicked on.

I will test this wait trick.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#3

Another possible reason for you problem:

region.click() does a mouse click in the center of the region. In case that your button is outside the center of the region, your click may be outside the button.

Try

if region.exists(button):
    region.getLastMatch().click()

Revision history for this message
Anders W (avanceanders) said : 		#4

As I understand it the region.click() will use the last match by default when not specifying any match/image?

Revision history for this message
RaiMan (raimund-hocke) said : 		#5

@Manfred: Anders is right: if region has a lastMatch, it is used until overwritten otherwise the center of region is used. So his approach is absolutely correct.

@Anders: switching to the web page might indeed have such an effect of content "flying in" somehow.

Revision history for this message
Anders W (avanceanders) said : 		#6

@RaiMan: this wait trick seems to work better. But since this is a web application a web page may sometimes take longer time to load. Wouldn't example below be a better solution?

if region.exists(button): # wait for page to load and button to appear
    wait(0.3) # wait for "animation effect" to finish
    region.click(button) # search for button again since it may have been moved and then click

Revision history for this message
Best RaiMan (raimund-hocke) said : 		#7

The challenge apparently is, to wait until the button is in its final place.

This indeed might take a varying amount of time, depending on the time it takes until all (dynamic/scripted) elements of the page have finished their setup and are in its final place/shape.

So the exists() has to be made more robust, to detect, when the button is in its final place.

Something like this:

while True:
    finalButton = region.exists(button, 0)
    wait(0.3)
    if finalbutton and finalButton.exists(button, 0): break
finalbutton.click()

This checks, wether the button is still in its place, after found by region.exists(button, 0)

I will put this on the request list: option to wait until a found pattern is no longer moving (sthg. like Pattern(img).isMoving())

Revision history for this message
Anders W (avanceanders) said : 		#8

Thanks RaiMan, that solved my question.