Problem with mouse clicks on images

Asked by Anders W on 2020-10-29

Hi,
I have problem with mouse clicks on found images.
After verifying that a image exists, sometimes the mouse pointer is not moved to the correct area of the region before the click.

Example:
if region.exists(button):
    region.click()

This sometimes result in a click to the left of the button

SikuliX version 2.0.4
Java version 1.8.0_271

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
2020-11-02
Last query:
2020-11-02
Last reply:
2020-10-30

This question was reopened

RaiMan (raimund-hocke) said : #1

Is the button visible before and moving to its place as a result of some action before (some animation)?

try with a short but appropriate wait before:

wait(0.3)
if region.exists(button):
    region.click()

Anders W (avanceanders) said : #2

I'm using SikuliX to test an web application so the action just before is switch to new web page where this button should appear an be clicked on.

I will test this wait trick.

Manfred Hampl (m-hampl) said : #3

Another possible reason for you problem:

region.click() does a mouse click in the center of the region. In case that your button is outside the center of the region, your click may be outside the button.

Try

if region.exists(button):
    region.getLastMatch().click()

Anders W (avanceanders) said : #4

As I understand it the region.click() will use the last match by default when not specifying any match/image?

RaiMan (raimund-hocke) said : #5

@Manfred: Anders is right: if region has a lastMatch, it is used until overwritten otherwise the center of region is used. So his approach is absolutely correct.

@Anders: switching to the web page might indeed have such an effect of content "flying in" somehow.

Anders W (avanceanders) said : #6

@RaiMan: this wait trick seems to work better. But since this is a web application a web page may sometimes take longer time to load. Wouldn't example below be a better solution?

if region.exists(button): # wait for page to load and button to appear
    wait(0.3) # wait for "animation effect" to finish
    region.click(button) # search for button again since it may have been moved and then click

Best RaiMan (raimund-hocke) said : #7

The challenge apparently is, to wait until the button is in its final place.

This indeed might take a varying amount of time, depending on the time it takes until all (dynamic/scripted) elements of the page have finished their setup and are in its final place/shape.

So the exists() has to be made more robust, to detect, when the button is in its final place.

Something like this:

while True:
    finalButton = region.exists(button, 0)
    wait(0.3)
    if finalbutton and finalButton.exists(button, 0): break
finalbutton.click()

This checks, wether the button is still in its place, after found by region.exists(button, 0)

I will put this on the request list: option to wait until a found pattern is no longer moving (sthg. like Pattern(img).isMoving())

Anders W (avanceanders) said : #8

Thanks RaiMan, that solved my question.