What is the correct way to click on drop-down menus that highlight?

Asked by Paul Chambre on 2018-09-19

In my test application, I have a drop-down menu whose entries highlight when the mouse pointer is over them. This seems to give Sikuli some difficulties. i.e. Sikuli can find the menu item, but the highlighting seems to sometimes make it lose it before it can click.

I think there may also be difficulties being caused by some Swing event timing issues.

This code works reliably, but I'd like to know if there is a more elegant way to do this that is as reliable.

        // Hover, then click, since hovering will cause the menu item to highlight, and that seemed to cause a race where it wouldn't always click
        appWindow.hover(OpenNotebook);
        appWindow.click(HighlightedOpenNotebook);
        Debug.action("Clicked OpenNotebook.");
        try {
            appWindow.wait(NotebookOpenDialog, 2);
        } catch (FindFailed reclick) {
            Debug.action("Initial Open Notebook click failed.");
            try {
                appWindow.wait(HighlightedOpenNotebook, 1);
                appWindow.click(HighlightedOpenNotebook);
            } catch (FindFailed ignored) {
                appWindow.click(MoreActions);
                // Hover, then click, since hovering will cause the menu item to highlight, and that seemed to cause a race where it wouldn't always click
                appWindow.hover(OpenNotebook);
                appWindow.click(HighlightedOpenNotebook);
            }
        }

Even with this hover on the unhighlighted menu item and clicking on the highlighted one, I see the initial click fail about 1 in 10 times, and fall through to the retry.

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2018-09-21
Last reply:
2018-09-21
RaiMan (raimund-hocke) said : #1

No need to do that:

appWindow.hover(OpenNotebook);
appWindow.click(HighlightedOpenNotebook);

Just click, where you are after the hover ;-)

so:
appWindow.hover(OpenNotebook);
appWindow.click(); # clicks the last match in region appWindow

or so:
appWindow.hover(OpenNotebook);
click(Mouse.at()) # click, where the mouse is

Paul Chambre (pchambredh) said : #2

I've tried a number of variants. Pretty sure that's one of them. I'll try it again, though, to be sure.

Paul Chambre (pchambredh) said : #3

Just tried it. It did not work. The mouse moved to the menu item, but the application did not respond to the click.

Paul Chambre (pchambredh) said : #4

With this code:

appWindow.hover(OpenNotebook);
appWindow.click();
Debug.action("Clicked OpenNotebook.");
try {
    appWindow.wait(NotebookOpenDialog, 2);
} catch (FindFailed reclick) {
    Debug.action("Initial Open Notebook click failed.");
    try {
        appWindow.wait(HighlightedOpenNotebook, 1);
        Debug.action("Clicking highlighted menu item");
        appWindow.click(HighlightedOpenNotebook);
    } catch (FindFailed ignored) {
        Debug.action("Re-clicking menu item.");
        appWindow.click(MoreActions);
        // Hover, then click, since hovering will cause the menu item to highlight, and that seemed to cause a race where it wouldn't always click
        appWindow.hover(OpenNotebook);
        appWindow.click();
    }
}

I get this in my log:

[log] CLICK on L(1633,600)@S(0)[0,0 1920x1080] (595 msec)
[log] Clicked OpenNotebook.
[log] Initial Open Notebook click failed.
[log] Clicking highlighted menu item
[log] CLICK on L(1640,600)@S(0)[0,0 1920x1080] (594 msec)

RaiMan (raimund-hocke) said : #5

The part, where you put my suggestion is not visited ("Re-clicking menu item." is not printed).

my suggestion is also for the visited part:

not:
        appWindow.wait(HighlightedOpenNotebook, 1);
        appWindow.click(HighlightedOpenNotebook);

but:
        appWindow.wait(HighlightedOpenNotebook, 1);
        appWindow.click();

The Debug.action() here does not make sense, since a click is logged anyways.

BTW: reducing the waittime below 3 seconds does not make sense: If the image is already found after one second, the the search terminates with success. If it is not found within 1 second (high risk) you get a FindFailed even if after 2 - 3 seconds the image would have been found.

If there is a risk, that an image search takes longer in some cases, just set the waittime to 5 or even 10 seconds, which would give some robustness for special situations, but does not harm if it succeeds in shorter times.

Paul Chambre (pchambredh) said : #6

Hi,

Your suggestion was:

"Just click, where you are after the hover ;-)

so:
appWindow.hover(OpenNotebook);
appWindow.click(); # clicks the last match in region appWindow

or so:
appWindow.hover(OpenNotebook);
click(Mouse.at()) # click, where the mouse is"

The very first two lines of my method are:
appWindow.hover(OpenNotebook);
appWindow.click();

So it certainly is visited.

RaiMan (raimund-hocke) said : #7

Initially you complained about some menu item that changes on hover.

So the problematic area seems to be here (which is what the log tells):

 catch (FindFailed reclick) {
    Debug.action("Initial Open Notebook click failed.");
    try {
// this is what I targeted in comment #5
        appWindow.wait(HighlightedOpenNotebook, 1);
        Debug.action("Clicking highlighted menu item");
        appWindow.click(HighlightedOpenNotebook);
    } catch (FindFailed ignored) {

Paul Chambre (pchambredh) said : #8

I'm getting a FindFailed while waiting on NotebookOpenDialog because the dialog is never opening. I'm watching the app. I can see that after "[log] Clicked OpenNotebook.", the app is sitting there with the mouse hovering over the Open Notebook menu item. It's not failing to find it because of the two second timeout.... it's not opening at all.

So the question is still, how better to ensure that the click takes.

Sounds like you're saying I need to call:

appWindow.hover(OpenNotebook);
appWindow.click();

Again. And again?

RaiMan (raimund-hocke) said : #9

Uuups, just went back in your answers and found, that you are on macOS --- sorry.

On Mac we have the bad situation, that a click on an app, that might be seen in the foreground does not have the focus.

So please take care, that the targeted application has the focus, before trying to click something.

This can be done by y a click somewhere in the app window, that does not do anything, but activate the window or by using the feature App.focus().

When I am doing my test on Mac, I always have this at the beginning:

App.focus("safari"); // as an example
RunTime.pause(1); // just to give the GUI some time

Paul Chambre (pchambredh) said : #10

This is not the first click event in the application. It first clicks to connect, and then, once connected, it's going to the menu... nothing has occurred outside that would cause it to lose focus between these events.

I'm still thinking it's some sort of Swing weirdness. My app is Swing based.

Oh well... I have written a robustClick method that includes the wait for the next expected item, and retries, and that seems to work reliably.

RaiMan (raimund-hocke) said : #11

Glad, that you found a solution.