SikuliX

Having trouble with inconsistent drop-down behavior on Windows

Asked by Paul Chambre

I am trying to automate an action where I open a drop-down menu and then click on one of the items from the menu. This is a Java application using Swing, on Java 8, and Windows 10.

Opening the menu works fine, but recognizing the item on the menu seems to change from run to run. I don't know if Windows is doing some font scaling or smoothing, but the png and what I see on the screen look pretty close, to my eye, but apparently not to Sikuli.

I've tried both click and a hover + click.

I wanted to try OCR instead, to see if that was any more resilient to whatever else might be changing between runs, but I am having trouble finding examples of that.

I have added:
Settings.OcrTextRead = true;
Settings.OcrTextSearch = true;

but when I try:
Region menuItem = appWindow.exists("Open Notebook", 0);

I get:
[error] Region: handleImageMissing: Response.ABORT: aborting
[error] Region: Did you want to find text? If yes, use text methods (see docs).
Exception in thread "main" java.lang.RuntimeException: Exists: Abort:ImageMissing: Open Notebook
 at org.sikuli.script.RunTime.abortScripting(RunTime.java:1107)
 at org.sikuli.script.Region.exists(Region.java:2372)
 at com.illumon.guitest.TestIrisGui.main(TestIrisGui.java:139)

Thanks for the help,
Paul

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
Paul Chambre
Solved:
Last query:
Last reply:
Revision history for this message
masuo (masuo-ohara) said : 		#1

>but the png and what I see on the screen look pretty close, to my eye, but apparently not to Sikuli.

Is your issue FindFailed error?
 I recommend to use SikuliX IDE to check match objects are there or not on the screen.
 Capture image, and click the caputured image, then pattern setting window open.
 Select ”matching preview” tab, then you can check match objects on the screen.

Revision history for this message
Paul Chambre (pchambredh) said : 		#2

That actually brings up another question:

How can I try matching preview when switching to the Sikuli IDE will cause the drop-down menu in the application to close (when it loses focus)?

For the original question: I'd still like to know how to enable text matching.

Revision history for this message
Paul Chambre (pchambredh) said : 		#3

By the way, yes, I am getting find failed, even with similarity as low as .6, but only for the drop-down menu item. The button that opens the drop down is matching to >.99.

I really think there's something wrong with how drop-down menus are being "seen" by Sikuli... maybe specific to Windows, or VMs, as this works much better (though still a bit flaky) on OSX.

Currently I can work around this by creating a Location based on an offset from the coordinates of the found button that opens the drop-down, and then clicking that, but that seems pretty kludgey and less resilient to GUI changes.

Also, reading around a bit more, it sounds like things like .click("some_text") don't work in Sikuli's Java API unless you do your own work to integrate Tesseract somehow into the core methods.

Revision history for this message
Paul Chambre (pchambredh) said : 		#4

Solved.

The issue was that I have a screen snip of the lower left corner of my app so I can find the extents of the application window when the app is not maximized. This snip was matching multiple areas at the default of .7. This would often result in only a partial extent for the app window region. By raising the similarity to .85, I get the one correct hit on the corner of the app window, and then my drop down menu item is within the region.

Thanks for the suggestion on the Match Preview view of the IDE for visualizing what is or is not matching.

Revision history for this message
masuo (masuo-ohara) said : 		#5

answer for #2

Open drop-down menu, then capture screen(push print screen button if your PC is Windows).
Paste to image editor application, then use SikuliX IDE matching preview feature.