matching in the IDE is not the same than in an executed script

Asked by Carlos

I got into a problem that I am not able to solve in spite that I have been searching for a solution in the web and several combinations of tests.

I am using sikuli to test our developed application in Windows 8.1. I have developed several tests without any problem but, in one of the specific find operations I am getting different behaviours in the Matching preview window from the one gotten from running the script externally.

The operation is:
        click(find(imgSet.getImage("DeleteConfirmation")).find(imgSet.getImage("Yes")))

An the images are defined as:
img_01["Yes"]=Pattern("Yes.png").similar(0.90)
img_01["No"]=Pattern("No.png").similar(0.90)
imgSet = image_wrapper.imageDictWrapper(img_01)

Script file with:
 [ERROR] FindFailed ( can not find P(Yes.png) S: 0.9 in M[714,527 413x145]@S(0) )

While matching preview see it (see attached snapshot).

Any help will very much appreciated.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
Carlos
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

surely not a bug

Revision history for this message
RaiMan (raimund-hocke) said :
#2

Since Preview in IDE searches the whole screen, I guess the match you are searching indues not contain the image "Yes" completely.

BTW: since you already matched the whole dialog box, clicking yes or no should better be done with an offset from the matches center.

Revision history for this message
Carlos (cillana) said :
#3

Hi RaiMan,

Thanks for your fast answer to my issue.

Answering to your first paragraph, no, the 'Yes' image is completely visible when the match operation is done (I wanted to add another snapshot to show it to you but I do not see the opetion in the ticket.

Regarding your second paragraph, yes I know that there are the offset option but I would rather prefer to extract the 'yes' and 'no' buttons in order to 'reuse' it in another forms and not to modify the offsets in case the image change.

I overcome the situation just typing 'Key.ENTER' which is a good way (with no need of offsets nor images) to work with it, but still curious why I did not got this working.

Any guess?

Revision history for this message
RaiMan (raimund-hocke) said :
#4

Ok, 1st paragraph might not be clear enough.

Of course the image is fully visible (as Preview shows).

But your match of the dialog box restricts the search region by intention.
So you have to check, wether the yes image can be found WITHIN the rectangle of the dialog box match.
Be aware, that only one row of pixels at the bottom of the image and the search region might make the difference.

Revision history for this message
Carlos (cillana) said :
#5

The 'Yes' image was captured from the 'DeleteConfirmation' dialog and there should not be any 'missing' row of pixels in the search region.
Additionally, avoiding that restrinction and directly looking for the 'Yes' image
        click(find(imgSet.getImage("Yes"))
, I get the same error:
[ERROR] FindFailed ( can not find P(Yes.png) S: 0.9 in R[0,0 1920x1200]@S(0) )
Exception in thread "MainThread" java.lang.IllegalStateException: No match found
 at java.util.regex.Matcher.group(Unknown Source)
 at org.sikuli.scriptrunner.JythonScriptRunner.findErrorSourceWalkTrace(JythonScriptRunner.java:346)
 at org.sikuli.scriptrunner.JythonScriptRunner.findErrorSource(JythonScriptRunner.java:311)
 at org.sikuli.scriptrunner.JythonScriptRunner.runPython(JythonScriptRunner.java:216)
 at org.sikuli.scriptrunner.JythonScriptRunner.runScript(JythonScriptRunner.java:158)
 at org.sikuli.basics.SikuliScript.main(SikuliScript.java:199)
 at org.sikuli.ide.SikuliIDE.main(SikuliIDE.java:245)

Revision history for this message
RaiMan (raimund-hocke) said :
#6

ok, then the next reason might be, that the yes-button cannot be found with a minimum similarity of 0.9 (can it be deactivated/greyed/shaded) in the situation of the failing find op.

If you like, you might send me a screenshot at the time of crash and the yes-image for inspection.
Use my mail at https://launchpad.net/~raimund-hocke

Revision history for this message
Carlos (cillana) said :
#7

In the first snapshot I attached (I do not know if can still see it after changing the ticket from bug to question) you can see both application control (at the time of the crash) and the matching preview.

The control does not get deactivated/greyed/shaded in any moment. The 'yes-control' is 'active' with an internal 'glow' but does'nt disappear after the application is crashed. Additionally, this dialog is modal so it is impossible to get overlapped nor loose the focus.

Finally, I tested (in the matching preview) that with 0.9 similarity it matches the 'yes' control even if it is active (with the blue internal glow) and if it is not.

Revision history for this message
RaiMan (raimund-hocke) said :
#8

Again:
- what Preview shows in that moment might be helpful but does not really count
- relevant only is the screen situation at the moment of the crashing operation

to debug, you should break it up as:
dbox = find(imgSet.getImage("DeleteConfirmation"))
dbox.highlight(5) # to have some time to watch
dbox.find(imgSet.getImage("Yes")))

Revision history for this message
Carlos (cillana) said :
#9

I did what you suggested and visually checked that 'yes' image was inside the 'DeleteConfirmation' dialog. I did that capturing the image (with the red square) and overlaying the 'yes' image over the yes button and confirmed that the yes image did not overlap the red square.

Nevertheless, the 'yes' image have some border around the button. I recaptured it adjusting it to the button and then it did work.

Considering that the first 'yes' image did not overlap with the red square, is the capture region narrower than the 'dialog region'? that's it, does it need some margin?

Revision history for this message
Carlos (cillana) said :
#10

After several e-mails exchanged with RaiMan the issue here is that I was using captured images which did not provide good matching with the screen ones. After edition of these images the script worked as expected.

The recommendations from RaiMan are:

An optimal screenshot contains as little (even) background as possible and concentrates on the distinguishing visual elements (see example in script).
The goal should be to get similarity scores beyond 0.95.
In my own scripts I usually get scores of 1.0.
Rule of thumb: The amount of surrounding background pixels should be <20% of all pixels in the image.
Interesting: in your case the No-button is matched with a slightly higher score than the Yes-button.

Thanks a lot for the help!!

Revision history for this message
RaiMan (raimund-hocke) said :
#11

thanks for documenting this here. all the best.