Pattern Matching precision problem

Asked by GZ on 2018-02-25

There seems to be a problem between what is shown as a match in the Pattern Settings / Matching Preview and what is actually a match when using find or hover operations.

Previously this used to be exactly the same.
I've never seen this problem previously, this is new!!!
I have tried this is both 1.1.1 stable and the latest 1.1.2 nightly with Java 1.8.0_161-b12 and it is the same problem.

*** More information in the comments as I've investigated further ***

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
GZ
Solved:
2018-02-27
Last query:
2018-02-27
Last reply:
2018-02-25
GZ (george-zador) said : #1

And here is an other problem, Pattern Matching is not the same for the hover operation and for the findAll operation.

For the exact same pattern on the exact same image these are the minimum matching settings that work:

hover(Pattern("icon_12.png").similar(0.88))

for y in findAll(Pattern("icon_12.png").similar(0.78)):
    hover(y)

I think no matter what the operation is the pattern matching settings should produce the same result?!!

GZ (george-zador) said : #2

Yes I can 100% reproduce that for hover and for findAll there is a very different minimum matching threshold, meaning the same number is resulting in a different matching precision!!!
And the difference in between the precision of these two operations gets bigger with a more difficult image.

And this makes scripts like this impossible, because when having many images the matching threshold is globally set, and exists() will think it has found the image (because it is interpreting a lower precision) but findAll() will not be able to find it because the same threshold is giving it a different matching precision.

    for x in stuff:
        if region1.exists(x,0):
            for y in region1.findAll(x):
                region1.hover(y)

*** THIS USED TO WORK AROUND DECEMBER SO SOMETHING EITHER IN SIKULI OR IN JAVA HAS CHANGED ***
I was on 1.1.1 in december but I've re-downloaded now and done a new clean install so if the 1.1.1 installer has changed then something in there has changed which is breaking this!!!

RaiMan (raimund-hocke) said : #4

What system? what Java?

*** THIS USED TO WORK AROUND DECEMBER SO SOMETHING EITHER IN SIKULI OR IN JAVA HAS CHANGED ***
I just checked: I did not change anything, that could have influenced the image search implementation.

I made a test, but could not reproduce the problem you report, this is my script:

icons = ["icon.png"]
icons.append(Pattern(icon).similar(0.94))
for icon in icons:
  print "\n**** icon:", icon
  match = find(icon)
  print "find:", match.getScore(), match
  matches = findAll(icon)
  while matches.hasNext():
    match = matches.next()
    print "findAll:", match.getScore(), match

... and this is the output:

**** icon: icon.png
find: 0.94692838192 M[288,571 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:296,579 [299 msec]
findAll: 0.946929812431 M[288,571 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:296,579 [-1 msec]
findAll: 0.94692838192 M[67,514 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:75,522 [-1 msec]
findAll: 0.945029377937 M[876,307 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:884,315 [-1 msec]
findAll: 0.910441339016 M[74,571 16x16]@S(S(0)[0,0 1280x800]) S:0,91 C:82,579 [-1 msec]
findAll: 0.910438299179 M[252,514 16x16]@S(S(0)[0,0 1280x800]) S:0,91 C:260,522 [-1 msec]
findAll: 0.910372138023 M[876,286 16x16]@S(S(0)[0,0 1280x800]) S:0,91 C:884,294 [-1 msec]
findAll: 0.849138140678 M[471,582 16x16]@S(S(0)[0,0 1280x800]) S:0,85 C:479,590 [-1 msec]
findAll: 0.849138140678 M[433,524 16x16]@S(S(0)[0,0 1280x800]) S:0,85 C:441,532 [-1 msec]
findAll: 0.798312544823 M[1076,467 16x16]@S(S(0)[0,0 1280x800]) S:0,80 C:1084,475 [-1 msec]
findAll: 0.797399103642 M[876,383 16x16]@S(S(0)[0,0 1280x800]) S:0,80 C:884,391 [-1 msec]
findAll: 0.79739767313 M[876,409 16x16]@S(S(0)[0,0 1280x800]) S:0,80 C:884,417 [-1 msec]
findAll: 0.719521164894 M[20,2 16x16]@S(S(0)[0,0 1280x800]) S:0,72 C:28,10 [-1 msec]

**** icon: P(icon.png) S: 0.94
find: 0.94692838192 M[288,571 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:296,579 [249 msec]
findAll: 0.946929812431 M[288,571 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:296,579 [-1 msec]
findAll: 0.94692838192 M[67,514 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:75,522 [-1 msec]
findAll: 0.945029377937 M[876,307 16x16]@S(S(0)[0,0 1280x800]) S:0,95 C:884,315 [-1 msec]

Are your sure, that nothing has changed in your target GUI.
Images that only get scores of 0.9 and lower are not optimal. The more background the image has towards the edges, the lower the score.
read: http://sikulix-2014.readthedocs.io/en/latest/basicinfo.html#sikulix-how-does-it-find-images-on-the-screen

So check if in your case something has changed in the background pixels around your image. Recapture offending images concentrating on the key aspects and avoiding as much surrounding background as possible.

RaiMan (raimund-hocke) said : #5

I have to admit, that the Capture Preview in the IDE 1.1.x has some problems and can only be taken as a hint.
I will not change that in 1.1.x though.

GZ (george-zador) said : #6

Hi RaiMan,

Thanks for the reply.
As I said, I have tried this is both 1.1.1 stable and the latest 1.1.2 nightly with JAVA 1.8.0_161-b12 (32+64 bit).
I am on Windows 7 64bit, fully up to date.

Hmm this is such a strange bug.
And I can totally reproduce that exists() and findAll() requires totally different precision values to find a match on the image.
No, nothing changes in the UI, I made sure of that.
And this script used to work without problems at the end of last year. So strange!

Also until recently I never got long Java error messages and hangs, that is something totally new. So I am thinking maybe the latest Java 8 update has changed something that is messing with things!!

Also I've noticed that with exists( ,0) with the wait time of 0 I get false positives and false negatives, which never used to happen before. I am strongly thinking something is wrong with Java as that is the only thing that is changed on my system since December..

What do you think?

GZ (george-zador) said : #7

In addition to my previous reply, I can still reproduce it.

These give the same result, so 88 vs 81:
hover(Pattern("icon_12.png").similar(0.88))
for y in findAll(Pattern("icon_12.png").similar(0.81)):
    hover(y)

I think the harder it is to find an image the more this bug shows up.

Is there a way I could overwrite the matching precision for every findAll() operation on the operation itself? Because the images are part of a list and I would prefer to keep one list and just modify the matching precision on the operation not on the image.

I mean all this is a fix until we can track down what is causing this..

GZ (george-zador) said : #8

OK, it is working for now with different set of images for the findAll operation so the precision is working there as well.
I will mark this as solved or now.