sikuli-java: findAll() with a small image returns false matches

Hi there,

Actually there's a pretty good way to find out what was the actual similarity for each recognized image. To find that just use the "highlight" option.

More info here: http://doc.sikuli.org/region.html#Region.highlight

When "highlight" is used, you will see all the details related to the recognized Region. This will allow you to choose the precise similarity.

Eugene

Hi Eugene,

I did use highlight and I could see what the false matches were. But the images are quite different. I have not seen such an issue with larger images, which is probably understandable. To the naked eye, it was hard to understand why the images were 70% same - they were of different color (one green the other grey) and the text on it were different as well (Yes v/s No). So I was wondering if the search algorithms were not that precise when used against very small images.

Thanks.

Hi,

Yes, you're right. According to my experience with Sikuli, it doesn't have any color detection mechanism so usually a color difference will not trigger any similarity change. I think that image size is also an important factor in image recognition however I have not enough knowledge on that matter.

Maybe someone else knows better than me :)

Cheers,
Eugene

1. if both images (yes and no) are visible on the screen and you take an image of yes to search, the match should select yes in any case (provided, that the image shot does not contain much background (even colored areas) towards the edges, which reduces the score and risks false positives).
hence: as long as the distinguishing inner pixel area is nearly as large as the image shot, the match score should be >0.9.
this should work down to images of about 10x10, below that it might get worse.

… and all this is not a must, but only increasing risk for false positives, so there are many cases, where it might work as expected.

2. same case, but only <no> is visible. using <yes> as image and 0.7, you might get a match. The only way to avoid this: use a Pattern with a higher score (the score at which yes would be found (should be >0.9, see above))

3. as mentioned by Eugene: color does not add much difference to the score due to the used statistical match algorithm, but if the wanted image is visible on the screen, it should be found as the best match.
Only in cases, where it is not visible, there is a risk for false positives, if the minSimilarity is not high enough.

If you want to be sure:
match = find(<yes>)
if match.exists(Pattern(<yes>).exact()):
    print "it its yes"
else:
    print "it is not yes, might be no"

looks a bit crude, but this can be used for example to first find a button, and then check, wether it is activated or not (greyed out).

Thanks for the detailed explanation.

The 'yes' and 'no' images were of dimension 24 X 15. Even when there was
one 'yes' and a few 'no' on the screen, it ended up matching the 'yes' and
'no' with similarity of 0.70.

Didn't realize that color did not make a difference to the search
algorithm. In case one wants to check the status of say a button which has
different colors under different conditions, is there a way to work around
this limitation ?

Thanks.
On 9 May 2014 16:56, "RaiMan" <email address hidden> wrote:

> Your question #248324 on Sikuli changed:
> https://answers.launchpad.net/sikuli/+question/248324
>
> RaiMan proposed the following answer:
> 1. if both images (yes and no) are visible on the screen and you take an
> image of yes to search, the match should select yes in any case (provided,
> that the image shot does not contain much background (even colored areas)
> towards the edges, which reduces the score and risks false positives).
> hence: as long as the distinguishing inner pixel area is nearly as large
> as the image shot, the match score should be >0.9.
> this should work down to images of about 10x10, below that it might get
> worse.
>
> … and all this is not a must, but only increasing risk for false
> positives, so there are many cases, where it might work as expected.
>
> 2. same case, but only <no> is visible. using <yes> as image and 0.7,
> you might get a match. The only way to avoid this: use a Pattern with a
> higher score (the score at which yes would be found (should be >0.9, see
> above))
>
> 3. as mentioned by Eugene: color does not add much difference to the score
> due to the used statistical match algorithm, but if the wanted image is
> visible on the screen, it should be found as the best match.
> Only in cases, where it is not visible, there is a risk for false
> positives, if the minSimilarity is not high enough.
>
> If you want to be sure:
> match = find(<yes>)
> if match.exists(Pattern(<yes>).exact()):
> print "it its yes"
> else:
> print "it is not yes, might be no"
>
> looks a bit crude, but this can be used for example to first find a
> button, and then check, wether it is activated or not (greyed out).
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/sikuli/+question/248324/+confirm?answer_id=3
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/sikuli/+question/248324
>
> You received this question notification because you asked the question.
>

the code sample at the end of comment #4 is how to do it.

Does that mean that Pattern.exact(image) causes the search algorithm to be cognizant of the color as well ? (or is it simply that using minSimilarity of 1.0, it will identify the image precisely using the white/grey/black pixels)

The OpenCV.matchTemplate is always used the same way, only in case of exact(), the images (base image and searched image) are taken in their RGB representation (3 channels) and in all other cases both images are converted identically to grayscale (1 channel), so it is a bit faster.