Trouble finding matches for images

Asked by alex

For my Sikulix script I captured a few images from the screen and use them in SikuliX e.g. for clicking on them.
This works most of the time, but sometimes there is a weird issue I could not understand or find information about.

Here is the deal:
SikuliX fails to find an image on the screen. I can see that it is clearly there. I am aware of the similarity value and how to use it.
However what happens here: when I go to the match preview in SikuliX it also finds that the image is on screen as it should be with the set similarity. But instead of marking the region with a bright red color, it is now marked with a pinkish or magenta color.
So basically every time I get a FindFailed, the match preview shows a pink region instead of a red one.
The image in question is the only one of its kind on the screen and the match preview only finds one image on the screen that it marks with a pink region.
If I slide the similarity value lower (starting e.g. from 80%), the found image remains pink. If I slide it even lower, other unrelated regions begin to pop up marked with the same pink color. Sliding similarity slightly above the initial value ther is no more found image at all.

What exactly is happening here? What do the different colors in the match preview mean? And how can I tell SikuliX to accept an image on the screen as found even if this situation occurs?

Using version 1.1.3 on windows 10, but had this issue since 1.1.1.

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

--- Preview colors
the colors with that matches are shown correspond to the color bar below the similarity slider:
- reddish: similarity high >0.85
- pinkish: similarity medium: around 0.7
- other colors: low <0.6

--- image quality
see: http://sikulix-2014.readthedocs.io/en/latest/basicinfo.html#sikulix-how-does-it-find-images-on-the-screen
images should always be found with a score above 0.8 - 0.9.
images that get a lower score bare the risk to not be found or even create false positives.

--- Preview difference to score at runtime
When running the script, similarity scores are rounded to 2 decimals.
Preview internally uses findAll() and handles the score fully as float values without rounding.
This may lead to inconsistencies as you have.

--- suggestions

1. recapture your image, so it has as little background towards the edges and concentrates on the key aspects. This should give you a similarity score beyond 0.8 - 0.9.

2. IDE feature "Show": in the top bar of the IDE there is a button "Show", that works so:
- put the cursor somewhere in the line with the image/pattern
- take care that the image is visible on the screen
- click the button
- the image should be found and highlighted (if not: nothing happens, null is written in the message area)
For just checking this is slightly better than Preview, because it internally uses the find feature.
I will revise this feature to be more verbose.

Revision history for this message
alex (privatesnowball) said :
#2

When I first capture the images from the GUI, SikuliX finds them with a similarity value of 99%.
Then after a few passes, the situation described may occur. I recaptured the images multiple times. Sometimes it helped for a while, but then the error came back.
Capturing a larger portion of the image works for a few, but other images can not have a larger capture area because the region around them is subject to change which would definitely result in a FindFailed.
It is probably worth noting that I have this issue exclusively when at least one of the image dimensions is rather small in the order of 20-30 pixels. But as I said, I can not capture larger images for all of them.

Revision history for this message
RaiMan (raimund-hocke) said :
#3

since generally the captured image are fix and do not change, there must happen some changes on the screen in the area, where you think your capture should be found.

The smaller the images the fewer changing pixels may lead to this effect.

I have to admit, that these situations are hard to debug.

this might be helpful:
if not exists(img):
  saveLastScreenImage()
  exit(1)

where img is the offending image.

You will have a LastScreenImage.png (the screenshot, where in your image was not found) in the folder SikulixStore in the <sikulix appdata> folder (see docs).

all the best

visually compare the image with the

Revision history for this message
RaiMan (raimund-hocke) said :
#4

You need some imageviewer, that allows to zoom into an image down to the pixel level (each pixel is shown as a square)

Revision history for this message
alex (privatesnowball) said :
#5

I was able to narrow down the issue a little bit more.
The problem only occurs if the image is not found at the exact same location where I captured it in the first place. Even slightly moving the window that the application is running in by a few pixels on the screen can trigger these events. Then it can also happen for larger images.

Revision history for this message
RaiMan (raimund-hocke) said :
#6

Since with SikuliX all is about squares of pixels found on the screen having the same size (width x height) as the captured image, not found situations always are caused by changes to the pixels on the screen.

So analyse the situation on the screen at time of failure as suggested in comment #3.

Revision history for this message
alex (privatesnowball) said :
#7

I have the last screen on the monitor. I can analyze it however I want. OpenCV/Sikulix initially throws a FindFailed. Then when I analyze the screen where the FindFailed occured SikuliX finds the image, but with a lower similarity and marks the region where the image is with purple color instead of red.
Since it is a static image, there is no point in analyzing a screenshot instead of the screen itself.

Revision history for this message
RaiMan (raimund-hocke) said :
#8

I cannot see, that you did anything, what I suggested in comments #1 and #3.

If you get a FindFailed, then the image was not found on the screen in that moment.
This moment can only be documented and analysed with a shot according to comment #3.

... Since it is a static image ...
I am sure it is, but the screen is not!

Go back to
http://sikulix-2014.readthedocs.io/en/latest/basicinfo.html#sikulix-how-does-it-find-images-on-the-screen

... and understand how SikuliX works.

BTW: comment #5 is simply wrong.

Can you help with this problem?

Provide an answer of your own, or ask alex for more information if necessary.

To post a message you must log in.