feature points extract --- not planned as SikuliX feature

Asked by larryli on 2020-03-09

whether sikulix will support feature points match or not, for example, SURL. SIFT and so on?

as the implementation of doFindImage, it will use matchTemplate finially.
maybe feature points match will be more precise, how do you think?

Question information

English Edit question
Sikuli Edit question
No assignee Edit question
Last query:
Last reply:
RaiMan (raimund-hocke) said : #1

SikuliX basically is a tool to find a specific given image (which usually was captured before) on a screen or in another image (usually a saved screenshot). The current implementation based on matchTemplate is pixel-based, which means, that the given image must be visible in the search area with the same size (width x height). It is found, if all pixels are the same (exact match). Besides that matchTemplate allows to get matches with a lower confidence, which means, that not all pixels are the same. The lower the confidence, the more pixels differ. It is up to the user, to accept matches with lower confidence as matches (which usually does not make sense with confidence values lower than 0.7 to 0.8).

This implementation has 2 main drawbacks:
- an image might not work in another environment, where it is not shown with the same size.
- an image might contain too much background pixels, that vary over time or in other environments
- the environment (usually browser) might render the image differently as where it was captured (usually handling of edges)

For the first issue we can try with resize, for the others with shot-optimisation, masking and accepting lower confidence values.
And we can have extra shots for different environments.

If your application case and your usage of SikuliX fit, then the matchTemplate based image search is precise.

It does not work reliably in situations of fast moving/changing objects and it cannot be used, where objects are shown with 3D aspects.

The current implementation is sufficient for automating/testing of GUI based applications and (though heavily used) has its limitations with game bots (especially video-like games).

Conclusion: as long as the main target for SikuliX are GUI based applications, then there is no need to use other techniques.

... but since the whole world of OpenCV is available in SikuliX (Java level API), every user of SikuliX can implement own features relatively easy.

larryli (larryli2020) said : #2

Dear RaiMan

thanks for your answer. I can understand your mean.
in my thinking, some time, there is backgroup for catpure, if backgroup is changed, it will not easy to detect by capture image.
so I am think it can provide key image, it means features of image, and then to make compare.

BTW, I see the github in your homepage, you are working on Python version also.
https://github.com/RaiMan/sikulix4python, whether you can want to import whole world of opencv?

RaiMan (raimund-hocke) said : #3

--- it will not easy to detect by capture image, if background changes
therefor it is strongly recommended, to have as little background in the image as possible. The background pixels should not be more than 10 - 20% of the whole image pixels. The more background pixels, the higher the risk for not found or false positives.
This is usually possible with precise capturing. In 2.1.0 there is a feature in the IDE Preview, to rework the image after capture.

 --- whether you can want to import whole world of opencv (running Python)
As mentioned, the whole world of OpenCV is already available at the Java level. The Python project is only a server-based implementation of SikuliX that you can use the SikuliX features in the real Python environment. Of course, a user can feel free in this case to import Python OpenCV, but SikuliX will always use the features on the Java level.

BTW: A main problem with image compare (matchTemplate) is, when the same pixel structures differ only in color. The match scores (values between 0 and 1) might then only differ by less than 0.0001. This cannot be solved with feature point extraction. The most prominent example are buttons with different states signalled by color.

Can you help with this problem?

Provide an answer of your own, or ask larryli for more information if necessary.

To post a message you must log in.