SikuliX

Same image not found on different devices/systems --- especially with textual content

Asked by Gjorgji

I'm creating scripts for testing an application on multiple OSes. The application is the same on all OSes.

The scripts run perfectly on all OSes, but I'm encountering an issue with finding some images if they were taken from the other OS. All screenshots were taken from Sikuli's IDE. Version of Sikuli used is 2.0.5.

Initially, I create the scripts and screenshots from the Linux (Ubuntu) which is a virtual machine run on Oracle VM Virtual Box on a Windows machine.
Then I ran the same scripts on a (Retina Display) MacBook Air (M1, 2020) with latest Big Sur on it. The images were not found and the scripts were failing.
Afterward, I retook the same screenshots on the Mac device, and the script was working perfectly on it, but now it was failing on the Linux VM. Confirmed that it failed on Windows as well because I wanted to see if it was perhaps failing because the Linux was a VM.

I found that if I lower the similarity to ~0.57 the screenshot is found, but this might cause me some issues in the future.

I did some research and found that Retina displays have a higher resolution, but that if a screenshot is taken from Sikuli's IDE, the resolution is halved to counter this.
Also, I did try with using `resize()` on the images with different values, and no luck there.

I really do not want to take multiple screenshots of the same things for each environment, so that's why I'm writing here.

Is there a way to accomplish what I want, that is a single image be found on multiple OSes and devices for the same application? Perhaps setting the displays to specific resolutions? Or is there a way to somehow modify the screenshots within the script to resolve this?

Thanks.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
Gjorgji (gorgisl) said : 		#1

Forgot to mention, all the devices have Java 11 installed from AdoptOpenJDK.

Revision history for this message
Best RaiMan (raimund-hocke) said : 		#2

No easy solution for this.

Since Java 11 getting the screen content is compatible for all screen setups (Windows, Mac, Linux) even for Retina/4K displays.
Eliminating the pixel-boost is done inside Java.

Resolution usually does not matter, as long as the display setup is run with 100% scaling.

Background: to find an image on a screen (in fact an image shot of the screen), it must be visible with the same width and height.
SikuliX's resize feature might help in situations, where you know the scaling factor (e.g. 120%).

If we now look at different systems, then there are big differences, how images might be rendered to the screen (compatible resolution and scaling supposed). So even if the image might be visible with the same width and height, the pixel content might differ substantially (which is not always obvious for the user's eyes).

Basic recommendations:
- make the shots with the SikuliX IDE (given in your case)
- concentrate the shot to the parts that makes it unique (as little surrounding background as possible)
- avoid textual content in the shot (font rendering is the one, that brings the most differences)

In the end, there is no other solution, than to have image sets for every system/screen setup, if the above mentioned measures do not work sufficiently.

similarities below 0.7 but higher than 0.5 indeed are risky, but in testing environments it might be possible, to make additional checks (e.g. match position inside app window), but of course lead to more effort.

Revision history for this message
Gjorgji (gorgisl) said (last edit ): 		#3

Thanks RaiMan, that solved my question.

Revision history for this message
Gjorgji (gorgisl) said : 		#4

Also, thanks RaiMan for the quick answer, appreciate it.

> - avoid textual content in the shot (font rendering is the one, that brings the most differences)

This here explains all of my issues. As I said, I wasn't experiencing this issue on some images, so after reading this I double-checked and indeed images with text in them are the ones that experience this.
For instance, buttons that have only icons are found without any issues on different systems.

Ahh, I was afraid this would be the case, but if the only option is to create images for each system/device, that is what I shall do.

Again, thanks, and my question was solved.

Revision history for this message
tomanderson119 (tomanderson119) said : 		#5

The same problem with the image I am facing for my quackity merch store https://quackitymerchandise.shop and I want to solve my problem immediately. Would you help me with it?

Revision history for this message
Gjorgji (gorgisl) said (last edit ): 		#6

The resolution I mentioned above helped me, that is taking an image of the same thing from different environments (OSes), and using the needed one.

I accomplished this by having a function that checks on which OS the script is ran, and returns it as a string. Let's say the function is named `which_os()`

Then when taking screenshots I have Sikuli set up to name the images manually, and follow a naming convention where at the end of the name I add `_OS`, i.e. `button1_linux.png`, `button1_macos.png`, `button1_windows.png`, etc.

Afterward, if for instance, I need Sikuli to click that button, I've written it like this:
`click("button1_" + which_os() + ".png")`

This way, if I keep following the same naming convention, no matter on which OS I ran the script, if there's a screenshot for that button for that OS, it will find it and use it, and it will not experience the issue mentioned above.

Revision history for this message
RaiMan (raimund-hocke) said : 		#7

Another option is to use SikuliX's ImagePath feature.
- create a unique image set for every needed variant
- at script start check the given variant somehow (e.g. searching some reference image)
- set the ImagePath accordingly

In any case you need a strict naming convention to minimize the overall effort.