Fault tolerance for target images

Asked by robin gupta on 2020-06-13

We are planning to use Sikuli for automation testing on the Salesforce platform. However, since the local development laptops’ screen resolution vs target machines/Jenkins’ box screen resolution could be different, therefore wanted to know about the fault tolerance of the Sikuli platform.

For example, would the image detection api be able to detect the image if the source screenshot is from 1440x900 resolution but the scripts are running on a server with 1280x800 set as the resolution.

Kudos on the progress of the library so far 👌

Question information

English Edit question
Sikuli Edit question
No assignee Edit question
Solved by:
Last query:
Last reply:
RaiMan (raimund-hocke) said : #1

For SikuliX internally using OpenCV's machTemplate() only the pixels are relevant.

If your images from the source machine (1440x900) are shown on the target machine (1280x800) with the same pixels in width and height, then the images would only look a bit smaller, but will be found by SikuliX.

Problems arise only in cases, where the rendering process on other machines add pixels (resized or even distorted).

Come back if your tests say something different.

Be aware: the target machines must either have a real screen or some fake-screen (e.g. with xvfb ...).

robin gupta (smilinrobin) said : #2

Thanks for the prompt response RaiMan. I am believing that I would have to do something like one of the below:
1. click(Pattern("1530562928013.png").similar(0.96))
2. Match multiple images from resolution/OS/Browsers of the target systems. Like :
if Reg.exists("12.png",0) or Reg.exists("13.png",0) or Reg.exists("14.png",0)or Reg.exists("15.png",0)or Reg.exists("28.png",0):

Please let me know if there are any other ways , in which we can achieve flexibility of the Sikuli library in matching the target image, with the source image.

RaiMan (raimund-hocke) said : #3

I do not understand your question - has nothing to do with the original question.

And I do not understand, what you mean by "achieve flexibility".

Best TestMechanic (ndinev) said : #4

I think I understand Robin concern and will try to share my approach.

Here are some recommendations based on my 8 years experience using Sikuli for testing applications on Mac, Windows and Linux:

1. Try whenever possible to use same resolution on test development and test execution machines.
Why?: Because many web and desktop applications now have responsive design and will scale their UI elements when width of app is changing. You can mitigate this by using fixed window size if resolution change is not an option.
The another problem with resolution is that visible area of the screen is different. So for example you may need to scroll down to see some element visible on higher resolution.
In general you need to keep not only resolution similar, but also things like fonts smotening/clear types, color schema... etc.

2. However when testing a multi platform application( or inability to keep resolution for one platform similar) you may use following approach
make a Sikuli ui-map file that contains only UI map definitions of your elements. Here is the example:

class NotepadUI:
  class scrSettings:
    button_Save = button_save.png
    button_Cancel = button_cancel.png

Then in your script use elements like this:

Make your tests running with this UI map on one OS(or in your case resolution)
Then create second(and third and so on) UI maps for different OSes(or ui configurations)

How this helps

RaiMan (raimund-hocke) said : #5

Thanks for this helpful information.

I will add a chapter to the docs based on your "multiplatform-images" experience.

This will the be the place, to talk about other possibilities and future enhancements.

--- YOU SAY:
... you may need to scroll down to see some element visible on higher resolution.
IMHO this must read "... on lower resolution"

--- your item 2. (image sets)
I think this can easier be achieved by creating set1.sikuli, set2.sikuli, ..., setN.sikuli using the SikuliX IDE, which all have the same set of images like
    button_Save = button_save.png
    button_Cancel = button_cancel.png

... using the handy feature of the IDE:
writing on a line:
imageFooBar =
... and capturing the image leaving the cursor on the line after the =
... and the image will be auto-named imageFooBar.png

At runtime you can switch the image sets based on environment just using
... and you will use the appropriate image when using sthg. like "imageFooBar.png".

--- Another aspect is resize:
If you know the standard factor with that the images are evenly resized in the target environment, you can use the Settings.AlwaysResize = factor feature and before applying an image it will be resized accordingly.
Individual resize factors for images can be applied using the Pattern class.

robin gupta (smilinrobin) said : #6

Awesome . Thanks @TestMechanic and @Raiman. This solved my problem.

robin gupta (smilinrobin) said : #7

Thanks TestMechanic, that solved my question.

TestMechanic (ndinev) said : #8

I will add comment because this actually is the most important topic when I need to convince someone to use Sikuli :-)

about 1) I meant that if you are using higher resolution you will see more UI elements on the screen. Then when running test on lower you may need to handle the risk of element not to be visible and to scroll down to see those images(visible ok on higher resolution without scrolling down)

about 2). I meant exactly that - have ui_map_win.sikuli, ui_map_mac.sikuli and ui_map_lin.sikuli and load whenever you need. for particular OS

RaiMan (raimund-hocke) said : #9

Thanks for clarification. Highly appreciated.