how to deal with visual differences between OS?

Asked by f

Hello,

I'm quite advanced in the project, and realised that my work is not portable to other OSes.
Most of the screenshots are not working when switching to another os, so the project may actually fail at this point.

It's audio plugins, so the interface is extremely similar between oses. But there are slight differences between how OSes handle some graphical elements. Even between to different osx versions.
For example fonts:
osx 10.10: https://dl.dropboxusercontent.com/u/17987279/Screen%20Shot%202016-05-09%20at%2009.40.14.png
osx 10.8: https://dl.dropboxusercontent.com/u/17987279/Screen%20Shot%202016-05-09%20at%2009.40.07.png
win 7: https://dl.dropboxusercontent.com/u/17987279/Screen%20Shot%202016-05-09%20at%2009.40.19.png
(but there are other differences, for instance shades)

So my questions are:
- Is there something I'm not doing correctly?
- Is there some documentations or posts about this problem that I missed?
- How do you deal with such things? Do you really have a set of screenshots for each oses (hardly feasible in my case)?

Thanks for any help

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Eugene S (shragovich) said :
#1

Your observation is correct. Since Sikuli uses OpenCV to work with patterns on the screen, it does so on a pixel level. That means that if the patterns have even slight differences, it might result in Sikuli being unable to find the pattern on the screen. Saying that, given the patterns are shown on the same resolution screen on different OSes, the matching should still work.

There are few things that could be done to fine tune successful detection results. If the differences are truly minor, you can change the minimal similarity score to a lower value so that pattern that matches with a lower percentage will still be considered as match.

In your case however, it looks like the differences are quite significant so I don't think that this approach will suite you. I have also noticed that the text on your images looks blurred. I am not sure about OSX but in windows you can change that. There is the cleartype setting and the "smooth edges of screen fonts" in Windows "Performance Options". All these should be consistent between OSes.

Revision history for this message
RaiMan (raimund-hocke) said :
#2

Yes, with SikuliX you will always need different image sets in such cases, that contain those images, that are different in different situations, because the visual approach is strictly pixel based.

Version 2 will have a better support for creation and management of such image sets.

The next nightly 1.1.1 (this week) has a new feature (FindFailed handler), that would allow you to implement your own solution (e.g. capture on the fly) to overcome this situation.

Font differences like you seem to have, could also be managed in some cases by using the text feature (but might not help due to not being very reliable in the moment).

Another option always is, to avoid those images completely and evaluate their places in the GUI relative to images, that are found reliably. This is most suitable and easy to implement with GUIs that are rather fixed in all situations.

Revision history for this message
f (f34nc015) said :
#3

Thanks to both of you for the answers
I will then have 2 questions.

First yes, lowering the similarity is not an option. I'm pretty high (95 for clicking, 99 for searching), but even by lowering a lot, it either still doesn't find, or generate false positives.

I already implemented some utilitary functions, and I have something kind of convenient to have sikuli screenshots and names when necessary. I'll give 1.1.1 a try though.
First question: when is v2 planned to be released?

Anyway, ok, I will see if the project is still viable with a set of image for each os...
And so am trying to reduce the number of images.

So for clicking, I will use your solution of defining the location of the clicking zone per coordinate, it works well

But for searching if a UI element looks like it should be, I naturally cannot use coordinates, I need a screenshot.
The problem now is that there is a lof of UI elements which can be slightly changed. So if I change all elements and check that the whole interface is as it should be, it doesn't work: because if one element is slightly different, I still have a general match of 99.
To make it work, I need to compare each small zone separately with how they should be, then I have a negative match when I should have.
So my idea was to have only one image of the ui fully modified, and then defining a zone in the image, and searching this zone on screen. But I don't find how to do it. Any clues?
(Is it clear? I mean having a screenshot of the ui totally modified, but searching on the screen only a part of this image)
Or any other alternative maybe?

Thanks again

Revision history for this message
RaiMan (raimund-hocke) said :
#4

... with 1.1.0+

completeGUI = Image.create("completeShot.png")
someIcon1 = completeGUI.getSub(x1, y1, w1, h1)
someIcon2 = completeGUI.getSub(x2, y2, w2, h2)

match = exists(someIcon1, 0)
score = match.getScore()

now score is a float value with some more than 2 decimals.
so you might decide yourself, what you take as equal

Be aware:Tthe internally used OpenCV feature matchTemplate() does not compare each pixel with each other, but uses some statistical summary over the pixels in the rectangle finally leading to the match score. Even in situations, where each pixel might be identical, the score will not be exactly 1 but something like 0.9999999999....

Revision history for this message
RaiMan (raimund-hocke) said :
#5

version 2:
- first nightly builds and info about features in summer
- planned final 2.0.0 end of this year

Revision history for this message
f (f34nc015) said :
#6

Hello thanks a lot for these answers.

I think I can see it working like that.

Now I have 3 questions, if it's not too much:
- when exist() doesn't find the zone at all, it gets stuck, it doesn't exit after the time specified. Is it a bug actually? How can I overcome that?
- I didn't find the doc for getSub or even the image object. Is it I who have a bug? :)
- is it possible not to be statistical with the search? It's kind of problematic that for a specific error, the bigger my zone is, the better the score is. Just getting a boolean, i.e. knowing if all pixels are exactly the same, would certainly do the trick

Thanks again

Revision history for this message
RaiMan (raimund-hocke) said :
#7

--- - when exist() doesn't find the zone at all ...
to make sure: we are talking about SikuliX 1.1.0 or even 1.1.1.

exists(someImage, time)

waits for maximum the time given and then simply comes back (other than find/wait, which through a FindFailed exception)
Be aware: the time value is specified as seconds.

--- - I didn't find the doc for getSub ...
you have to consult the JavaDocs for now, since Image will only be part of the official scripting API beginning with version 2

--- - is it possible not to be statistical ...
simply not with SikuliX. But even in your case this is not needed IMHO.
The challenge: even if all pixels are identical, you will not get an exact 1, it is always a fraction with more or less 9ers behind the comma (this is simply, because internally all is number crunching with floating point numbers and there you do not have exact integer values as result of operations with numbers having a fraction part. Integer results in case are produced by a suitable rounding algorithms).
So again: just decide wether score > 0.999999 or score > 0.9999999999 or whatever means equal to you.
Maybe it is a good idea, to run some test rows.

Can you help with this problem?

Provide an answer of your own, or ask f for more information if necessary.

To post a message you must log in.