[2.0.4] Pattern.resize() does not work in scale-down situations --- possible bug

Asked by Simon Plivav on 2020-12-20

this possible bug is tracked on GitHub:
https://github.com/RaiMan/SikuliX1/issues/406
-----------------------------------------------------------------------------------

Hello friends, using Sikulix 2.0.4 in Win7 - big trouble when running the script on a different system where the web app to automate is slightly scaled down. SCREEN.find() on the script patterns won't work anymore. So, I try at-runtime code to get the scaling factor for resizing of all the patterns to get find() to work, but SkuliX does not work as expected. Here the issue in 'manual' code:

pat = Pattern("1608112684558.png") # a 737x50 search pattern
pat.similar(0.1) # to avoid any FindFaileds

[On the scaled-down environment...]

pat.resize(1)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives 0.603262484074, OK

pat.resize(0.99)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives 0.753167867661, OK

pat.resize(0.98)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives exception below!
"...org.sikuli.script.SikuliXception: image to search (737, 50) is larger than image to search in (729, 49)"
-> Does not resize the pattern anymore but uses the previous resized pattern as search region!?

pat.resize(0.97) or pat.resize(1.0)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives exactly same exception as before!
"...org.sikuli.script.SikuliXception: image to search (737, 50) is larger than image to search in (729, 49)"
-> Very same error details with all different resize values!

***Now closing and restarting the SikuliX IDE ceteris paribus:***
pat.resize(0.98)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives 0.878550231457, OK
-> Now it works, but why?

pat.resize(0.97) or pat.resize(1.0)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives exception again
"...org.sikuli.script.SikuliXception: image to search (737, 50) is larger than image to search in (722, 49)"
-> Again, SikuliX bitching around with me, and needs to get restarted which would be, btw, acceptable with my wife, though...

Is this a (known?) bug in SikuliX, or is my understanding of .resize() bad? For sure I don't understand the exception message "image to search (w, h) is larger than image to search in (w, h)" - just makes no sense, also because the search region is SCREEN which is not in there. Please anyone help - I cannot deliver my solution if it cannot determine the needed pattern resizing factor based on a reference pattern for scale-changed applications across different screen resolutions///

Best!
Simon

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
2021-01-11
Last query:
2021-01-11
Last reply:
2021-01-11
Simon Plivav (simpliv742) said : #1

Just to also give this information...

The described issue with pat.resize() and reg.find(pat) does *not* occur if I am scaling up, i.e. when I provide resizing values >=1, even as the SikuliX documentation writes that pat.resize() only takes arguments between 0 and 1!!!

Here the code that finds matches after upscaling the search pattern, and that works:

pat = Pattern("1608539949576.png") # a pattern / search image of 676x41
pat.similar(0.3)

booContinue = True
numPatternScalingFactor = 1.0
numScorePrev = 0

while booContinue:
    numPatternScalingFactor+=0.005
    pat.resize(numPatternScalingFactor)
    numScore = SCREEN.find(pat).getScore()
    if numScore<numScorePrev or numPatternScalingFactor>1.50:
        booContinue = False
    print(numScorePrev)
    print(numPatternScalingFactor)
    numScorePrev = numScore

Simon Plivav (simpliv742) said : #2

Now I just switched the working upscaling code to a downscaling one by changing '+=' to '-=' in one line:

...
numPatternScalingFactor-=0.005
...

In line with the original issue description I get this exception:

[error] script [ test ] stopped with error in line 12
[error] org.sikuli.script.SikuliXception ( org.sikuli.script.SikuliXception: image to search (676, 41) is larger than image to search in (672, 40) )
[error] --- Traceback --- error source first
line: module ( function ) statement
12: main ( <module> ) numScore = SCREEN.find(patDozenBar).getScore()
[error] --- Traceback --- end --------------
0
0.995

This confirms that pat.resize() with arguments<1 does not work in repetion, doesn't it?

RaiMan (raimund-hocke) said : #3

an image used to search in another image must not be larger in width and/or height in pixels.

your case: image to search (676, 41) is larger than image to search in (672, 40)

Since you are using the constant SCREEN, I guess you are using setROI() to define the search region.

BTW: saying SCREEN.find() is the same as find(), since undotted functions are automatically bound to SCREEN.
It is not recommended to use SCREEN in your script or even modify it.

Simon Plivav (simpliv742) said : #4

Hi RaiMan, ty for the response! My search pattern is (676, 41), my search region is SCREEN which is in my case (1366x768). So, a .find() operation should work from that perspective, shouldn't it? I don't understand where Sikulix is getting the "(672, 40)" from - I have not specified such object or dimension explicitely, not via a Region(), ROI(), nor a Rectangle() statement.

Why is it replacing the 'SCREEN' in "numScore = SCREEN.find(pat).getScore()"?
What I note is that 676x(1-0.005)=672.62 and 41*(1-0.005)=40.795, so to me it looks like Sikulix is doing one downscaling of the search pattern object by the factor 0.005, does a first search, then replaces the region to search in with that smaller-now pattern, and, bum, it fails the next .find() because of "image to search (676, 41) is larger than image to search in (672, 40)".

Could could you maybe try, 2 minutes, to replicate the issue with my code? Freshly I just took a Windows Desktop icons snapshot pattern and I ran the three .find()s one-by-one:

pat = Pattern("1608930740607.png") # a 153x92 search pattern
pat.similar(0.1) # to avoid any FindFaileds

pat.resize(1)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives 0.99999922514, OK

pat.resize(0.99)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives 0.906620442867, OK

pat.resize(0.98)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives exception below

[error] JythonRunner: runPython: (
pat.resize(0.98)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives exception below
) raised: Traceback (most recent call last):
  File "<string>", line 2, in <module>
 at org.sikuli.script.Finder$FindInput2.isValid(Finder.java:1244)
 at org.sikuli.script.Finder$Finder2.doFind(Finder.java:616)
 at org.sikuli.script.Finder$Finder2.find(Finder.java:582)
 at org.sikuli.script.Finder.find(Finder.java:195)
 at org.sikuli.script.Region.doCheckLastSeenAndCreateFinder(Region.java:2832)
 at org.sikuli.script.Region.checkLastSeenAndCreateFinder(Region.java:2802)
 at org.sikuli.script.Region.doFind(Region.java:2749)
 at org.sikuli.script.Region.find(Region.java:2271)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
org.sikuli.script.SikuliXception: org.sikuli.script.SikuliXcep
tion: image to search (153, 92) is larger than image to search in (151, 91)

Then I closed and restarted the SikuliX IDE, and run the whole "program" in one go. Returns:

0.99999922514
0.906620442867
[error] script [ DesktopTest ] stopped with error in line 13
[error] org.sikuli.script.SikuliXception ( org.sikuli.script.SikuliXception: image to search (153, 92) is larger than image to search in (151, 91) )
[error] --- Traceback --- error source first
line: module ( function ) statement
13: main ( <module> ) mat = SCREEN.find(pat)
[error] --- Traceback --- end --------------

Oh, and when I use the replacement values (1, 1.1, 1.15) for upscaling of the pattern it works fine!!!
0.999999642372
0.586187005043
0.514091551304

What am I doing wrong?

Frohes Fest!
Simon

Simon Plivav (simpliv742) said : #5

Issue is still open for my project - does anyone have a solution how to make a SikuliX solution self-adjusting if/where the pixel size of screen object has changed (e.g. run on different laptop or the 2nd monitor)?

Best!
Simon

RaiMan (raimund-hocke) said : #7

Here is a tested example, how I succeeded, to evaluate the resize factor to apply to all images in scripts run against an environment, where the GUI elements are enlarged.

I am using macOS 11.1 on a Retina display with the standard resolution of 2048x1152.
When I activate a scaling option in the SystemPrefs, an image captured in the standard is still found without problems.

The attached example uses the scaling option from the app menu (in this case Safari) View -> ActualSize/ZoomIn/ZoomOut.
In the example I display a web page in Safari, capturing an image in the ActualSize state and then use the ZoomIn.
When zoomed in (enlarged) the image is no longer found.
I then use a loop and Pattern().resize(), to find the image again by stepping about 1 pixel up for the larger value of width and height.

After having found the reference image, I use the evaluated resize factor, to find other images by using Pattern().resize(factor).

My tests gave the following output:

— test.py in state ActualSize
imgRaiMan: time=0.365, score=99.99988
imgSmiley 100.00000
imgSikuliX 100.00000
imgSikuliX 99.99995

— eval.py one time CMD + from ActualSize
imgRaiMan: time=10.250, n=27, factor=1.146, score=99.63880
imgSmiley score=95.28467
imgSikuliX score=92.74814
imgRaimund score=92.10581

— eval.py two times CMD + from ActualSize
imgRaiMan: time=17.860, n=51, factor=1.258, score=99.72572
imgSmiley score=93.89349
imgSikuliX score=90.91188
imgRaimund score=94.77862

— eval.py three times CMD + from ActualSize
imgRaiMan: time=27.328, n=79, factor=1.377, score=99.60801
imgSmiley score=80.45675
imgSikuliX score=66.09237
imgRaimund not found with score > 60

Some comments:
- the score stays rather stable for the first graphics-only image
- the smiley seems to be handled differently (greater reduction of score)
- for images containing text the score-reduction is significantly (the more text the greater)
- above a enlargement of 30% this approach is is not really helpful

The approach used in evalResize.py surely should be tested in other scenarios.
Internally the basic resize feature of Java for BufferedImage objects is used.

To run the scripts (mandatory 2.0.4) in the IDE:
- unzip the stuff somewhere
- In the IDE open dialog go to the folder and select/open test.py/evalResize.py

As you might be aware: the images are shared between all scripts in the same folder.

Automatic naming of images in the IDE can be achieved so:
- in a new line write imgName =
- let the cursor after the =
- use the capture button/short-cut
… and the image will be captured, inserted and named imgName.png
Now in the rest of the script you can use the variable imgName instead of „imgName.png“.

https://www.dropbox.com/s/hpn1lcxcz9v8ek2/testResize.zip

Simon Plivav (simpliv742) said : #8

Ok, thanks a lot, Raiman. I see confirmed that scaling up the search patterns works. This should work in cases where, for example, vector-type images on a web page, or web games, scale up when they have "more room" on the screen to expand when programmed so, adjusting their pixel size according to the other web page content.

Btw, I understand now that a monitor with a different screen resolution per se would not change the pixel size of web page on-screen elements, and hence cause no issue. Is this correct?

Still a problem (SikuliX bug?) I see in the scale-down situation, like I had: A web app (game) had shrunk on an other-than-dev environment because the web page there had, for unknown reasons but as a matter of fact, additional HTML elements present that took up space - and the web app shrank thereof.
Similarly, when I ran my Chrome zoomed to 90%, and changed your code line "rFactor += getStep(imgEval, rFactor)" to "rFactor -= ge..." I got, as before, the exeption "rg.sikuli.script.SikuliXception: image to search (179, 72) is larger than image to search in (178, 71)".

As down-scaling of patterns apparently does not work, and also is giving unexplained exception messages as per my above posts, shouldn't what I found be a review or change item for SikuliX 2.0.5? Can I do anything to help, if needed?

Best!
Simon

Best RaiMan (raimund-hocke) said : #9

Thanks for additional comments. No help needed in the moment.

I have to admit, that I did not test the scale-down situation.
Will do and fix as needed.

The possible bug is tracked on GitHub.

Simon Plivav (simpliv742) said : #10

Thanks RaiMan, that solved my question.