[2.0.4] Pattern.resize() does not work in scale-down situations --- possible bug

Just to also give this information...

The described issue with pat.resize() and reg.find(pat) does *not* occur if I am scaling up, i.e. when I provide resizing values >=1, even as the SikuliX documentation writes that pat.resize() only takes arguments between 0 and 1!!!

Here the code that finds matches after upscaling the search pattern, and that works:

pat = Pattern("1608539949576.png") # a pattern / search image of 676x41
pat.similar(0.3)

booContinue = True
numPatternScalingFactor = 1.0
numScorePrev = 0

while booContinue:
    numPatternScalingFactor+=0.005
    pat.resize(numPatternScalingFactor)
    numScore = SCREEN.find(pat).getScore()
    if numScore<numScorePrev or numPatternScalingFactor>1.50:
        booContinue = False
    print(numScorePrev)
    print(numPatternScalingFactor)
    numScorePrev = numScore

Now I just switched the working upscaling code to a downscaling one by changing '+=' to '-=' in one line:

...
numPatternScalingFactor-=0.005
...

In line with the original issue description I get this exception:

[error] script [ test ] stopped with error in line 12
[error] org.sikuli.script.SikuliXception ( org.sikuli.script.SikuliXception: image to search (676, 41) is larger than image to search in (672, 40) )
[error] --- Traceback --- error source first
line: module ( function ) statement
12: main ( <module> ) numScore = SCREEN.find(patDozenBar).getScore()
[error] --- Traceback --- end --------------
0
0.995

This confirms that pat.resize() with arguments<1 does not work in repetion, doesn't it?

an image used to search in another image must not be larger in width and/or height in pixels.

your case: image to search (676, 41) is larger than image to search in (672, 40)

Since you are using the constant SCREEN, I guess you are using setROI() to define the search region.

BTW: saying SCREEN.find() is the same as find(), since undotted functions are automatically bound to SCREEN.
It is not recommended to use SCREEN in your script or even modify it.

Hi RaiMan, ty for the response! My search pattern is (676, 41), my search region is SCREEN which is in my case (1366x768). So, a .find() operation should work from that perspective, shouldn't it? I don't understand where Sikulix is getting the "(672, 40)" from - I have not specified such object or dimension explicitely, not via a Region(), ROI(), nor a Rectangle() statement.

Why is it replacing the 'SCREEN' in "numScore = SCREEN.find(pat).getScore()"?
What I note is that 676x(1-0.005)=672.62 and 41*(1-0.005)=40.795, so to me it looks like Sikulix is doing one downscaling of the search pattern object by the factor 0.005, does a first search, then replaces the region to search in with that smaller-now pattern, and, bum, it fails the next .find() because of "image to search (676, 41) is larger than image to search in (672, 40)".

Could could you maybe try, 2 minutes, to replicate the issue with my code? Freshly I just took a Windows Desktop icons snapshot pattern and I ran the three .find()s one-by-one:

pat = Pattern("1608930740607.png") # a 153x92 search pattern
pat.similar(0.1) # to avoid any FindFaileds

pat.resize(1)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives 0.99999922514, OK

pat.resize(0.99)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives 0.906620442867, OK

pat.resize(0.98)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives exception below

[error] JythonRunner: runPython: (
pat.resize(0.98)
mat = SCREEN.find(pat)
print(mat.getScore()) # gives exception below
) raised: Traceback (most recent call last):
  File "<string>", line 2, in <module>
 at org.sikuli.script.Finder$FindInput2.isValid(Finder.java:1244)
 at org.sikuli.script.Finder$Finder2.doFind(Finder.java:616)
 at org.sikuli.script.Finder$Finder2.find(Finder.java:582)
 at org.sikuli.script.Finder.find(Finder.java:195)
 at org.sikuli.script.Region.doCheckLastSeenAndCreateFinder(Region.java:2832)
 at org.sikuli.script.Region.checkLastSeenAndCreateFinder(Region.java:2802)
 at org.sikuli.script.Region.doFind(Region.java:2749)
 at org.sikuli.script.Region.find(Region.java:2271)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
org.sikuli.script.SikuliXception: org.sikuli.script.SikuliXcep
tion: image to search (153, 92) is larger than image to search in (151, 91)

Then I closed and restarted the SikuliX IDE, and run the whole "program" in one go. Returns:

0.99999922514
0.906620442867
[error] script [ DesktopTest ] stopped with error in line 13
[error] org.sikuli.script.SikuliXception ( org.sikuli.script.SikuliXception: image to search (153, 92) is larger than image to search in (151, 91) )
[error] --- Traceback --- error source first
line: module ( function ) statement
13: main ( <module> ) mat = SCREEN.find(pat)
[error] --- Traceback --- end --------------

Oh, and when I use the replacement values (1, 1.1, 1.15) for upscaling of the pattern it works fine!!!
0.999999642372
0.586187005043
0.514091551304

What am I doing wrong?

Frohes Fest!
Simon

Issue is still open for my project - does anyone have a solution how to make a SikuliX solution self-adjusting if/where the pixel size of screen object has changed (e.g. run on different laptop or the 2nd monitor)?

Best!
Simon

Here is a tested example, how I succeeded, to evaluate the resize factor to apply to all images in scripts run against an environment, where the GUI elements are enlarged.

I am using macOS 11.1 on a Retina display with the standard resolution of 2048x1152.
When I activate a scaling option in the SystemPrefs, an image captured in the standard is still found without problems.

The attached example uses the scaling option from the app menu (in this case Safari) View -> ActualSize/ZoomIn/ZoomOut.
In the example I display a web page in Safari, capturing an image in the ActualSize state and then use the ZoomIn.
When zoomed in (enlarged) the image is no longer found.
I then use a loop and Pattern().resize(), to find the image again by stepping about 1 pixel up for the larger value of width and height.

After having found the reference image, I use the evaluated resize factor, to find other images by using Pattern().resize(factor).

My tests gave the following output:

— test.py in state ActualSize
imgRaiMan: time=0.365, score=99.99988
imgSmiley 100.00000
imgSikuliX 100.00000
imgSikuliX 99.99995

— eval.py one time CMD + from ActualSize
imgRaiMan: time=10.250, n=27, factor=1.146, score=99.63880
imgSmiley score=95.28467
imgSikuliX score=92.74814
imgRaimund score=92.10581

— eval.py two times CMD + from ActualSize
imgRaiMan: time=17.860, n=51, factor=1.258, score=99.72572
imgSmiley score=93.89349
imgSikuliX score=90.91188
imgRaimund score=94.77862

— eval.py three times CMD + from ActualSize
imgRaiMan: time=27.328, n=79, factor=1.377, score=99.60801
imgSmiley score=80.45675
imgSikuliX score=66.09237
imgRaimund not found with score > 60

Some comments:
- the score stays rather stable for the first graphics-only image
- the smiley seems to be handled differently (greater reduction of score)
- for images containing text the score-reduction is significantly (the more text the greater)
- above a enlargement of 30% this approach is is not really helpful

The approach used in evalResize.py surely should be tested in other scenarios.
Internally the basic resize feature of Java for BufferedImage objects is used.

To run the scripts (mandatory 2.0.4) in the IDE:
- unzip the stuff somewhere
- In the IDE open dialog go to the folder and select/open test.py/evalResize.py

As you might be aware: the images are shared between all scripts in the same folder.

Automatic naming of images in the IDE can be achieved so:
- in a new line write imgName =
- let the cursor after the =
- use the capture button/short-cut
… and the image will be captured, inserted and named imgName.png
Now in the rest of the script you can use the variable imgName instead of „imgName.png“.

https://www.dropbox.com/s/hpn1lcxcz9v8ek2/testResize.zip

Ok, thanks a lot, Raiman. I see confirmed that scaling up the search patterns works. This should work in cases where, for example, vector-type images on a web page, or web games, scale up when they have "more room" on the screen to expand when programmed so, adjusting their pixel size according to the other web page content.

Btw, I understand now that a monitor with a different screen resolution per se would not change the pixel size of web page on-screen elements, and hence cause no issue. Is this correct?

Still a problem (SikuliX bug?) I see in the scale-down situation, like I had: A web app (game) had shrunk on an other-than-dev environment because the web page there had, for unknown reasons but as a matter of fact, additional HTML elements present that took up space - and the web app shrank thereof.
Similarly, when I ran my Chrome zoomed to 90%, and changed your code line "rFactor += getStep(imgEval, rFactor)" to "rFactor -= ge..." I got, as before, the exeption "rg.sikuli.script.SikuliXception: image to search (179, 72) is larger than image to search in (178, 71)".

As down-scaling of patterns apparently does not work, and also is giving unexplained exception messages as per my above posts, shouldn't what I found be a review or change item for SikuliX 2.0.5? Can I do anything to help, if needed?

Best!
Simon

Thanks RaiMan, that solved my question.