How to do capture() with an getScore = 0.91 or greater?

Asked by Shawn Lao

Hi there,

I'm a newbie to Sikuli, Jython, and Python. I love the tool and how flexiable it is. I will be replacing EggPlant with Sikuli, that how good your tool is and plus I love it's open source.

So on that note, I'm trying to do a capture of a set x, y, w, h and a match of 0.91 (I've read that it's best to use 0.91 rather then 1 for pixel to pixel compare.

For example, I tried the following with failure:
#Capture the view
newarea = capture (1, 171, 1437, 668, 0.91)
print "File ", newarea
shutil.move(newarea, TempPath + "blah.png")
sleep(2)

#Check the captured view
m = find(TempPath + "blah.png")
if m:
  print m.getScore
else:
  print "epic fail"

Result:
Traceback (most recent call last):
  File "C:\DOCUME~1\shawn.lao\LOCALS~1\Temp\sikuli-tmp1538899642902305472.py", line 44, in <module>
    paging2()
  File "C:\Sikuli\project\bespin\scripts\common\paging2.sikuli\paging2.py", line 39, in paging2
    shutil.move(newarea, TempPath + "blah.png")
  File "C:\Program Files\Sikuli\sikuli-script.jar\Lib\shutil.py", line 197, in move
  File "C:\Program Files\Sikuli\sikuli-script.jar\Lib\shutil.py", line 197, in move
  File "C:\Program Files\Sikuli\sikuli-script.jar\Lib\os.py", line 351, in rename
 at java.io.File.<init>(Unknown Source)

 at sun.reflect.GeneratedConstructorAccessor2.newInstance(Unknown Source)

 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)

 at java.lang.reflect.Constructor.newInstance(Unknown Source)

 at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:163)

java.lang.NullPointerException: java.lang.NullPointerException

I'm wondering if there is a way to set the capture to 0.91? I.e. capture (Pattern(1, 171, 1437, 668).similar 0.91)

Cheers
Shawn

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

--- 1. the function capture as such only makes a screenshot of the specified area and puts the image as .png into a file. it is a 1-to-1 representation. It seems, that the capture does not check for exactly 4 parameters, since it does not error.

--- 2. similarity only makes sense, when you try to find something based on an image and it may be, that the searched visible object is not exactly the same as the captured probe (some different color, some shadow, ...). For this you have the class Pattern, that allows to specify a similarity for the search.
The standard similarity value is 0.7, which works in many cases. you will find, that many images will be found with a similarity (score) of above 0.9, since they are nearly exactly the same (mostly some differences in the background).
There is no "best" to use similarity (like your 0.91). This depends on your needs and situation. The only experience for the current version is that it is better to use 0.99 than 1.0 to get an exact match (seems to be a bug).

--- 3. when using shutil.move, the specified path has to exist (I doubt it does, since the error seams to be related to this operation).

--- 4. you write
m = find(TempPath + "blah.png")
if m:

if the image cannot be found on the screen at that moment, you will get a FindFailed error and the script stops. in this case you will not get to the if m:
So use exists() instead.

--- 5. my version

#Capture the view
newarea = capture (1, 171, 1437, 668)
print "File ", newarea
shutil.move(newarea, TempPath + "blah.png") # TempPath has to exist
sleep(2)

#Check the captured view, has to be visible on the screen.
m = exists(Pattern(TempPath + "blah.png").similar(0.91))
if m:
  print m.getScore
else:
  print "epic fail"

--- 6. comment/question
The area, you are trying to "compare" (something should change within 2 seconds, otherwise your snippet does not make sense) is rather big. What is your intention?

Revision history for this message
Shawn Lao (shawn-lao) said :
#2

Thanks for the quick feedback. As to my intention (I.e. --- 6. comment/question), I a checking the 3D renders on a web browsers and eventually mobile devices. Click on the following for further details:

http://www.youtube.com/watch?v=YJ3nFBUzmX0

I forgot to mention 'TempPath' is a variable. I.e. TempPath = "C:\\Sikuli\\images\\temp\\Compares\\"

I tried the exists() with similar(.0.91), but it returned errors. I executed a bunch of captures of the same size and got a variety of getScore values returned. So, I guess a '1 to 1 representation' or '0.99' similarity may be out of the question. Below are the values:

#=====================================================
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 2
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.90, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 3
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.90, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 4
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.89, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 5
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.89, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 6
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.89, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 7
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.88, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 8
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.87, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 9
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.84, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 10
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.84, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 11
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.84, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 12
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.84, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 13
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.85, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 14
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.85, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 15
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.85, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 16
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.86, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 17
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.87, target=center>
Click 2D coordinate F. Dead-center.
[sikuli] click on (699,495), MOD: 0
Page up to slice 18
<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668] score=0.89, target=center>
[sikuli] click on (488,155), MOD: 0

#=====================================================

I tried the following after a capture but got errors as well:
newarea = capture (1, 171, 1437, 668)
print "File ", newarea
shutil.move(newarea, TempPath + "blah1.png")
sleep(2)

m = exists(Pattern(CompPath + "blah1.png").similar(m.getScore))
if m:
 print "The similarity check passed."
 print m.getScore
else:
 print "Exists fail:"
 print m.getScore

Result:
    m = exists(Pattern(CompPath + "CHK_2d_CoursePageBC-TC.png").similar(m.getScore))
UnboundLocalError: local variable 'm' referenced before assignment

Any suggestions?

Cheers,

Revision history for this message
Shawn Lao (shawn-lao) said :
#3

oops, my bad...

'CompPath' should be 'TempPath'

Revision history for this message
RaiMan (raimund-hocke) said :
#4

Result:
    m = exists(Pattern(CompPath + "CHK_2d_CoursePageBC-TC.png").similar(m.getScore))
UnboundLocalError: local variable 'm' referenced before assignment

It means what it says: on the right side you have m.getScore, but m is not yet defined as local variable.

Tip: you have to say getScore(), to call the function. otherwise you address the function object as such. Print is so intelligent to print a string representation (<bound method edu.mit.csail.uid.Match.getScore of Match[1,171 1437x668]....).

With such a large area, you will never get a similarity near one (thats caused by the internal processing).

But I'm still not sure, what your problem is.

Revision history for this message
Shawn Lao (shawn-lao) said :
#5

If I will never get a similarity near one for such a big area then so be it. I'll be happy if Sikuli returns a getScore of '0.8' or greater. What do you suggest to be a good area to capture so I can get a similarity near one?

Here's my pseudo thought process to give you an insight on my problem:

1) Create a script that will do a 3D manipulation of a data set.

2) I will execute the script once and it will do a capture() for every render/change to the data and place it in an image library. This will save me time in doing a manual capture for each view.

3) Once the initial run is completed, the capture() functions will be commented out and a check function will be enabled to ensure the .PNG exists. However, I want the similarity to try to match as close to one as possible, the check will do the 'm = exists(Pattern(CompPath + "Image_That_Was-Originally_Captured.png").similar(0.91)). If the .PNG similarity does not exist, then it will do another screen capture and place it in a temp folder. After that I will call an in-house tool to do a difference compare between the original captured .PNG and the current .PNG.

4) So here's my problem, if the similarity of getScore is returning different similarity values the the exist() compare check will always fail since it does not match 0.91. If I cannot get the simlarity to 0.91, is there a way for me to do a check like the following?

m= exists(Pattern(CompPath + "Image_That_Was-Originally_Captured.png").similar(>=0.8))

Hope this helps.

Cheers

Revision history for this message
Best RaiMan (raimund-hocke) said :
#6

Your question:
possible? m= exists(Pattern(CompPath + "Image_That_Was-Originally_Captured.png").similar(>=0.8))
Yes, that is exactly, how it works:
m=exists(Pattern(some-picture).similarity(0.8))
means: it is a match, if the similarity is at least 0.8, otherwise FindFailed (m is set to None).

But I think you have to analyze your situation again:

Meanwhile I made some tests and found out, that even large areas can be found with a score of 1.00. This is true, if they are really equal pixel by pixel.

So it must be the case, that at time of capture some parts of your captured rectangle is slightly different from the situation at time of search. The sequence of find's you reported above, proves, that things are changing slightly over the time.

So in the consequence you have to analyze, what these differing/changing parts of your region are and try to leave them out. You can concentrate on parts that are more stable by breaking up your large image into a series of smaller ones.

May be it is possible, to say, if all sected parts are equal, the whole image is equal. If one part differs, then you make your screenshot of the whole scenario and use your compare tool.

Without seeing what really happens, that is all I can say.

Revision history for this message
Shawn Lao (shawn-lao) said :
#7

Thanks for the quick and detailed reply. I will keep playing with the capture range and timing to see how it goes. If not, I can live with the similarity of 0.80. :-)

Cheers,

Revision history for this message
Shawn Lao (shawn-lao) said :
#8

Thanks RaiMan, that solved my question.