I am not able to match my pattern precisely

Asked by Phi

I am trying to test if a button is enable or disabled. My similarity is set to .97. The matching is finding the button in its enabled and disabled state. Setting similarity to 1 does not find anything. I am using 1.0 rc2.

I have the images here http://img851.imageshack.us/g/fillbutton.png/

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

You captured the button in the enabled case.

It is found in disabled case with score 0.97 and in enabled case with score 1.0.

So the following approach differentiates between enabled and disabled.

r = Region(<the toolbar>) # see comment

img = "fillbutton.png"

m = r.exists(img)
print "tool = ", m # for debug only
if not m:
   print "tool not found"; exit(1)

if m.getScore() > 0.99:
 print "tool enabled"
else:
 print "tool disabled"

-- comment
For such mission critical decisions, it is vital, to restrict the search region as narrow as possible. In this case it should be the region occupied by the relevant part of the toolbar. Such a region can usually be evaluated from the application windows region:

r = App("your app").window().above(1).below(25).below(50)

from the upper border of app window (above(1)) we skip the title bar (below(25)) and select the tool bar (below(50)). The numbers might have to be adapted to your situation.

BTW: in my test, I did not have any problem to find the button in the enabled case with Pattern("fillbutton.png").similar(1)

Revision history for this message
Phi (iamphi) said :
#2

I can't get your results. Here is my test code and test results: http://img8.imageshack.us/i/toolbartestresult.png/
My sikuli project: http://dl.dropbox.com/u/344540/testToolbar.sikuli.zip
I am running the test on win7. I couldn't get it to run on os x 10.6.

My code:
[code]
fillEnableButtonImg = "fillButton.png"
fillEnableToolbarImg = "toolbarEnabled.png"
fillDisableToolbarImg = "toolbarDisabled.png"

def testMatch(fillToolbarImg, similarity):
 fillButtonPattern = Pattern(fillEnableButtonImg).similar(similarity)
 fillToolbarRegion = Region(find(Pattern(fillToolbarImg)))
 fillMatch = fillToolbarRegion.exists(fillButtonPattern)
 print similarity, " fillMatch ", fillMatch

print "enabled toolbar test"
testMatch(fillEnableToolbarImg, 1.0)
testMatch(fillEnableToolbarImg, 0.99)
testMatch(fillEnableToolbarImg, 0.97)
testMatch(fillEnableToolbarImg, 0.5)

print "disabled toolbar test"
testMatch(fillDisableToolbarImg, 1.0)
testMatch(fillDisableToolbarImg, 0.99)
testMatch(fillDisableToolbarImg, 0.97)
testMatch(fillDisableToolbarImg, 0.5)
[end code]

My results:
enabled toolbar test
1.0 fillMatch None
0.99 fillMatch Match[1163,518 31x17] score=1.00, target=center
0.97 fillMatch Match[1163,518 31x17] score=1.00, target=center
0.5 fillMatch Match[1163,518 31x17] score=1.00, target=center
disabled toolbar test
1.0 fillMatch None
0.99 fillMatch Match[1163,571 31x17] score=1.00, target=center
0.97 fillMatch Match[1163,571 31x17] score=1.00, target=center
0.5 fillMatch Match[1163,571 31x17] score=1.00, target=center

Revision history for this message
Phi (iamphi) said :
#3

My last post is kinda bogus. I know why the code doesn't work, but not how it was working for me to get that screenshot!

Revision history for this message
RaiMan (raimund-hocke) said :
#4

Ok, I made a test on my Mac (I guess with these things there is no difference between Mac and Windows), and I found the same results as you had. I cannot understand why I got the previous results.

I recaptured the button images from the contained toolbar images

[code]
fillButtonEnabled = "fillButtonEnabled.png"
fillButtonDisabled = "fillButtonDisabled.png"

def myFind(btn, tb):
 m = toolbars[tb].find(Pattern(btn))
 m.highlight(1)
 print "Found %s on %s as %s"%(btn, tb, str(m))

toolbarEnabled = "toolbarEnabled.png"
toolbarDisabled = "toolbarDisabled.png"

dir = getBundlePath()
import os

toolbars = {}
# find both buttons on both toolbars
for tb in (toolbarEnabled, toolbarDisabled):
 os.popen('open "'+os.path.join(dir, tb)+'"') # opens the toolbar image in Preview.app
 wait(2)
 toolbars[tb] = selectRegion("select " + tb)
 for btn in (fillButtonEnabled, fillButtonDisabled):
  myFind(btn, tb)
[end code]

My results:

[log] highlight Match[114,197 32x13] score=1,00, target=center for 1.0 secs
Found fillButtonEnabled.png on toolbarEnabled.png as Match[114,197 32x13] score=1,00, target=center

[log] highlight Match[115,197 32x13] score=1,00, target=center for 1.0 secs
Found fillButtonDisabled.png on toolbarEnabled.png as Match[115,197 32x13] score=1,00, target=center

[log] highlight Match[105,223 32x13] score=1,00, target=center for 1.0 secs
Found fillButtonEnabled.png on toolbarDisabled.png as Match[105,223 32x13] score=1,00, target=center

[log] highlight Match[106,223 32x13] score=1,00, target=center for 1.0 secs
Found fillButtonDisabled.png on toolbarDisabled.png as Match[106,223 32x13] score=1,00, target=center

If interested in my script: https://files.me.com/rhocke/s4wkeh

Conclusion:
With Sikuli's current search implementation, you do not have a chance to find differences in your scenarios between enabled and disabled.
The reason might be, that internally the first try is done with gray scaled versions of the image to be searched and the region where to search. In your case it seems that we get a 1.0-match in all cases (I guess the brightness of the pixels does not matter).
You have to look for another way, to find out the state of the toolbar.

I think we need an additional Pattern option that makes it possible to get different scores in such situations. I will hand this case over to the developers and turn this question to a request bug. I know that they are currently working on a further refinement and optimization of the search engine to be contained in the next version. May be it fits.

Can you help with this problem?

Provide an answer of your own, or ask Phi for more information if necessary.

To post a message you must log in.