SikuliX

How to handle when there are high similarity scores for enabled and disabled button Patterns?

Asked by Andrew C

Hello,

I am trying to use sikuli to operate a flash player application. I am coding in Java using the sikuli-script.jar from the SikuliX-1.0.1 release.

In this application there is a button that has two states - enabled and disabled. My approach is to wait for the button to turn from disabled to enabled so I can click on it.

My code logic follows (in pseudo code)
if disabled button exists on screen
{
waitvanish disabled button //wait for the disabled button to disappear
}
wait for enabled button //confirm enabled button exists on screen.

My problem is that the similarity score for my image of the disabled button is extremely high. The score of my collected image of the disabled button to the enabled button on screen was 0.987. (The score of the collected image of the enabled button to the enabled button on screen was 1.0). In person you can see a difference in the two collected images. The shading is darker on the disabled button. The enabled button has a pink/red coloured ring and the disabled button is more burgundy coloured ring.

I have tried setting the minimum similarity to 0.99, but then the rest of the program falls over.

What else can I do have the enabled and disabled button become more distinguished? Is my code logic okay?

thanks in advance
Andrew

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
obiwan-92 (obiwan-92) said : 		#1

Hello,

That's strange.
Normally the color is not relevant during the comparison except if you put the similarity to the maximum value.

Can you zip and post the two image of your button ?

Regards.

Revision history for this message
Best RaiMan (raimund-hocke) said : 		#2

Distinguishing between two button states is one of the more tricky situations in Sikuli.

Generally you have to have optimised images of both button states (as little background as possible, concentrating on the unique aspects).
You have to be sure, that both images match their button state with a score of 1.0 and have a similar size.

for both buttons you now have to setup Patterns with similarity 0.99 (in IDE preview) or as
Pattern btnEnabled = new Pattern("btnEnabled.png").exact();
Pattern btnDisabled = new Pattern("btnDisabled.png").exact();

(exact() is the same as similar(0.99) and works correctly in version 1.0.1+)

In your case, this should work:

Screen s = new Screen();
if (s.exists(btnDisabled)) {
  s.getLastMatch().grow(10).wait(btnEnabled);
} else {
  s.wait(btnEnabled);
}

grow(10) assures, that there is some margin around the match region, so different image sizes are compensated.
I guess you know: wait() will give up after 3 seconds ;-)

This could be optimised, when it is possible to work in the region of the button from the beginning.

Revision history for this message
Andrew C (jndrew) said : 		#3

Hello,

This is the first time that I have used Launchpad. I hope this zip file
reaches you.

cheers,
Andrew

On Fri, Jan 17, 2014 at 3:56 PM, obiwan-92 <
<email address hidden>> wrote:

> Your question #242340 on Sikuli changed:
> https://answers.launchpad.net/sikuli/+question/242340
>
> Status: Open => Answered
>
> obiwan-92 proposed the following answer:
> Hello,
>
> That's strange.
> Normally the color is not relevant during the comparison except if you put
> the similarity to the maximum value.
>
> Can you zip and post the two image of your button ?
>
> Regards.
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/sikuli/+question/242340/+confirm?answer_id=0
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/sikuli/+question/242340
>
> You received this question notification because you asked the question.
>

Revision history for this message
Andrew C (jndrew) said : 		#4

Thanks for your responses. I'll give that a try sometime.

Since I posted that question, I did some further investigation and found that the score of the enabled button to the enabled button on screen would range from 1.0 to 0.999998 over time while the application was run.

To make some progress, I used the lower value as a threshold to polled the score of the button to determine whether button state had changed.

ie
while (time has not run out)
{
if current button scope >= threshold score //check is current score is in the range of the enabled button, exit loop if is
break
else
thread.sleep(2000); //sleep for 2 seconds
}

Not sure whether this is the best or most robust approach. Any thoughts?

Revision history for this message
Andrew C (jndrew) said : 		#5

Thanks RaiMan, that solved my question.

Revision history for this message
RaiMan (raimund-hocke) said : 		#6

A solution is something that works with respect to the objectives ;-)
But there are many aspects of the solution, that can be discussed.

My solution is a bit more compact and delegates the waiting, comparing and looping down to Sikuli.

Not knowing your concrete situation, I dare to say, that

Region reg = <some region containing the button>
reg.wait(btnEnabled, maxTime)

would do the job anyway:
If the button is already enabled, It would come back immediately.
Otherwise it would wait maxTime for the button to get enabled.

To directly support some workflow:

if (!reg.exists(btnEnabled, maxTime)) {
  // do something sensible if button does not get enabled within the given time
} else {
  click(reg.getLastMatch()
}