Action (Click/Type) on similar elements on a screen

Asked by Vin Uppinkudru on 2018-01-22

I am newbie to Sikulix and I am trying to automate an application in Java Sikulix. Apologies if similar question has been answered in the past. I am feeling that the documentation is difficult to follow for an absolute beginner. Hence asking you all experts.

The form I am trying to automate has a number of text boxes (like name etc) and each box has a label. I tried capturing each of the box with its label. This approach is not working as the texts box patterns appears to be matching with each other.

Could you please explain the solution with detailed explanation and full code sample in Java.

Appreciate you help and looking forward to hear from you all. Thanks

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
Vin Uppinkudru
Solved:
2018-01-24
Last query:
2018-01-24
Last reply:
2018-01-23
RaiMan (raimund-hocke) said : #1

read and understand:
http://sikulix-2014.readthedocs.io/en/latest/basicinfo.html#sikulix-how-does-it-find-images-on-the-screen

This should tell you something about similarity score.

Then understand the class Pattern, that combines an image with a wanted similarity.
Use the SikuliX IDE to check your images about similarity score or do some logging in your Java.

The docs are mainly to get a feeling about the features and information on how to specify.

There are tons of examples in the net.

--- I tried capturing each of the box with its label
only capture the labels with as little background as possible and use Pattern with target offset.

Vin Uppinkudru (neouppin) said : #2

Hi RaiMan,

Thank you so much for you help. I was able to use the Target Offset on the pattern and i was able to achieve the goal.

Below is the snippet i used

         Step1 : Pattern p1 = new Pattern("label.PNG");

        Step2: Pattern p2= new Pattern(p1).targetOffset(100,0);

        Step3: Compare the image with pattern p2 matches to a desired image

        Step3: screen.type(p2, "text");

One thing i would like to improvise is to implement Step3 where the i would like match pattern at the target offset with a desired image.

RaiMan (raimund-hocke) said : #3

after step 3:

// get the match from step 3
Match matchP2 = screen.getLastMatch()

//create a suitable region where to look for the "desired image"
regionP2 = matchP2.grow(left, right, top, bottom)

// look for the image
match desiredMatch = regionP2.exists("desiredImageOrPattern", 0)
if (null != desiredMatch) .... ; // found
else .... ; // not found

for exists and grow see the docs.

Vin Uppinkudru (neouppin) said : #4

Thanks RaiMan.