Help to click on image based on text info observed

Asked by Saravanakumar

Hello team,

I am working on a web based app. that has applet within it. I am using Selenium Java and Sikuli for automation testing. In the web page that contain applet, the UI is depicted as shown below (I have depicted this as text):

Idle Run_Icon Edit_Icon Info_Icon Cross_Icon
Active Run_Icon Edit_Icon Info_Icon Cross_Icon

NOTE: The *_Icon are images that are rendered on Applet. This is in PNG format.

I am able to find Run_Icon using following technique using Sikuli:
matches = s.findAll("img/ESRSrun.PNG");
if(m.exists("img/ESRSrun.PNG") != null){
        if(m.getScore() > 0.9){
                     m.click();
              System.out.println("clicked active state...");
 }
}

I am able to find text called using Match textMatch = s.find("Idle");
My requirement is to click on the Run_Icon only when the text reads Idle as text.

Question : How to I measure and move the mouse cursor towards right of text called "Idle" and click on "Run_Icon"

Please share your inputs and ideas.
Thank you
Saravanakumar

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

what version of Sikuli?

I strongly recommend to switch to 1.1.0 (http://sikulix.com)

Revision history for this message
RaiMan (raimund-hocke) said :
#2

this works with version 1.1.0 using this web page:

top = "top.png" # image of the title "Question information"
reg = find(top).below()
#reg.highlight(2)
icon = "icon.png" # image of the yellow icons
mIcons = reg.findAllByRow(Pattern(icon).similar(0.95))
for mIcon in mIcons:
  print mIcon
  mIcon.hover()

label = reg.findText("English")
#label.highlight(2)
label.rightAt(15).hover()
wait(2)

Revision history for this message
Best RaiMan (raimund-hocke) said :
#3

sorry, you have to transscript this to Java code.

Revision history for this message
Saravanakumar (murugesan-saravanakumar) said :
#4

Thank you RaiMan. I will try your solutions and get back to you.

Revision history for this message
Saravanakumar (murugesan-saravanakumar) said :
#5

As per your advice, I am now moved my Sikuli installation to version 1.1

Revision history for this message
Saravanakumar (murugesan-saravanakumar) said :
#6

Dear Raiman,

Good Day. I trust this message should bring smiles. Thank you for helping me out with you code blocks and directions to solve my problem. Below is my code snippet written in Java _ Sikuli1.1X

 public int MonitorAppletScreen(){
  int retryCount = 1;
  int status = 0; // 0 means failure; 1 means success
  Settings.OcrTextRead = true;
  Settings.OcrTextSearch = true;
  int outerLoop = 0;

  //System.out.println("Inside MonitorAppletScreen method");
  SetMouseCursorDefaultLocation();
  try{
   appletScreen = new Screen();
   int x = (appletScreen.w) / 2;
   int y = (appletScreen.h) / 2;
   loc = new Location(x, y);
   appletScreen.hover(loc);
   appletScreen.wait(1.0);
   //while(outerLoop < 10){
    //SK:: ROW WISE - START
    while(retryCount <= 2){
     if(retryCount == 1){
      m = appletScreen.findText("Idle");
      m.hover();
      r = m.add(0, 361, 15, 10);
      img = r.find("img/ESRSrun.PNG");
      appletScreen.wait(1.0);
      img.click();
      appletScreen.wait(1.0);
     }else{
      m = m.below().findText("Idle");
      m.hover();

      r = m.add(0, 361, 15, 10);
      img = r.find("img/ESRSrun.PNG");
      appletScreen.wait(1.0);
      img.click();
      appletScreen.wait(1.0);
     }
     Thread.sleep(1500);
     retryCount = retryCount + 1;
    }
    //SK:: ROW WISE - END
    retryCount = 1;
    outerLoop = outerLoop + 1;
    Reset();
    appletScreen = new Screen();
    loc = new Location(60, 60);
    appletScreen.hover(loc);
    appletScreen.wait(1.0);
   //}
  }catch(Exception ex){
   status = 0;
   //ex.printStackTrace();
   //System.out.println(".........................................................................");
   //System.out.println("Exception occurred in MonitorAppletScreen method" + ex.getMessage());
  }

  return (status);
 }

BTW - The FindAllText was not working with current version. Hence I found a way out using positional (below() method) mechanisms.

Thank you so much for your support.
Saravanakumar