Sikuli with Selenium Java

The API you are using is not SikuliX.

The imports seem to reference some RC3 stuff (sikuli-script.jar ??).

Version 1.0.1 does not have:
org.sikuli.script.internal.hotkey.*;
org.sikuli.script.natives.*;

the click function for RC3 reads
scrn.click(some_image)

... so what should click(some_image, 100) mean?

SikuliX has OCR based on Tesseract with the usage:
String text = some_region.text()

but I think the calculator numbers are not OCR-readable.

SikuliX does not know a feature:
deskIcon.getProperty("Calc_Scrn").contains("200")

Taking your systematic, the deskIcon.getProperty("Calc_Scrn") should return an image-file-name-string, that I doubt contains "200", so this should always be false.

So where did you get your "Sikuli-API-ideas" ???

As I have mentioned in the description "deskIcon" is the location where I have kept all the images.

I am able to click on Calculator's button using my code. click(some_image, 100) means that "click on the image and wait for 100 milliseconds before next click.

By using deskIcon.getProperty("Calc_Scrn").contains("200") in If condition I want to validate that "IF THE "Calc_Scrn" (image) CONTAINS VALUE 200 THEN PRINT PASS OR PRINT FAIL."

Actually, I don't know how to get and validate the value in "Calc_Scrn" (image).

For more specification I am trying the below program:

Open Calculator -> Click on 1 -> Click on 0 -> Click on 0 -> Click on + -> Click on 1 -> Click on 0 -> Click on 0 -> Click on = (The answer appears is 200. Now If the value in answer text box is 200 then print PASS else print FAIL.

Thanks.

what you wanted to do is clear - its somehow obvious.

... but you did not answer the question:
So where did you get your "Sikuli-API-ideas" ???

a feature
click(image, waittime)

does not exist in SikuliX.

So what version of Sikuli are you using?

i installed sikuli 1.6

i have added sikuli .jar api to my selenium library and then i wrote this code.

does that helped u?

if u want we can have a screeshare session. just mail it to me so that you will get more idea on it.

what is sikuli 1.6 ????

I have downloaded "Sikuli-r930-win32" from Sikuli website. And then added .jar files from downloaded version to my selenium project. So that I can code using Selenium JAVA.

You can refer below link for example:

http://doc.sikuli.org/faq/030-java-dev.html

Hi RaiMan, any update or solution to my problem?

still you did not answer my questions in comment #3

You have to use the Sikuli API correctly.

For your question where did you get your "Sikuli-API-ideas" ??? I have implemented it as per its mentioned on http://doc.sikuli.org/faq/030-java-dev.html

So what version of Sikuli are you using? I have downloaded and extracted "Sikuli-r930-win32" and then added .jar files to my Selenium Library.

I thinks these are the proper answer I have.

repeating things does not make it clearer.

the mentioned link shoes how to access sikuli-script.jar in a Java program, but does not talk about specific API features like yours:

click(image, wait time)

and

 if (deskIcon.getProperty("Calc_Scrn").contains("200")){

So where did you get the idea, that this should work as you expect?

Did you ever look into the docs?

On the other hand I strongly recommend, to use version 1.0.1+ if you are planning to use Sikuli for more than some playground actions.

Start here: http://www.sikuli.org/download.html
and read carefully, before doing anything.
This is important: https://github.com/RaiMan/SikuliX-API/wiki/Usage-in-Java-programming

Thanks, I will go through it.

Can you tell me the solution for which I am trying for "How to verify the value (200) from calculator." I am pursuing it from more that 15 days.

Please give your solution or code for the issue which I have documented initially.

 Thanks in advance.

SikuliX has the OCR feature Region.text()
see docs

about problems and advices you find many entries here.

I have downloaded Sikuli X from the link you have provided but after installing it I didn't find sikuli-java.jar in the folder. I have already added sikuli-script.jar in the class path of the project.

I tried with OCR Region.text() and could not find the any solution. As this is one of the first script I am writing using Sikuli and Java, please help me with this.

I know I have taken much time of yours but this will be very helpful for me in future. As lot of script writing depends on verification and validations using IF conditions.

I am unable to import "import org.sikuli.basics.SikuliXforJython;" in eclipse.

so you did not have run setup accordingly.

If you want the IDE + sikuli-java.jar select option 3
If you only want only sikuli-java.jar select option 4

This is clearly mentioned in the setup dialog and behind the help buttons.

sikuli-script.jar is from version RC3

if you use 1.0.1, it is sikuli-java .jar (as clearly mentioned in the above link comment #11 This is important: …
… and you should follow the advice exactly.

… and if you decide to try 1.0.1, purge anything from your system and environment, that belongs to RC3.

Hi RaiMan,

I was able to follow the instruction properly and I got the below error when i tried to execute the script:

[error] ResourceLoaderBasic: copyResource Not possible: C:\Program Files\Sikuli X\libs\jawt.dll (Access is denied)
[error] ResourceLoaderBasic: check: libs dir is empty, has wrong content or is outdated
[action] ResourceLoaderBasic: check: Please wait! Trying to extract libs to: C:/Program Files/Sikuli X/libs
[error] ResourceLoaderBasic: check: Fatal Error 102: not possible to empty libs dir
[error] Terminating SikuliX after a fatal error(102)! Sorry, but it makes no sense to continue!
If you do not have any idea about the error cause or solution, run again
with a Debug level of 3. You might paste the output to the Q&A board.

As clearly mentioned in the docs and infos about how to setup 1.0.1+:

Do not use a folder in the program folder. Put the Sikuli stuff into a folder, that is fully user writable.
Do not use blanks in the folder name.

recommended:
c:\SikuliX\

Your suggestions are helping but I am still not able to read the value.

I have written below code to read the value:

if (scrn.find(deskIcon.getProperty("Calc_Scrn")).text().equals("200")){
   //scrn.find(deskIcon.getProperty("Calc_Scrn")).inside().text().contains("200")
   System.out.println("Pass");
  }
  else{
   System.out.println("Fail");

But when I Watch scrn.find(deskIcon.getProperty("Calc_Scrn")).text() It says --no text--

And when I trie dto watch scrn.find(deskIcon.getProperty("Calc_Scrn")).text().equals("200") It says False.

This is the only propblem I am facing now.

Please help me with this.

I successfully ran this script on Windows 7 with the Calculator app and got finally printed
Pass: 200

imgRechner = Pattern("titleOn.png").exact().targetOffset(-38,38) # window title activated
if not exists(imgRechner, 0):
    click(Pattern("titleOff.png").exact()) # window title not activated
find(imgRechner)
regResult = Region.create(getLastMatch().getTarget(), 235, 20) # get the result field
click(Pattern("btnClearAll.png").exact())
regResult.highlight(2)
click(Pattern("btn4.png").exact())
click(Pattern("btnMult.png").exact())
click(Pattern("btn5.png").exact())
click(Pattern("btn0.png").exact())
click(Pattern("btnResult.png").exact())
Settings.OcrTextRead = True
wait(1)
txtResult = result.text()
if "200" in txtResult:
    print "Pass:", txtResult
else:
    print "Fail"

ok, you have to convert this to Java ;-)

Well that's the problem......... I have to convert it in JAVA and I am stuck with IF condition. :-(

ok, in Java:

Screen s = new Screen();
Pattern imgRechner = new Pattern("titleOn.png").exact().targetOffset(-38,38); // window title activated
if null = s.exists(imgRechner, 0) {
    s.click(Pattern("titleOff.png").exact()); // window title not activated
}
s.find(imgRechner);
Region regResult = Region.create(s.getLastMatch().getTarget(), 235, 20); // get the result field
s.click(new Pattern("btnClearAll.png").exact());
regResult.highlight(2);
s.click(new Pattern("btn4.png").exact());
s.click(new Pattern("btnMult.png").exact());
s.click(new Pattern("btn5.png").exact());
s.click(new Pattern("btn0.png").exact());
s.click(new Pattern("btnResult.png").exact());
Settings.OcrTextRead = true;
s.wait(1);
String txtResult = result.text();
if txtResult.contains("200")
    System.out.println "Pass: " + txtResult;
else
    System.out.println "Fail";

Hi RaiMan,

Thanks for the code. I am able to execute the code successfully after making some changes. However, I am getting error message.

Below is the code I am executing:

public class SikuliTest {

 public static void main(String[] args) throws IOException, FindFailed, InterruptedException {
  Properties deskIcon = new Properties();
  deskIcon.load(new FileInputStream("D:\\SeleniumPractice\\WebandDesk\\OR\\PropertyFiles\\DeskIcons.properties"));

  App.open("calc.exe");
  Screen s = new Screen();

  Pattern imgRechner = new Pattern(deskIcon.getProperty("titleCalc")).exact().targetOffset(-33, -1); // window title activated
  if (s.exists(imgRechner, 0) == null) {
      s.click(new Pattern (deskIcon.getProperty("titleCalc")).exact()); // window title not activated
  }
  s.find(imgRechner);

  Region regResult = Region.create(s.getLastMatch().getTarget(), 69, -77); // get the result field
  regResult.highlight(2);

  s.click(deskIcon.getProperty("Calc1"));
  s.click(deskIcon.getProperty("Calc0"));
  s.click(deskIcon.getProperty("Calc0"));

  s.click(deskIcon.getProperty("CalcPlus"));

  s.click(deskIcon.getProperty("Calc1"));
  s.click(deskIcon.getProperty("Calc0"));
  s.click(deskIcon.getProperty("Calc0"));

  s.click(deskIcon.getProperty("CalcEqul"));

  Settings.OcrTextRead = true;

  String txtResult = regResult.text();
  if (txtResult.contains("200"))
   System.out.println("Pass: " + txtResult);
  else
      System.out.println("Fail");
  }
}

And I am getting below error message:

[log] App.open calc.exe(5168)
[log] CLICK on L(899,293)@S(0)[0,0 1152x864]
[error] Region(866,292,69,-77) outside any screen - subsequent actions might not work as expected
[log] highlight R[866,292 69x-77]@Screen null E:Y, T:3.0 for 2.0 secs
[error] Region(863,289,75,-71) outside any screen - subsequent actions might not work as expected
Exception in thread "main" java.lang.NullPointerException
 at org.sikuli.script.ScreenHighlighter.captureScreen(ScreenHighlighter.java:156)
 at org.sikuli.script.ScreenHighlighter.highlight(ScreenHighlighter.java:133)
 at org.sikuli.script.ScreenHighlighter.highlight(ScreenHighlighter.java:143)
 at org.sikuli.script.Region.highlight(Region.java:1490)
 at org.sikuli.script.Region.highlight(Region.java:1504)
 at SikuliTest.main(SikuliTest.java:29)

Please help with this NullPointerExecption for highlight().

Region regResult = Region.create(s.getLastMatch().getTarget(), 69, -77); // get the result field

not really clear what you want to say with a negative height??

 Region.create(s.getLastMatch().getTarget(), 69, -77)

corresponds to
 Region.create(someLocation, width, height)

where you have:
someLocation = s.getLastMatch().getTarget()
is valid and gets the top left corner of the new region

with a width of 69 and a
height of -77 ????????

Ok!! 69 and -77 is x and y for target :-(

I dont know how to calculate or get the width and height.

Any help to calculate height and width of image.

Hey got the height and width but I am getting error as "[error] TextRecognizer: init: tessdata folder not found at C:/SikuliX/libs/" Hence, String txtResult = regResult.text(); is showing "No Text"

Please help

… height and width of image

of what image?

you have
Match m = s.getLastMatch()

and m is a Region that has m.w and m.h which corresponds to the width and height of the image that was use with the last search op in s:
 s.find(imgRechner);

hence the respective image is imgRechner.

ok.

Did you run setup with the text option?

If you did, then just delete the libs folder once and start all over with your test - the libs folder will be recreated now hopefully ;-) with the tessdata folder.

I am still getting an error message as:

[error] TextRecognizer: init: tessdata folder not found at C:/SikuliX/libs/
[error] Region.text: text recognition is now switched off

Please help

I deleted libs folder and ran the test but same error again :-(

then your sikuli-java.jar does not contain the tessdata folder (not selected at setup).

So you have to run setup again:
- delete everything in the folder containing sikulix-setup.jar except sikulix-setup.jar
- run setup again
- select options 4 and 5
- after successful setup (now have sikuli-java.jar) delete the libs folder
- proceed with your work

Yes it helped and the script executed properly.

Now the only problem is regResult.text() is showing "--No Text--"

Hence, it goes to 'else' in below If condition

  String txtResult = regResult.text();
  if (txtResult.contains("200"))
   System.out.println("Pass: " + txtResult);
  else
      System.out.println("Fail");
  }

If you want to see the complete code then below is the code.

public class SikuliTest {

 public static void main(String[] args) throws IOException, FindFailed, InterruptedException {
  Properties deskIcon = new Properties();
  deskIcon.load(new FileInputStream("D:\\SeleniumPractice\\WebandDesk\\OR\\PropertyFiles\\DeskIcons.properties"));

  App.open("calc.exe");
  Screen s = new Screen();

  Match m = s.getLastMatch();

  Pattern imgRechner = new Pattern(deskIcon.getProperty("titleCalc")).exact().targetOffset(-33, -1); // window title activated
  if (s.exists(imgRechner, 0) == null) {
      s.click(new Pattern (deskIcon.getProperty("titleCalc")).exact()); // window title not activated
  }
  s.find(imgRechner);

  Region regResult = Region.create(s.getLastMatch().getTarget(), 102, 16); // get the result field
  regResult.highlight(2);

  s.click(deskIcon.getProperty("Calc1"));
  s.click(deskIcon.getProperty("Calc0"));
  s.click(deskIcon.getProperty("Calc0"));

  s.click(deskIcon.getProperty("CalcPlus"));

  s.click(deskIcon.getProperty("Calc1"));
  s.click(deskIcon.getProperty("Calc0"));
  s.click(deskIcon.getProperty("Calc0"));

  s.click(deskIcon.getProperty("CalcEqul"));

  Settings.OcrTextRead = true;

  String txtResult = regResult.text();
  if (txtResult.contains("200"))
   System.out.println("Pass: " + txtResult);
  else
      System.out.println("Fail");
  }
}

This is the only issue I am facing now. Thanks in advance.

Found out the cause, actually it is not finding and highlighting the right side of calculator screen (where we get the numbers) that's why it is showing as "--No Text--".

I tried lots of steps but unable to highlight the specific area of calculator's screen.

… but the --no text-- returned value means, that you still do not have your text feature ready.

at the beginning of your code put:

org.sikuli.basics.Debug.setDebugLevel(3)

and check the log output.

You may send the log output silently to my mail at https://launchpad.net/~raimund-hocke

I found the issue and my issue is resolved :-)

Thanks for your help Raiman..... You are very helpful and instant responces

Thanks RaiMan, that solved my question.