SikuliX

[1.0.1] Java: Pattern(BufferedImage) is not found (silently does nothing) --- usage: load images from jar

Asked by joseytw

Hello,

I am a relative newcomer to Java coding and Sikuli both so please keep that in mind as you try to guide me on my way! Also, before coming here, I searched through existing questions on this site and many others, but have been unable to find something that explains a solution in a way that I understand.

Using Eclipse, I am attempting to write stand alone executable in Java that searches for images and clicks on them using org.sikuli.script classes but I cannot seem to get either an executable jar or exe version of my code to match images on-screen. In Eclipse, when I create a pattern using relative path names the script matches the image on-screen. For example, if I do:

Pattern imagePattern = new Pattern("src/Images/PizzleField.PNG");
or
Pattern imagePattern = new Pattern("bin/Images/PizzleField.PNG");

Then run my script within Eclipse, the script will find the above image without a problem. But, if I right click my project and Export as a runnable JAR file then attempt to run that JAR file, the image is no longer found. If I unzip the executable JAR, I can see that the Images directory with all of the images is there so the image are definitely being added to the JAR file. My research has led to look at using something along the lines of:

this.getClass().getClassLoader().getResource("Images/PizzleField.PNG");

to handle resources that are stored in a JAR file. Since it looks like I can create a Pattern using a BufferedImage, I have tried to use that to access and manipulate the images in my executable JAR file. Here is the code I am currently toying with to try and get this to work in Eclipse:

public boolean test() throws Exception {

  URL imageURL = this.getClass().getClassLoader().getResource("Images/PizzleField.PNG");
  String imageString = imageURL.toString();
  BufferedImage imageBuffered = ImageIO.read(imageURL);

  System.out.println("imageURL is: " + imageURL);
  System.out.println("imageString is: " + imageString);
  System.out.println("imageBuffered is: " + imageBuffered);

  Pattern imagePattern = new Pattern(imageBuffered); //("bin/Images/PizzleField.PNG");

  Screen newScreen = new Screen();
  float floatPercentage = convertIntMatchToFloatMatch(80);
  System.out.println("=====Searching for image: " + imageBuffered); //"bin/Images/PizzleField.PNG");
  if(newScreen.exists(imagePattern.similar(floatPercentage), 3) != null){
   System.out.println("+++++Found image : " + imagePattern);
  }
  else{
   System.out.println("ERROR: Could not find an onscreen match for the following image ----> " + imagePattern);
  }
}

When I run that code in Eclipse, the output I get is:

imageURL is: file:/C:/eclipseprojects/ffauto/ffauto/bin/Images/PizzleField.PNG
imageString is: file:/C:/eclipseprojects/ffauto/ffauto/bin/Images/PizzleField.PNG
imageBuffered is: BufferedImage@7646bb9f: type = 6 ColorModel: #pixelBits = 32 numComponents = 4 color space = java.awt.color.ICC_ColorSpace@1dc80063 transparency = 3 has alpha = true isAlphaPre = false ByteInterleavedRaster: width = 326 height = 22 #numDataElements 4 dataOff[0] = 3
=====Searching for image: BufferedImage@7646bb9f: type = 6 ColorModel: #pixelBits = 32 numComponents = 4 color space = java.awt.color.ICC_ColorSpace@1dc80063 transparency = 3 has alpha = true isAlphaPre = false ByteInterleavedRaster: width = 326 height = 22 #numDataElements 4 dataOff[0] = 3
ERROR: Could not find an onscreen match for the following image ----> P(-- BufferedImage --) S: 0.8

It looks like the imagePattern using the buffered image displays the contents of the image in the console log, but it never finds a match. If I change:

Pattern imagePattern = new Pattern(imageBuffered);
to
Pattern imagePattern = new Pattern("bin/Images/PizzleField.PNG");

so that the pattern is based on a String instead of a BufferedImage, then a match is found on-screen.

What am I doing wrong in the code above that is preventing my BufferedImage based Pattern to fail to find a match?

My Project Heirarchy is:

FFAuto
|
|-------src
| |
| |-------(default package)
| |
| |-------Images

Thanks in advance for any help. If any additional details are needed please let me know!

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
joseytw (joseytw) said : 		#1

Note: I have imported sikuli-java.jar as an external JAR and have been using the classes found in org.sikuli.script

Revision history for this message
RaiMan (raimund-hocke) said : 		#2

Sorry for the effort you have put in this excellent research (but might not have been in vain though ;-)

The usage of a BufferedImage to set up a Pattern already works, but the find operation using such a pattern silently simply does nothing in the moment, sorry.

I will implement it the next days and take your question as a request bug meanwhile.

There is no other solution available currently for images inside a jar, so your app would only work inside eclipse with regular image file names relative to the project folder.

If you are in a hurry with the jar containing the images:
Just write the buffered image to a file outside the jar and use this filename (might always be the same for find operations not running in parallel - adds some 10 milliseconds). Hope you get the solution before going into production with your approach ;-)

Revision history for this message
joseytw (joseytw) said : 		#3

Thank you for your prompt reply - I really appreciate you taking the time to read and respond to my post! After reading your response I just want to make sure I am clear on what it means. When you say:

“There is no other solution available currently for images inside a jar, so your app would only work inside eclipse with regular image file names relative to the project folder.”

It pretty much means that if I export my code as a runnable JAR that has images saved inside of it, that I will not be able to get Sikuli to use the images stored in that JAR file to match an image on-screen even if I use something like ‘this.getClass().getClassLoader().getResource(resouce)’ or try to point Sikulu to the images inside the JAR file using relative paths.

As I understand it, my 2 options are:
1) Keep my images outside of the JAR file and point Sikuli at those using either String Patterns or BufferedImage patterns
2) Patiently wait for the BufferedImage fix to be implemented (I would be happy to wait by the way!)

Is this correct?

(Since I have spent so much time trying to get this to work I want to be very clear on what my options are moving forward. I know you are probably busy with a million things and I would like to thank you again for taking the time to help clarify things for me!)

Revision history for this message
RaiMan (raimund-hocke) said : 		#4

at 1: currently only Pattern objects defined with image filenames as strings can be used for searching. Buffered images have first to be written to real files in the filesystem (which you could do in this case, when working with the jar).
find(Pattern(BufferedImage)) will not work in any way.

at 2: I guess a solution will be available towards end of next week and in between I would be glad if I could contact you directly for some short tests. If you agree, just send me a mail silently to my mail at https://launchpad.net/~raimund-hocke.
No problem if you do not want, do not feel beholden.

Revision history for this message
joseytw (joseytw) said : 		#5

What kind of crazy talk is that?

You have been kind enough to answer my questions and the least I could do is return the favor by helping test a solution for this. I would be happy to help and will send an email shortly!

Thanks again RaiMan!

Revision history for this message
RaiMan (raimund-hocke) said : 		#6

is a request bug now

Revision history for this message
Deses (deses12) said : 		#7

Wow, I was going to open a new question for exactly this problem, I'm glad joseytw did it first.

I could do some more testing if you want, I want this feature so badly. :)

Revision history for this message
RaiMan (raimund-hocke) said : 		#8

@Deses
pls go to the related bug and subscribe to it.
I have already implemented getting images from a jar and use them with find() as buffered images and other features that internally use find() (findAll, observe and some minor aspects have to be adapted still).
The usage is rather convenient, since with one setImagePath() you can say, which jar on the classpath to use by just giving a classname and for the cases you do not run from the jar (e.g. when running/debugging in the IDE) you can give an alternate file path to load the images from.
so a find("images/some_image.png") will work in both cases.

I am a little bit behind, but hope I can give you something to test latest on monday.

Revision history for this message
madhu (madhu-14am) said : 		#9

hi all,

I'm very new to sikuli. I have automated one project using sikuli and java. Now i want to convert it to jar file .In eclipse i right click on the project and export as jar file and jar file is created. whenever i double click on it images are not recognized . So any solution? Please advise?

Revision history for this message
bal krishna gupta (bkg-krishna) said : 		#10

I am still facing this issue. Any help for this will be highly appreciated.