Sikuli doesn't find image on screen but does in image explorer.

Asked by hydrochaeris

Hello !

You can also find my question on stackoverflow : https://stackoverflow.com/questions/69913012/sikuli-on-java-does-not-recognize-images

I am new to Sikuli, and wanted to use it to automatize some tasks on my computer. I use Java in my every-day job, so I am using Sikuli in a java program (+maven + eclipse). I am using windows 10, no VM, two monitors (I tried to unplug the 2nd one, but it doesn't fix the problem).

The maven dependency I use is the following :

<dependency>
    <groupId>com.sikulix</groupId>
    <artifactId>sikulixapi</artifactId>
    <version>2.0.5</version>
</dependency>

I tried to do some simple stuff. I did a few screenshots of parts of my screen using windows' screenshot tool, and wanted sikuli to hover it. It works quite fine for one image, but not at all for the others. It seems that the bigger the image the better it works as I did not have success for any small images. The one working is a screen of a whole window (reduced to ~1/4 of my screen). I also tried to find a button inside this window, to find the windows logo on bottom left, to find a screen of my package explorer, but none work correctly.

I played with similar() using various values, but it didn't improve the results. In some cases (button inside the window) it did find a result for some low similar value, but it was another button. The weird part is : its finding this other button which is bright blue, while the one i'm looking for is purple.

My pc background never changes, I did some screen.highlight() and its looking at the correct screen (dual screen). It's not an issue with the path to images (already solved this one).

Do you have any idea of what I could try ? I have read about people having different success rate depending on whether they were using Sikuli IDE or another IDE. So maybe I could give sikuli IDE a try.

The code I'm using to test :

public class CleanTest {

    static Screen screen = new Screen();

    public static void main(String[] args) throws FindFailed, AWTException, IOException, InterruptedException {
        String pathYourSystem = System.getProperty("user.dir") + "\\";
        Pattern pLauncher = new Pattern(pathYourSystem+"img\\full_launcher.PNG").similar(0.9d);
        Desktop.getDesktop().open(new File("path_to_an_exe_opening_a_launcher"));
        screen.wait(pLauncher, 300);
        screen.mouseMove();
        System.out.println("launcher found");
    }

}

It works with the "full launcher" image, but it doesn't find a sub-part of the launcher (a button). I tried to make some code to test if there was some threshold for the similar parameter :

        double similarValue = 1d;
        Pattern pLauncher = new Pattern(pathYourSystem+"img\\the_button.PNG").similar(similarValue);
        Desktop.getDesktop().open(new File("path_to_an_exe_opening_a_launcher"));

        while(!screen.has(pLauncher)) {
            similarValue-=0.1;
            pLauncher = pLauncher.similar(similarValue);
        }
        System.out.println(similarValue);
        screen.mouseMove();

I also did a bit more testing: if i take a screenshot of a part of my IDE then run a simple

while(!screen.has(<myScreenPath>)){}
 screen.getLastMatch().highlight(2);

It doesn't highlight anything. That's very weird. I tried to unplug my second monitor and to use different tools to screenshot, without success.
What I did: take a screenshot of a constant part of my screen (menu icons of eclipse, for example). Save it. Launch the previous code. Nothing get highlighted. Then without stopping the execution, I open the image file. It does find it and highlight it.

Thanks for your help.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
Best RaiMan (raimund-hocke) said :
#1

Generally it is not recommended, to use the Windows screenshot tool to get your images.

Use the SikuliX IDE to get your images.
Simply make a script containing images only and store it to a folder.

Having a line starting with
image1 =

and then using the Screenshot button automatically names the image as image1.png.

Additionally with the IDE Preview feature you can check and rename your images and create Pattern settings.

.. and last but not least you can make short tests with just a few commands, before doing the more complex things with Java.

... and one more thing: since it is Jython, that is used to run scripts in the SikuliX IDE, you can use plain Java coding, if the things needed besides Java are on the class path.

Revision history for this message
hydrochaeris (hydrochaeris) said :
#2

Thank you for your answer.

I will give SikuliX IDE a try.

If I take screenshots using the IDE, can I save it on my PC ? If I do, I will try to use the IDE to take the screenshots then use them in my current java project. If it doesn't fix the problem I may try to fully develop my project on SikuliX IDE.

I will let you know how it went.

Revision history for this message
RaiMan (raimund-hocke) said :
#3

With my advice from comment #1 you can name your imagefiles individually and store them whereever you want.

If you have solved this image issue, then there is no need to not use Java programming.

Scripting in the IDE (depending on complexity of workflow) might be easier, but there is no easy debugging like in Eclipse with Java.

All the best.

Revision history for this message
hydrochaeris (hydrochaeris) said :
#4

I tried to screenshot from sikulix IDE, and use these images in my java project. The few images I have tested worked correctly, thank you!
Is there any way to change the default save folder for screenshots taken from Sikulix IDE?

Thanks again

Revision history for this message
hydrochaeris (hydrochaeris) said :
#5

Thanks RaiMan, that solved my question.

Revision history for this message
RaiMan (raimund-hocke) said :
#6

--- Is there any way to change the default save folder for screenshots taken from Sikulix IDE?

Currently this is the only way:

- open a new script (new tab)
- select File -> Save as
- in the save dialog select the Fileformat "as plain folder"
- select the folder, where you want to have your image folder
- name your image folder
- klick save

Now you can use this folder (in fact it is a script folder with a .py file, which must be there and have the same name as the containing folder!) inside and outside the IDE.

... and you can use the features as mentioned in comment #1

In version 2.0.6 I will add a feature, that allows to just have an IDE tab, that just represents an image folder (hence not being a script folder)