Sikuli - Image Scaling and Comparison Issue

Asked by Rishikesh on 2018-10-17

Hi RaiMan,

Firstly, really appreciate your work and dedication.

I am new to Sikuli. I have had numerous problems in working with Sikuli but I was able to fix most issues by referring to threads on this forum.

I have a problem that I am unable to fix now and need your expert advise.

I am developing a project using Sikuli and Java at work. I work at Dell. We need to basically install and test drivers for new Dell machines and I am automating this procedure.

for any file during installation, we have a sequential set of buttons - Install -> Next (can appear multiple times during UI) -> Finish -> Close, in this same order. We have taken the images of these buttons and passed it to an array and it will compare each screenshot within the said region and perform the click.

The script works fine when I use it on the developing system. However, when I make a runnable Jar and test it on another machine, it does not find the image. After much research, it seems to be due to scaling issue - the resolution on each machine would or might be different and sikuli does not find that image. If I take a screenshot on the testing system with current resolution it works fine. We need to generalize this to work on any machine with any resolution.

I have tried with and without pattern search, with 50-90% similarity as well, it still does not work.

I have changed the resolution of the screen and tested with the generic screenshots and it still does not work.

I have also tried using Maven project and it still is the same issue, works on dev system, does not find image on testing system.

Tools used - Eclipse Photon, Sikuli x V1.1 - 1.4 (tried each version), Java 8+, JRE 1.8 and above

If you are testing a driver to install or reproduce this below issue and code - you can use this link and download the driver.

https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverid=h8m3c

*********************************************
Code as below:
********************************************

public static void main(String[] args) throws FileNotFoundException, IOException, InterruptedException, FindFailed {
  // TODO Auto-generated method stub

  String installtest = "C:\\1234567\\TEST.EXE";
  String dupls = "C:\\1234567\\genimages\\D6.PNG";

  Screen sc = new Screen();

  ArrayList<Image> myArr = new ArrayList<Image>();
        File dir=new File("C:\\1234567\\genimages\\");
        int fileCount=dir.list().length;
        System.out.println("File Count:"+fileCount);
        File[] flist = dir.listFiles();

        List <String> onelist = new ArrayList<String>();

        for (int i = 0; i < fileCount; i++)
        {
                       String fin=flist[i].toString();
                       onelist.add(fin);
                       myArr.add(ImageIO.read(new FileInputStream(("C:\\1234567\\genimages\\D"+i+".PNG"))));
                       System.out.println(fin);
        }
        System.out.println("size of myArr: "+myArr.size());

        System.out.println("One List Array: "+onelist);

        ProcessBuilder pb = new ProcessBuilder(installtest);
  pb.start();
  System.out.println("Initiating File Installation");
  Thread.sleep(3000);

  Region appRegion;

  Loop:
     while(true)
     {

      Thread.sleep(5000);
      appRegion = App.focusedWindow();

     int i=0;
          for(String s: onelist)

          {
           Pattern p = new Pattern(s).similar(0.7f);

           System.out.println("Checking Image: s["+ i +"]");
           if(appRegion.exists(p) != null)

           {
            System.out.println("Image Found at Location: s["+ i +"]");
            sc.click(p);
            Thread.sleep(2000);
            System.out.println("Removed Element from Array: "+s);
            onelist.remove(i);
            break;
           }

           else
           {
            System.out.println("Moving to next image: s["+ ++i +"]");//do nothing
           }
          }

 System.out.println("Updated One List Array: "+onelist);

     if(appRegion.exists(dupls) != null)
     {
      System.out.println("Close Image Found");
      sc.click(dupls);
      Thread.sleep(3000);
      break Loop;
     }

     else
     {
      continue;
     }

    }

 System.out.println("End of Script");

 }

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2018-10-17
Last reply:
2018-10-21
RaiMan (raimund-hocke) said : #1

1 --- Read and understand:
https://sikulix-2014.readthedocs.io/en/latest/basicinfo.html#sikulix-how-does-it-find-images-on-the-screen

2 --- use 1.1.4
in any case (best support).
Wether it is sikulixapi.jar as a library jar or a dependency in a Maven/Gradle project does not matter.

3 --- compare the original shots with those on an offending system
using an image viewer, to find out the differences. If it is only size in pixels, then there is a new resize option (see Pattern class).
If it is in fact different rendering, then you have to create a different image set for each environment.

BTW: you should have a look at the image path feature in SikuliX and have an image naming convention, to avoid those ugly path string handlings and this unnecessary load into an array list.

Rishikesh (rishiy12k) said : #2

Hi RaiMan,

Thank you for the quick response. Really appreciate it.

#1 - Thank you for the link. I have read through the article. I have also gone through the SikuliX 2014 documentation. However, I could not find any topics on "SikuliX’s recommendation how to create target images".

#2 - I shall heed your advise and utilize v1.1.4

#3 - There is no difference in the pixel size - the dimensions are the same when viewed on any screen. So, I suppose the issue is with rendering. The installer GUI differs quality in different resolutions and hence the clarity and scale down look of the buttons changes accordingly. It is not very feasible for me to take images from different systems - we have 100's of systems. However, would you rather suggest take screenshots in different resolutions, perhaps?

#4 - I shall research more on the image path feature and update you if I face any issue.

RaiMan (raimund-hocke) said : #3

at --- #1: SikuliX’s recommendation how to create target images
... yes, you are right, no extra article on that.
most important:
concentrate on the key visual aspects of an image and have as little background as possible towards the edges

at --- #3: I can only give you recommendations, if you send me some samples of the same image in different system setups.
mail to sikulix---at---outlook---dot---com

Rishikesh (rishiy12k) said : #4

Hi RaiMan,

Thank you for the response.

I have sent you a mail with the sample screenshots.

Can you help with this problem?

Provide an answer of your own, or ask Rishikesh for more information if necessary.

To post a message you must log in.