Sikuli not mathing images taken with native MAC OSX Screen Snipping Tool

Asked by Chris Rhoads

I'm having issues with sikuli matching images.
I'm writing a selenium script in Java with sikuli-api 1.2.0 that logs into a remote pc program, finds a terminal by image of its terminal ID, clicks it and then downloads a client zip file.

It mostly works except sikuli is only able to recognize and act on 1 image and Ive got about 30 I need to use.

The one image that it recognizes was taken with Nimbus screenshot tool, the rest were taken with the native Mac OSX screen shot tool.

wsRibId gets recognized - taken with Nimbus
lmiDownloadBarIcon - not recognized taken with native MAC tool

Both are in the same folder within the project.

Ive tried with and without similarity with their pattern objects....

Any help is greatly appreciated!

Code is below.

import junit.framework.Assert;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.sikuli.basics.Debug;
import org.sikuli.script.*;

import javax.swing.*;
import java.awt.event.KeyEvent;

import static org.sikuli.script.Commands.click;

/**
 * Created by chrisrhoads on 4/11/17.
 *
 * Point Of Sale Terminal (POS) Remote Rib Visual Testing
 * Powered By Sikuli API -- www.sikuli.org
 *
 */
public class Main {

    //Start of Client download images
    private static final Pattern lmiDownloadBarIcon = new Pattern(Main.class.getResource("LogMeInDownloadBarIcon.png")).similar((float) 0.75);
    private static final Pattern lmiDownloadZip = new Pattern(Main.class.getResource("LogMeInZip.png")).similar((float) 0.75);
    private static final Pattern lmiClientLaunch = new Pattern(Main.class.getResource("LMIClientIcon.png")).similar((float) 0.75);
    private static final Pattern osxOpenConfirmation = new Pattern(Main.class.getResource("osxOpenConfirmation.png")).similar((float) 0.75);

    //End of Client download images

    //Start of Remote RIB Exit Objects
    private static final Pattern ribSignOut = new Pattern(Main.class.getResource("RIB_Client_SignOutBtn.png")); //shows up after hitting Ctrl+Alt+Delete with active sign in to VM
    private static final Pattern clientXit = new Pattern(Main.class.getResource("RIB_Client_Window_Exit.png"));
// private static final Pattern clientXitConfirm = new Pattern(Main.class.getResource("RIB_Client_Window_Exit_Confirm"));

    //End of Remote RIB Exit Objects

    private static final Pattern sideBarAd = new Pattern(Main.class.getResource("sideBarAd.png"));

    public static void main(String[] args) throws FindFailed, InterruptedException {
        Screen s = new Screen();
        Debug.on(3);
        WebDriver driver = new ChromeDriver();

        driver.navigate().to("https://www.logmein.com");
        driver.manage().window().maximize();
        WebDriverWait wait = new WebDriverWait(driver, 10);

        driver.findElement(By.linkText("Log In")).click();

        //Thread.sleep(3000);
        System.out.println("Login Button #1 was just clicked.");
        wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("#email")));

        WebElement email = driver.findElement(By.cssSelector("#email"));
        WebElement password = driver.findElement(By.cssSelector("#password"));
        WebElement logInBtn = driver.findElement(By.cssSelector("#btnSubmit"));

//login creds are not legit for sake of public posting

        email.click();
        email.clear();
        email.sendKeys("<email address hidden>");

        password.click();
        password.clear();
        password.sendKeys("password");

        logInBtn.click();

        try
        {
            //wsLogInBtn1 = new Pattern();

            s.wait(wsRibId,30);
            s.click(wsRibId);
            System.out.println("First Attempt To Find Image.");
        }
        catch(FindFailed f)
        {
            System.out.println("Exception In First Attempt: " +f.getMessage());
            Assert.fail("Image wasn't found. Please use correct image.");
        }

        Thread.sleep(1000);
        System.out.println("Finding Remote RIB.");

        //In case image/object wasn't clicked in first attempt and cursor stays in the same screen, then do second attempt.
        if(s.exists(wsRibId) != null)
        {
            try
            {
                s.getLastMatch().doubleClick(wsRibId);
                System.out.println("Second Attempt To Find Image.");
                //System.out.println("Object: " +imageLocation + " is clicked successfully.");
            }
            catch(FindFailed f)
            {
                System.out.println("Exception In Second Attempt: " +f.getMessage());
                //System.out.println("FindFailed Exception Handled By Method: ClickObjectUsingSikuli. Please check image being used to identify the webelement. supplied image: " +imageLocation);
            }
        }

//****************************FAILURE POINT*************************************************************
At this point the zip file will be visible in the lower dock of the Chrome Window, which is where the script fails.
Ive tried with several other images as well, using keyEvents to tab to the downloads folder and then look for the
FileName.zip and click or double click on it, I cant get anything to locate.

I had to switch to the mac screenshot tool in orfer to capture outside a web page but many tutorials I have seen
using snippet and navigated Windows file structure based purely on images and it works for them.

//****************************FAILURE POINT*************************************************************

    try
        {
            //wsLogInBtn1 = new Pattern();
            s.wait(lmiDownloadBarIcon,30);
            s.click(lmiDownloadBarIcon);
            System.out.println("First Attempt To Find Image.");
        }
        catch(FindFailed f)
        {
            System.out.println("Exception In First Attempt: " +f.getMessage());
            //System.out.println("FindFailed Exception Handled By Method: ClickObjectUsingSikuli. Please check image being used to identify the webelement. supplied image: " +imageLocation);
            Assert.fail("Image wasn't found. Please use correct image.");
        }

        Thread.sleep(1000);
        System.out.println("Finding Remote RIB.");
        if(s.exists(lmiDownloadBarIcon) != null)
        {
            try
            {
                s.getLastMatch().doubleClick(lmiDownloadBarIcon);
                System.out.println("Second Attempt To Find Image.");
                //System.out.println("Object: " +imageLocation + " is clicked successfully.");
            }
            catch(FindFailed f)
            {
                System.out.println("Exception In Second Attempt: " +f.getMessage());
                //System.out.println("FindFailed Exception Handled By Method: ClickObjectUsingSikuli. Please check image being used to identify the webelement. supplied image: " +imageLocation);
            }
        }

//************************CONSOLE OUTPUT********************************
[debug] Region: wait: waiting 30.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in S(0)[0,0 1680x1050]
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [1090 msec])
[debug] Region: wait: waiting 3.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in S(0)[0,0 1680x1050]
[debug] Region: checkLastSeen: still there
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [125 msec])
[debug] Region: init: (0, 0, 5, 5)
[debug] CLICK on L(360,754)@S(0)[0,0 1680x1050] (805 msec)
First Attempt To Find Image.
Finding Remote RIB.
[debug] Region: exists: waiting 3.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in S(0)[0,0 1680x1050]
[debug] Region: checkLastSeen: still there
[debug] Region: exists: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png has appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [133 msec])
[debug] Region: wait: waiting 3.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in M[317,747 86x14]@S(0)
[debug] Region: checkLastSeen: still there
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [5 msec])
[debug] DOUBLE CLICK on L(360,754)@S(0)[0,0 1680x1050] (695 msec)
Second Attempt To Find Image.
Logging in to remote RIB Console
Remote Client Software Download started.
Sending Keystrokes to try to access the downloads folder
[debug] Region: wait: waiting 30.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png to appear in S(0)[0,0 1680x1050]
[debug] Region: handleFindFailed: P(/Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png) S: 0.5
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png did not appear [30205 msec]
Exception In First Attempt: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png: (272x48) in S(0)[0,0 1680x1050] E:Y, T:3.0
Exception in thread "main" junit.framework.AssertionFailedError: Image wasn't found. Please use correct image.
 at junit.framework.Assert.fail(Assert.java:47)
 at Main.main(Main.java:228)
[debug] RunTimeAPI: final cleanup
[debug] FileManager: deleteFileOrFolder:
/var/folders/17/wvgjzfkd06z_qp2gmzfwfvgh0000gn/T/Sikulix_201706522

Process finished with exit code 1

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
Chris Rhoads
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

not sure with what Sikuli you are working (sikuli-api 1.2.0 does not fit with your dump).

In any case you should try with the latest version 1.1.1 (http://sikulix.com).

Revision history for this message
Chris Rhoads (crhoads1024) said :
#2

Will verify and update.

Revision history for this message
Chris Rhoads (crhoads1024) said :
#3

Verified and Reinstalled sikulixapi.jar v1.1.1 into my java project.

Image matching issue still existing.

*********************************CONSOLE OUTPUT*****************************************

Starting ChromeDriver 2.28.455517 (2c6d2707d8ea850c862f04ac066724273981e88f) on port 35216
Only local connections are allowed.
Apr 17, 2017 12:04:42 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS
Login Button #1 was just clicked.
Login Button #2 was just clicked. RIB list should now be visible.
Sleep ended, waiting for visibility of central_profile_menu'
Visibility of central_profile_menu CONFIRMED
[debug] Region: wait: waiting 30.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in S(0)[0,0 1680x1050]
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [1085 msec])
[debug] Region: wait: waiting 3.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in S(0)[0,0 1680x1050]
[debug] Region: checkLastSeen: still there
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [131 msec])
[debug] Region: init: (0, 0, 5, 5)
[debug] CLICK on L(360,754)@S(0)[0,0 1680x1050] (767 msec)
First Attempt To Find Image.
Finding Remote RIB.
[debug] Region: exists: waiting 3.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in S(0)[0,0 1680x1050]
[debug] Region: checkLastSeen: still there
[debug] Region: exists: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png has appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [130 msec])
[debug] Region: wait: waiting 3.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png to appear in M[317,747 86x14]@S(0)
[debug] Region: checkLastSeen: still there
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeIn_Computers_RibIdNum.png appeared (M[317,747 86x14]@S(S(0)[0,0 1680x1050]) S:0.98 C:360,754 [4 msec])
[debug] DOUBLE CLICK on L(360,754)@S(0)[0,0 1680x1050] (694 msec)
Second Attempt To Find Image.
Logging in to remote RIB Console
Remote Client Software Download started.
Sending Keystrokes to try to access the downloads folder
[debug] Region: wait: waiting 30.0 secs for /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png to appear in S(0)[0,0 1680x1050]
[debug] Region: handleFindFailed: P(/Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png) S: 0.5
Exception in thread "main" junit.framework.AssertionFailedError: Image wasn't found. Please use correct image.
[debug] Region: wait: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png did not appear [30074 msec]
 at junit.framework.Assert.fail(Assert.java:47)
Exception In First Attempt: /Users/chrisrhoads/IdeaProjects/SikuliScript_POS/target/classes/LogMeInDownloadBarIcon.png: (272x48) in S(0)[0,0 1680x1050] E:Y, T:3.0
 at Main.main(Main.java:228)
[debug] RunTimeAPI: final cleanup
[debug] FileManager: deleteFileOrFolder:
/var/folders/17/wvgjzfkd06z_qp2gmzfwfvgh0000gn/T/Sikulix_488896334

Process finished with exit code 1

Revision history for this message
Chris Rhoads (crhoads1024) said :
#4

After much experimentation I discovered the issue is with the images themselves.

The native mac OS snipping tool does not take the image at the proper resolution, it seems retina resolution is too high.

I used Apowersoft Mac Screenshot.

In order for it to work you must set its resolution to 'Standard' and not retina.

The screenshots I took with this resolution setting began working immediately.

I hope this helps anyone else struggling with this!

Revision history for this message
Jai P (jaip) said :
#5

I used the grab.app on Mac and was able to get Sikuli to match the target image. Apowersoft Mac Screenshot did not work for me. I am on macOS Sierra 10.12.5

Revision history for this message
Jai P (jaip) said :
#6

Actually, the Photos app on the Mac is the only pseudo-reliable way to get Sikuli to find the image. You can use it to crop the image to contain the text to be selected.