Java FX app: using SikuliX features in handlers freezes UI

Asked by Julian on 2018-03-28

Hey,
i am new to Java and sikuli and i do need some help.
I work with Java 8 and the Intellij IDEA.

This is my Main.java class:
package main;

import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.stage.Stage;

public class Main extends Application {

    @Override
    public void start(Stage primaryStage) throws Exception{
        Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
        primaryStage.setTitle("Hello World");
        primaryStage.setScene(new Scene(root, 300, 300));
        primaryStage.show();
    }

    public static void main(String[] args) {
        launch(args);
    }

}

And this is my Controller.java:

package main;

import javafx.animation.KeyFrame;
import javafx.animation.Timeline;
import javafx.event.ActionEvent;
import javafx.event.EventHandler;
import javafx.scene.control.Button;
import javafx.scene.control.Label;
import javafx.util.Duration;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Location;
import org.sikuli.script.Match;
import org.sikuli.script.Screen;

import java.awt.*;
import java.awt.event.InputEvent;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.concurrent.ThreadLocalRandom;

public class Controller {

    public Label lbl_score;
    public Button btn_startstop;
    public int score = 0;
    public boolean bool_isWorking= false;

    public void initialize() {
       //pasted this from stackoverflow
        Timeline fiveSecondsWonder = new Timeline(new KeyFrame(Duration.seconds(20), new EventHandler<ActionEvent>() {

            @Override
            public void handle(ActionEvent event) {
                testFunction();
            }
        }));
        fiveSecondsWonder.setCycleCount(Timeline.INDEFINITE);
        fiveSecondsWonder.play();
    }

    public void setBtn_startstop(ActionEvent e) {
        if (bool_isWorking) {
            bool_isWorking= false;
            btn_startstop.setText("Start");
        } else {
            bool_isWorking= true;
            btn_startstop.setText("Stop");
        }
    }

    public void testFunction() {
        if (bool_isWorking) {

            if (bool_isWorking) {
                System.out.println("picture1");
                searchforImage("picture1");
            }

            if (bool_isWorking) {
                System.out.println("picture2");
                searchforImage("picture2");
            }

    }

    public void searchforImage(String picture) {

        boolean found = false;
        int clickPositionX = 0;
        int clickPositionY = 0;
        Screen s = new Screen();

        try {

//searches if the image appears and creates "random" coordinations as i dont want the program to click on the exact same spot every time, but using s.click instead of s.find did not help
            Path resurceDirectory = Paths.get("src", "main", "pictures");
            Match m = s.find(resurceDirectory + "/" + picture + ".png");

            int x = m.x;
            int y = m.y;

            Location n = m.getCenter();
            int v = n.x;
            int w = n.y;

            int k = v - x;
            int g = x + 2 * k;

            int o = w - y;
            int u = y + 2 * o;

            clickPositionX = ThreadLocalRandom.current().nextInt(x, g + 1);
            clickPositionY = ThreadLocalRandom.current().nextInt(y, u + 1);

            System.out.println("gefunden!");
            found = true;

        } catch (FindFailed e) {
            // TODO Auto-generated catch block
            // e.printStackTrace();
        }

        if (found) {
            mousemove(clickPositionX, clickPositionY);
        }
    }

    //makes the mousemove less annoying
    public void mousemove(int x, int y) {
        if (bool_isWorking) {
            PointerInfo a = MouseInfo.getPointerInfo();
            java.awt.Point b = a.getLocation();
            int xOrig = (int) b.getX();
            int yOrig = (int) b.getY();

            try {
                Robot r = new Robot();

                r.mouseMove(x, y);
                // press the left mouse button
                r.mousePress(InputEvent.BUTTON1_MASK);
                // release the left mouse button
                r.mouseRelease(InputEvent.BUTTON1_MASK);

                // move the mouse back to the original position

//avoids that the mouse gets stuck between my monitors
                r.mouseMove(1873, 610);
                r.mouseMove(xOrig, yOrig);

            } catch (Exception e) {
                //System.out.println(e.toString());
            }
        }
    }
}

I hope you guys can at least kinda understand my code. The problem i have is that once i run the code my UI starts to freeze while sikuli is searching for an image. Any way to work around this?

Thanks for your help!

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
Julian
Solved:
2018-04-04
Last query:
2018-04-04
Last reply:
2018-04-04

This question was reopened

RaiMan (raimund-hocke) said : #1

With JavaFX you are in a GUI situation, that has some restrictions with usage of Java Robot (which internally in SikuliX is used too) with respect to the EventDispatch queue (handling of actions for GUI elements like buttons ....).

Robot actions can generally not be used in handlers directly, but must be delegated to extra threads/processes.
Consult the net for adequate solutions and concepts.

If you are new to Java and SikuliX, then you should start a bit easier just with a "commandline" Java Program (normal application with no GUI).

Julian (buckfae) said : #2

Okey, ima try it and post again if i need further help.
But i think i can do it myself.

I did a lot of console applications and simple GUI-Applications so i do personally feel that i am ready to do some GUI with Sikuli.

Thanks for your help, your work is awesome, keep it up!

Julian (buckfae) said : #3

https://beginnersbook.com/2013/03/multithreading-in-java/

This is probably the way for me to go?

RaiMan (raimund-hocke) said : #4

yes.

... but you should lookout for info snippets on "JavaFX using Java Robot" - might be that JavaFX has features, to do that more elegant.

Julian (buckfae) said : #5

I sadly run into a new problem:

public class Controller{

    //This Listview is used as log
    public ListView<String> listView_log;
    public static ObservableList<String> listView_log_text = FXCollections.observableArrayList();

    //starts a the new Thread "Workroutine" every 20 seconds and activates the actualizer method every 3 seconds
    public void initialize() {

        debug_log("Started the Work!");

        //starts a the new Thread "Workroutine" every 20 seconds
        Timeline timeline_workRoutineStarter = new Timeline(new KeyFrame(Duration.seconds(20), new EventHandler<ActionEvent>() {

            @Override
            public void handle(ActionEvent event) {
                Thread t = new Thread(new Workroutine());
                t.start();
            }
        }));
        timeline_workRoutineStarter.setCycleCount(Timeline.INDEFINITE);
        timeline_workRoutineStarter.play();

        //activates the actualizer method every 3 seconds, the actualizer method refreshes listView_log
        Timeline timeline_refresh_listView_log = new Timeline(new KeyFrame(Duration.seconds(3), new EventHandler<ActionEvent>() {

            @Override
            public void handle(ActionEvent event) {
                ui_actualizer();
            }
        }));
        timeline_refresh_listView_log.setCycleCount(Timeline.INDEFINITE);
        timeline_refresh_listView_log.play();

    }

 //refreshes the UI, is triggered every 3 seconds by timeline_refresh_listView_log in the initialize() function
    public void ui_actualizer() {

        listView_log.setItems(listView_log_text);
       }

 //refreshes only the listView_log
    public void debug_log(String text) {

        String currentTime = new SimpleDateFormat("HH:mm:ss").format(Calendar.getInstance().getTime());
        listView_log_text.add(currentTime + ": " + text);
        System.out.println(currentTime + ": " + text);
        listView_log.setItems(listView_log_text);

    }

---------------------------------------------------------------------------------------------------------------

public class Workroutine extends Thread {

 public void run() {
// stuff happens, searches for images, works fine now
//if i want to print a result to the Textview from the controller, i call
debug_log("Important stuff!")
}

public void debug_log(String text) {
        String currentTime = new SimpleDateFormat("HH:mm:ss").format(Calendar.getInstance().getTime());
        // Avoid throwing IllegalStateException by running from a non-JavaFX thread.
        Platform.runLater(
                () -> {
                    Controller.listView_log_text.add(currentTime + ": " + text);
                }
        );
    }

So obv the problem was that now that the workroutine is in another thread, i can't access my Listview from the Controller anymore. I can't make it just static, as then the whole ListView doesn't work anymore.
So i made the ObservableList static and i change just the ObservableList from my other thread; the ui_actualizer-function refreshes this every 3 seconds.

I am pretty sure there is a way more elegant solution to this, but i couldn't find it yet.
Thanks for your support!

RaiMan (raimund-hocke) said : #6

1. to let a Thread dynamically access content in the caller, you either have to use the ThreadContext feature or add a constructor to the Thread class (what I usually do):

public class Workroutine extends Thread {

Object someCallerStuff = null; // replace with the type needed

public WorkRoutine(Object someCallerStuff) {
   this.someCallerStuff = someCallerStuff;
}

now you can access someCallerStuff in the run method.

BTW: It is very consuming to create a new thread every time needed. In such a sequential situation you might create it once and then reuse it.

Julian (buckfae) said : #7

Thank you very much, everything working fine right now!
I changed my run() method to:

 public void run() {

        //starts a the new Thread "Workroutine" every 20 seconds
        Timeline timeline_WorkroutineStarter = new Timeline(new KeyFrame(Duration.seconds(20), new EventHandler<ActionEvent>() {

            @Override
            public void handle(ActionEvent event) {
                workroutine();
            }
        }));
        timeline_WorkroutineStarter .setCycleCount(Timeline.INDEFINITE);
        timeline_WorkroutineStarter .play();

    }

And everything that used to be in run() was just moved to the workroutine() method

My initialize() - function in the Controller does now look like this:

    //starts a the new Thread "Workroutine" every 20 seconds and activates the actualizer method every 3 seconds
    public void initialize() {

        debug_log("Started the work!");

        Thread t = new Thread(new Workroutine(listView_log));
        t.start();
    }

As for now, everything seems to work like charm, thank you very much for your support!

RaiMan (raimund-hocke) said : #8

Well done ;-)
Thanks for kind feedback. Have fun.

Julian (buckfae) said : #9

I run into bugs if i have my timeline into the run function. Then somehow the ui freezes again if i call Thread.sleep or anything like that. Now it's in the controller class again.