[HowTo] node-java : Using SikuliX features through API --- tipps and tricks

Asked by Finn Ellis on 2018-03-06

------------------------ Mac: Instantiating SikuliX class hangs: solution (from comment #12)

process.env['JAVA_STARTED_ON_FIRST_THREAD_' + process.pid] = '1'

before instantiating any Sikuli classes through node got it working. :)


I'm using the node-java module (https://github.com/joeferner/node-java) as a bridge between some nodejs test scripts and the SikuliX Java API. It's working great on Windows, but when I try to run the same scripts on Mac, it hangs at the first point where it tries to instantiate any class from SikuliX. Specifically, it opens up a little GUI app whose label is just "bin" and which has no windows; command-clicking to see its location takes me to /usr/local/bin (which raises more questions than it answers). When I force quit the app, the script doesn't resume, it just ends.

Using the API and the command line tool work fine (although of course they're not compatible with my Node scripts) -- and using node-java with any other Java package on Mac also works fine. So neither one is broken alone, and I'm not sure what about the combination causes this error. I realize how much of an edge case this is, but any suggestions about what might be going on or where I could look would be much appreciated.

I've tried SikuliX 1.1.1 and 1.1.2, in both cases by completing the setup normally (taking options #1 and #2) and then copying the generated sikulixapi.jar into the folder I'm working in. Java JDK version is 1.8.0, macOS 10.13.3.

Question information

English Edit question
Sikuli Edit question
No assignee Edit question
Solved by:
Last query:
Last reply:
Finn Ellis (finnre) said : #1

A little more info: if I pass java.awt.headless=true then the problem doesn't happen, although of course I get null pointer exceptions when I try to actually do anything. I'm not really running headless, but whatever is broken only happens when Sikuli (or AWT, anyway) knows that.

RaiMan (raimund-hocke) said : #2

take care, that the JVM gets the option -Dsikuli.debug=3

This should reveal more debug output at startup and help to narrow the offending part.

Helpful of course would be, to reduce the testcase to the minimum of configuration/script content, where it still fails.

Could you provide such a minimum constellation and post it here?

Finn Ellis (finnre) said : #3

Yep, it's pretty simple:

var java = require('java');


var screen = java.newInstanceSync('org.sikuli.script.Screen');

All it's doing is importing the node-java bridge, adding the sikulix jar to the classpath (and the debug option), and then trying to instantiate a Screen (or any Sikuli class that I've tried). The mystery app pops up when I instantiate the class, and nothing ever prints -- not even debug output from Sikuli, weirdly enough.

If I add this line, next to the other option:


then the script does finish successfully, but it outputs (from the console.log line):

nodeJava_org_sikuli_script_Screen { h: 1, w: 1, y: 0, x: 0 }

which of course isn't my real screen size. :) Even in this case, though, there's no debug output from Sikuli, which makes me wonder if it's getting eaten by node-java.

I posted an issue with node-java as well, in case it's helpful to see screenshots of what I'm seeing: https://github.com/joeferner/node-java/issues/422

RaiMan (raimund-hocke) said : #4

uuuuups - sorry - must be:


Finn Ellis (finnre) said : #5

Oh ha, yeah, that does it. Here's the output I get before it hangs:

[debug] Debug.init: from sikuli.Debug: on: 3
[debug] RunTimeINIT: loadOptions: check: /Users/finnre/slack/slack-desktop/spec/e2e
[debug] RunTimeINIT: loadOptions: check: /Users/finnre
[debug] RunTimeINIT: loadOptions: check: /Users/finnre/Library/Application Support/Sikulix/SikulixStore
[debug] RunTimeINIT: global init: entering as: API

RaiMan (raimund-hocke) said : #6

ok, I will have a look and come back tomorrow.

Be prepared to run a setup then with the latest nigthly build of SikuliX 1.1.2

Finn Ellis (finnre) said : #7

Looking forward to it, thanks! (I think our time zones are pretty far apart, so it may take me a day to report back.)

Best RaiMan (raimund-hocke) said : #8

Apparently there is a problem in the approach of node-java on Mac with the Java
which is the base to evaluate the accessible Graphics devices (screens, printers, ...).

The implementation in this area is system dependent:
on Mac we have the implementing class
which in turn internally accesses some native libraries.

The offending code in SikuliX seems to be (not executed, if java.awt.headless=true):
GraphicsEnvironment genv = GraphicsEnvironment.getLocalGraphicsEnvironment();
GraphicsDevice[] gdevs = genv.getScreenDevices();

To verify you should try to execute just these 2 statements against plain Java.

If this fails, then you have found a problem with node-java, that should be reported there.

RaiMan (raimund-hocke) said : #9

Apparently nothing to do anymore for me here.

It seems, that in the current shape of node-java, SikuliX on Mac is not useable this way.

But there are many other options, to integrate with SikuliX.

Finn Ellis (finnre) said : #10

Ah, that's a bummer! I'd seen the earlier bug report but didn't have enough Java/AWT understanding to know whether I was hitting the same thing. Thanks so much for digging into it.

Finn Ellis (finnre) said : #12

I added this to the Github thread(s) as well, but for anyone else who looks here first: my manager found a solution to this! Adding:

process.env['JAVA_STARTED_ON_FIRST_THREAD_' + process.pid] = '1'

before instantiating any Sikuli classes through node got it working. :)

Finn Ellis (finnre) said : #13

If it's all right with you, as I learn more things about this I'm going to continue putting them here, as an aid to future searchers -- if you'd rather that go elsewhere, please let me know. :)

One caveat of this approach is that it appears to support screen-reading operations but not screen-writing; so mouse movement and image finding work, but highlighting fails and hangs. (I suspect that the built-in image-capturing mechanism might also fail, because it shades the screen to indicate the capture area, but I haven't had reason to test it.)

RaiMan (raimund-hocke) said : #14

Every information you contribute is always welcome.

I have changed the question a little bit so it is more generic.

You might add comments here for questions and needed enhancements in SikuliX.

Tipps and tricks should be added to the question itself (see my example).

Feel free, to do it as you like - if I have any improvements I will tell.

--- highlight and built-in image-capturing mechanism might also fail
... I guess you are talking about the interactive capture feature (Screen.userCapture() and Screen.selectRegion())

the problem here might be again some threading issue (or even Java EventQueue).
Both features internally work with a screen overlay frame, that has handlers attached for keyboard/mouse actions.
This can only be evaluated with extensive debugging with the source code.

Since I am not sure about the overall value of node-java, I did not step into it until now.