[HowTo] node-java : Using SikuliX features through API --- tipps and tricks

A little more info: if I pass java.awt.headless=true then the problem doesn't happen, although of course I get null pointer exceptions when I try to actually do anything. I'm not really running headless, but whatever is broken only happens when Sikuli (or AWT, anyway) knows that.

take care, that the JVM gets the option -Dsikuli.debug=3

This should reveal more debug output at startup and help to narrow the offending part.

Helpful of course would be, to reduce the testcase to the minimum of configuration/script content, where it still fails.

Could you provide such a minimum constellation and post it here?

Yep, it's pretty simple:

var java = require('java');

java.classpath.push('sikulixapi.jar');
java.options.push('-Dsikuli.debug=3');

var screen = java.newInstanceSync('org.sikuli.script.Screen');
console.log(screen)

All it's doing is importing the node-java bridge, adding the sikulix jar to the classpath (and the debug option), and then trying to instantiate a Screen (or any Sikuli class that I've tried). The mystery app pops up when I instantiate the class, and nothing ever prints -- not even debug output from Sikuli, weirdly enough.

If I add this line, next to the other option:

java.options.push('-Djava.awt.headless=true')

then the script does finish successfully, but it outputs (from the console.log line):

nodeJava_org_sikuli_script_Screen { h: 1, w: 1, y: 0, x: 0 }

which of course isn't my real screen size. :) Even in this case, though, there's no debug output from Sikuli, which makes me wonder if it's getting eaten by node-java.

I posted an issue with node-java as well, in case it's helpful to see screenshots of what I'm seeing: https://github.com/joeferner/node-java/issues/422

uuuuups - sorry - must be:
java.options.push('-Dsikuli.Debug=3');

thanks.

Oh ha, yeah, that does it. Here's the output I get before it hangs:

[debug] Debug.init: from sikuli.Debug: on: 3
[debug] RunTimeINIT: loadOptions: check: /Users/finnre/slack/slack-desktop/spec/e2e
[debug] RunTimeINIT: loadOptions: check: /Users/finnre
[debug] RunTimeINIT: loadOptions: check: /Users/finnre/Library/Application Support/Sikulix/SikulixStore
[debug] RunTimeINIT: global init: entering as: API

ok, I will have a look and come back tomorrow.

Be prepared to run a setup then with the latest nigthly build of SikuliX 1.1.2

Looking forward to it, thanks! (I think our time zones are pretty far apart, so it may take me a day to report back.)

Apparently there is a problem in the approach of node-java on Mac with the Java
GraphicsEnvironment.getLocalGraphicsEnvironment()
which is the base to evaluate the accessible Graphics devices (screens, printers, ...).

The implementation in this area is system dependent:
on Mac we have the implementing class
sun.awt.CGraphicsEnvironment
which in turn internally accesses some native libraries.

The offending code in SikuliX seems to be (not executed, if java.awt.headless=true):
GraphicsEnvironment genv = GraphicsEnvironment.getLocalGraphicsEnvironment();
GraphicsDevice[] gdevs = genv.getScreenDevices();

To verify you should try to execute just these 2 statements against plain Java.

If this fails, then you have found a problem with node-java, that should be reported there.

Apparently nothing to do anymore for me here.

It seems, that in the current shape of node-java, SikuliX on Mac is not useable this way.

But there are many other options, to integrate with SikuliX.

Ah, that's a bummer! I'd seen the earlier bug report but didn't have enough Java/AWT understanding to know whether I was hitting the same thing. Thanks so much for digging into it.

I added this to the Github thread(s) as well, but for anyone else who looks here first: my manager found a solution to this! Adding:

process.env['JAVA_STARTED_ON_FIRST_THREAD_' + process.pid] = '1'

before instantiating any Sikuli classes through node got it working. :)

If it's all right with you, as I learn more things about this I'm going to continue putting them here, as an aid to future searchers -- if you'd rather that go elsewhere, please let me know. :)

One caveat of this approach is that it appears to support screen-reading operations but not screen-writing; so mouse movement and image finding work, but highlighting fails and hangs. (I suspect that the built-in image-capturing mechanism might also fail, because it shades the screen to indicate the capture area, but I haven't had reason to test it.)

Every information you contribute is always welcome.

I have changed the question a little bit so it is more generic.

You might add comments here for questions and needed enhancements in SikuliX.

Tipps and tricks should be added to the question itself (see my example).

Feel free, to do it as you like - if I have any improvements I will tell.

--- highlight and built-in image-capturing mechanism might also fail
... I guess you are talking about the interactive capture feature (Screen.userCapture() and Screen.selectRegion())

the problem here might be again some threading issue (or even Java EventQueue).
Both features internally work with a screen overlay frame, that has handlers attached for keyboard/mouse actions.
This can only be evaluated with extensive debugging with the source code.

Since I am not sure about the overall value of node-java, I did not step into it until now.

I've got another question for you -- I'm keeping it here for now since it's part of the same weird setup, but happy to start another thread if that would make more sense.

I have a script now (still using the Java API + node-java bridge) which runs fine locally but is hanging in CI, seemingly somewhere inside Mouse.click. I'm not sure what exactly the relevant environment difference is, but here's what I'm seeing.

The relevant bit of node code is as follows (where this.waitForImage is just a wrapper around screen.Wait which knows some things about where images are).

      console.log(`Attempting to paste '${text}' into the '${imageName}' field`);
      const field = await this.waitForImage(imageName);
      console.log(`Clicking into ${imageName} ...`);
      this.screen.clickSync(field);
      console.log(`Pasting into ${imageName} ...`);
      this.screen.pasteSync(field, text);

The output, with debug level 3, is:

Attempting to paste '[redacted]' into the 'slack-placeholder-workspace' field
Waiting for image: slack-placeholder-workspace
[debug] Image: loaded: /Users/travis/build/tinyspeck/slack-desktop/spec/e2e/images/slack-placeholder-workspace.png (file:/Users/travis/build/tinyspeck/slack-desktop/spec/e2e/images/slack-placeholder-workspace.png)
[debug] Image: cached: /Users/travis/build/tinyspeck/slack-desktop/spec/e2e/images/slack-placeholder-workspace.png (25 KB) (# 1 KB 25 -- 0 % of 64 MB)
[debug] Region: wait: waiting 60.0 secs for /Users/travis/build/tinyspeck/slack-desktop/spec/e2e/images/slack-placeholder-workspace.png to appear in S(0)[0,0 1024x768]
[debug] Region: wait: /Users/travis/build/tinyspeck/slack-desktop/spec/e2e/images/slack-placeholder-workspace.png appeared (M[415,279 175x38]@S(S(0)[0,0 1024x768]) S:0.94 C:502,298 [2641 msec])
Found image: slack-placeholder-workspace
Clicking into slack-placeholder-workspace ...
[debug] Region: init: (0, 0, 5, 5)

So it's finding the image and trying to click on it, but gets stuck trying and never gets as far as the paste call. I poked through the code and didn't find anything obvious, other than the fact that it's stopping somewhere before another debug statement.

I noticed in the changelog that the nightly build from a couple of days ago had some version detection improvements, so I tried that, but no luck.

Any other ideas, or any other information I can get for you?

It seems to run on TravisCI: do you have a screen defined?

I use for that:
before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sleep 5

Oh yes, it's definitely running -- I've defined a screen, and can confirm with screenshots that a graphical session has started and is running the applications it's supposed to be running. Even if I didn't have screenshots, Sikuli itself is reporting (in the excerpt above) that it does see the image it's looking for. The only problem (so far) is right there in between those two console.log() lines, where it gets stuck in the click() call and never gets back out. Specifically, it gives me the log output above, then nothing else for ten minutes, at which point the build job times out. Locally, when it gets to this point, it clicks where it's supposed to and continues as expected.

Ok, I have to stop here: I cannot help you: too many dimensions of complexity.

Tracking my situation back (some month ago ;-), I have to admit, that I do not have any testcase, where the click() feature was run on travis.

This is what I do there on an Ubuntu system:

language: java
jdk: oraclejdk9
install: mvn install -P build-fat -DskipTests=true -B
before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sleep 5
script:
 - mvn test -B

I will try to setup a testcase, that clicks and tries to proceed the next week.

And I will add debug level 4 messages to the click feature and add an option to switch those on/off.

Sorry, that I have to let you alone for now.

Ok, what you can do:
simply use the plain Java AWT Robot class yourself in your clickSynch wrapper and try to click ;-)

Oh, good call. I'll see if I can do that and report back. Thanks for helping out as much as you have, I know this is a weird case I'm working on. (The result is really cool though, I promise! ;) )