SikulixIDE-2.0.3 won't type in Oracle VirtualBox VM

Asked by Patrik Natali on 2020-03-06

I'm on Windows 10, and I have a Oracle VirtualBox VM of Linux. I can switchApp() and click() on elements (as long as I do not have full screen enabled), but I can not type in the input field. I confirmed that I can use the "Soft Keyboard".

I tried changing the host key, I tried changing the "Auto capture keyboard" option. I'm not sure what else there is to try? Sikulix IDE does not show any errors and acts as if it typed just fine, for example:

[log] CLICK on L[1041,681]@S(0) (532 msec)
[log] TYPE "testing"

All it does in this case is click on that spot, then it does not type at all, the script simply finishes.
I should add that I am using Jython, and OpenJDK 13.

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2020-03-07
Last reply:
2020-03-07
Eric G (gamemaster181) said : #1

I had the same type of problem where sikuli would use soft keyboard but still not register keys in a different window.
It because sikuli didn't have the right permissions. Have you tried starting from the administrator command-line:

<path to sikulix> java -jar sikulix-2.0.x.jar

RaiMan (raimund-hocke) said : #2

Already tried to use paste?

Patrik Natali (pat-natali) said : #3

I tried paste(), which also did not work.

My local user account is an administrator.
I opened the command prompt (cmd.exe) with "Run as administrator". I launched "java -jar sikulix-2.0.3.jar". It still did not work.

I tried running Oracle VM VirtualBox as administrator as well.

I tried right clicking the java.exe executable and sikulix executable, selecting "Properties" and checking the "Unblock" checkboxes, which apparently restrict it.

Unfortunately none of it worked.

VirtualBox 6.1.2.
Windows 10
OpenJDK 13.0.2
SikuliX 2.0.3

I can confirm that paste() and type() work with non-VM machines.

If it matters, I have two monitors and I noticed SikuliX seems to prefer the 2nd monitor.

Patrik Natali (pat-natali) said : #4

Now I have tried disabling virus protection and all malware protection I could think of. Still, it does not work :( Perhaps I can move SikuliX to the VM.

baze (baze555) said : #5

I have same problem with a GAME. I tried runing cmd as admin [java -jar] nothing works. However, I installed the older version where it sets up env (i think 1.3) or sth like that and it started working! However if someone knows a solution please write so we don't use the older version for nothing

Patrik Natali (pat-natali) said : #6

Baze, do you mean sikulix version 1.3?

baze (baze555) said : #7

Yes, try it sikulixsetup-1.1.3.jar (md5), it will probably work, however this is not a good solution .. Because it's older version so that means probably more bugs and missing extra options and stuff

Eric G (gamemaster181) said : #8

Have you looked into App.focus()?

https://sikulix-2014.readthedocs.io/en/latest/appclass.html

I'm just speculating here but it could be worth a try. This is just to make sure sikuli is focused on the right window when attempting inputs. Sometimes game clients or applications in general open up secondary windows while minimizing the original, that might mess things up.

App.focus() uses the pid identifier so use task manager or something to get the correct name.

Saying I'm trying to run the program on a game window that only appears after launching game client.

App.focus("Game window name"[index num if more than one instance])
##This brings game window to front

Eric G (gamemaster181) said : #9

Forgot to mention if you want to test if App.focus solves problem you should add it the start then follow it up with type() or paste(). Also the indexing starts at 0 so if you have 4 instances with the same identifier the 4th one would be referenced by index num 3.

Patrik Natali (pat-natali) said : #10

I tried switchApp() with my VM's name. It appeared to switch context successfully, but did not register keyboard events.

Eric G (gamemaster181) said : #11

That's tough...if everything is running at the same privileges and right window. The only other thing I can think of is a timing issue.
Maybe try adding in some delays. The keyboard actions might be too fast to be registered properly. Aside from that, have you considered the possibility of the VirtualBox being a fault? Does it work without a hitch? I don't have a lot of experience with VirtualBox but I think you should consult the user manual or related forums.

I have this snippet from Oracle VM VirtualBox User Manual section 1.8.2 (pg 23) "Basically, you activate the VM by clicking inside it. To return ownership of keyboard and mouse to your host operating system, VirtualBox reserves a special key on your keyboard for itself: the “host key”.
There is also "For technical reasons it may not be possible for the VM to get all keyboard input even when it does own the keyboard. Examples of this are the Ctrl-Alt-Del sequence on Windows hosts or single keys grabbed by other applications on X11 hosts like the GNOME desktop’s “Control key highlights mouse pointer” functionality"

Those are all my ideas so I hope RaiMan has the solution to your problem, good luck!

Eric G (gamemaster181) said : #12

If you're looking to automate testing, perhaps this python script is worth looking into?
https://github.com/Pernat1y/vm-automation
Sorry I couldn't help you out lol

Patrik Natali (pat-natali) said : #13

Holy cow, Baze you are right! I installed version 1.1.3 and it worked immediately. That's really bizarre.

RaiMan (raimund-hocke) said : #14

With version 2.0.x you could try with

Settings.AutoDetectKeyboardLayout = False

In the area of type() this is the only difference between 1.1.3 and 1.1.4.

The default is True, which activates a new feature, that tries to remove the restrictions to the PC-US keyboard layout of type().

... but maybe this is a conflict with VM's.

Can you help with this problem?

Provide an answer of your own, or ask Patrik Natali for more information if necessary.

To post a message you must log in.