Using mouseMove() in Games that capture the mouse

Asked by Lukas on 2017-12-31

I'm trying to automate an Unreal based first person 3D game that captures the mouse. I can use mouseMove() just fine for a bit but eventually get errors like:

[error] RobotDesktop: checkMousePosition: should be L(937,1289)@S(0)[0,0 2560x1440] but after move is L(937,1239)@S(0)[0,0 2560x1440] Possible cause in case you did not touch the mouse while script was running: Mouse actions are blocked generally or by the frontmost application. You might try to run the SikuliX stuff as admin.


[error] Location: outside any screen (937, 2160) - subsequent actions might not work as expected

Is there any way to tell SikuliX "just move the mouse to the right without any concern for positioning or screen size" to essentially make the character in the game spin in circles as if I was moving my mouse right for a while?

I now found out that this behavior seems to depend on the Game Engine. In some games the cursor stays in the center of the Screen while in others it's being moved.

To illustrate the issue I'm trying to solve I create a short screen capture:

In all cases I'm using a very simple loop
while True:
    mouseMove(100, 0)

The first game captures the mouse and keeps it at it's own windows center. For some reason however the mouse movement seems very erratic. Instead of just moving sideways it moves up and down as well. Also moving only 10 pixels seems to result in a full rotation in game.

The second game captures the mouse but doesn't keep the cursor in the center. I was using the guide extension to show when the cursor is being moved by SikuliX. Whenever the guide becomes active one can see the current cursor position and how it's moving towards the right. Movement stops once the cursor hits the right side of the screen.
Without the guide active however movement becomes quite erratic as well. As if it's jumping large distances but also moving up and down.

Any hints as to how I could solve this esp. with the second game would be greatly appreciated. Essentially what I'm looking for is a continuous mouse movement to the right so that the character in-game looks around until SikuliX finds the pattern I'm looking for.

Question information

English Edit question
Sikuli Edit question
No assignee Edit question
Solved by:
Last query:
Last reply:
RaiMan (raimund-hocke) said : #1

which system? which Java?

Windows 10 latest build?
see: bug 1730645

What do you mean by: game captures the mouse?

Do you mean, that it reacts on mouse moves with or without buttons pressed, depending on the location in the game before starting to move?

Lukas (lloesche) said : #2

Yes, that and when I move the mouse in any direction it never leaves the currently focused window. It's captured within the active window and can't leave it unless I ALT+Tab out of the task.

Lukas (lloesche) said : #3

Re system specs, Win 10 latest updates, Oracle Java 8 latest, SikuliX 1.1.2 Nightly from like yesterday or the day before.

Best RaiMan (raimund-hocke) said : #4

ok, then you have this problem.
The SikuliX internally used Java Robot for mouse actions somehow collides with the game engine.

Sorry, no solution.

If you have Java knowledge, you might experiment with the Java Robot from inside a SikuliX script (do not use the SikuliX mouse features).

Lukas (lloesche) said : #5

I tried the MouseMoveAround.jar in bug 1730645 and let it loop 200 times without any issues. Does not seem to be Windows specific but more likely depending on the way the game is capturing the mouse. The problem occurs when the cursor is already at e.g. x=2500 of a 2560 wide screen and I try to move it further right by e.g. 100 pixels.

Lukas (lloesche) said : #6

Thanks RaiMan, that solved my question.