SikuliX

Using mouseMove() in Games that capture the mouse

Asked by Lukas

I'm trying to automate an Unreal based first person 3D game that captures the mouse. I can use mouseMove() just fine for a bit but eventually get errors like:

[error] RobotDesktop: checkMousePosition: should be L(937,1289)@S(0)[0,0 2560x1440] but after move is L(937,1239)@S(0)[0,0 2560x1440] Possible cause in case you did not touch the mouse while script was running: Mouse actions are blocked generally or by the frontmost application. You might try to run the SikuliX stuff as admin.

and

[error] Location: outside any screen (937, 2160) - subsequent actions might not work as expected

Is there any way to tell SikuliX "just move the mouse to the right without any concern for positioning or screen size" to essentially make the character in the game spin in circles as if I was moving my mouse right for a while?

Edit:
I now found out that this behavior seems to depend on the Game Engine. In some games the cursor stays in the center of the Screen while in others it's being moved.

To illustrate the issue I'm trying to solve I create a short screen capture:
https://www.youtube.com/watch?v=dl3GINsNLrs

In all cases I'm using a very simple loop
while True:
    mouseMove(100, 0)

The first game captures the mouse and keeps it at it's own windows center. For some reason however the mouse movement seems very erratic. Instead of just moving sideways it moves up and down as well. Also moving only 10 pixels seems to result in a full rotation in game.

The second game captures the mouse but doesn't keep the cursor in the center. I was using the guide extension to show when the cursor is being moved by SikuliX. Whenever the guide becomes active one can see the current cursor position and how it's moving towards the right. Movement stops once the cursor hits the right side of the screen.
Without the guide active however movement becomes quite erratic as well. As if it's jumping large distances but also moving up and down.

Any hints as to how I could solve this esp. with the second game would be greatly appreciated. Essentially what I'm looking for is a continuous mouse movement to the right so that the character in-game looks around until SikuliX finds the pattern I'm looking for.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said : 		#1

which system? which Java?

Windows 10 latest build?
see: bug 1730645

What do you mean by: game captures the mouse?

Do you mean, that it reacts on mouse moves with or without buttons pressed, depending on the location in the game before starting to move?

Revision history for this message
Lukas (lloesche) said : 		#2

Yes, that and when I move the mouse in any direction it never leaves the currently focused window. It's captured within the active window and can't leave it unless I ALT+Tab out of the task.

Revision history for this message
Lukas (lloesche) said : 		#3

Re system specs, Win 10 latest updates, Oracle Java 8 latest, SikuliX 1.1.2 Nightly from like yesterday or the day before.

Revision history for this message
Best RaiMan (raimund-hocke) said : 		#4

ok, then you have this problem.
The SikuliX internally used Java Robot for mouse actions somehow collides with the game engine.

Sorry, no solution.

If you have Java knowledge, you might experiment with the Java Robot from inside a SikuliX script (do not use the SikuliX mouse features).

Revision history for this message
Lukas (lloesche) said : 		#5

I tried the MouseMoveAround.jar in bug 1730645 and let it loop 200 times without any issues. Does not seem to be Windows specific but more likely depending on the way the game is capturing the mouse. The problem occurs when the cursor is already at e.g. x=2500 of a 2560 wide screen and I try to move it further right by e.g. 100 pixels.

Revision history for this message
Lukas (lloesche) said : 		#6

Thanks RaiMan, that solved my question.

Revision history for this message
itomcurran (itomcurran) said : 		#7

It seems like you are facing challenges with mouse control in SikuliX, particularly when dealing with certain game engines that capture the mouse. To achieve continuous mouse movement without concerns for positioning or screen size, you can use the mouseMoveOffset function in SikuliX. This function allows you to specify the offset from the current mouse position.

Here's an example of how you can modify your script:

while True:
    mouseMoveOffset(100, 0)

This will move the mouse 100 pixels to the right from its current position in a continuous loop.

However, based on your description, it seems like there may be some issues with the mouse movement in the games you mentioned. SikuliX may not be able to handle certain game engines' mouse capture behavior effectively, leading to erratic movements.

To address this, you can try a few additional steps:

Adjust Mouse Speed:
Experiment with the mouse speed settings in your operating system. Sometimes, slowing down or speeding up the mouse movement can help with erratic behavior.

Run SikuliX as Administrator:
As mentioned in the error message, try running SikuliX as an administrator. Right-click on the SikuliX executable and select "Run as administrator." This might help bypass any restrictions on mouse movement imposed by the game or other applications.

Game-Specific Settings:
Check if the games you are automating have specific settings related to mouse input. Some games may have options to disable mouse capture or adjust sensitivity.

Game-specific SikuliX Configuration:
SikuliX provides configuration options that can be customized for specific applications. You can explore the SikuliX settings related to mouse control and adjust them based on the behavior you observe in your games.

Keep in mind that automating games like https://thecarromapk.com/bitaim-mod-apk/, especially in a way that involves continuous mouse movement, might be against the terms of service of the game and could lead to unintended consequences. Always make sure you comply with the rules and regulations of the games you are automating.