Move mouse for click method not working since new JRE and Windows update

Asked by Dan Quimby on 2017-11-01

wondering if anyone else is seeing this.

I install the latest big update for Windows this morning and also accepted an update to the latest JRE.

Now my when Sikuli moves the mouse to click on a match it doesn't land on the coordinates. It even recognizes that it missed and has an error message saying it tried to move the mouse to (x, y), but landed on (p, q) instead.

Any idea what is happening and how to fix it?

Thanks!

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2017-11-02
Last reply:
2017-11-07
arminius (arminius75) said : #1

is Sikuli running with Admin privilege?
Cause click will often not work for me if it doesn't.

Dan Quimby (daqu40) said : #2

Yes, I ensured it was running in Admin after the problem occurred. Unfortunately, that didn't resolve the issue.

RaiMan (raimund-hocke) said : #3

make a failing test case and run it in debug mode.

Post the case and the debug output.

Dan Quimby (daqu40) said : #4

Here is the debug output of the failure:

[debug] Debug: setLogger <TDMethods.myLogger instance at 0x2>
[debug] Debug.setLoggerAll: callback
[debug] JythonSupport: findModule: TDMethods.sikuli ([u'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\TDMethods.sikuli', u'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\TDAutoManager.sikuli', u'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\Lib', 'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\sikulix.jar\\Lib', u'__classpath__', u'__pyclasspath__/'])
[debug] JythonSupport: loadModulePrepare: TDMethods in C:\Users\Dan\Dropbox\Gaming\SikuliX\TDMethods.sikuli
[debug] ImagePath: duplicate not added: /C:/Users/Dan/Dropbox/Gaming/SikuliX/TDMethods.sikuli/
[debug] Debug: setLogFile: C:\Users\Dan\Dropbox\Gaming\SikuliX\TDLog.txt
[debug] Debug: setLogger <TDMethods.myLogger instance at 0x3>
[debug] Debug.setLoggerAll: callback

[debug] Image: loaded: 1494257654383.png (file:/C:/Users/Dan/Dropbox/Gaming/SikuliX/TDAutoManager.sikuli/1494257654383.png)
[debug] Image: cached: 1494257654383.png (1 KB) (# 1 KB 1 -- 0 % of 64 MB)

There are 2 instances of TD running. Would you like to iterate through them?
ml[0]: M[388,452 25x20]@S(S(0)[0,0 2560x1600]) S:1.00 C:400,462 [-1 msec]
running iteration 0
[log] CLICK on L(400,462)@S(0)[0,0 2560x1600]
[error] RobotDesktop: checkMousePosition: should be L(400,462)@S(0)[0,0 2560x1600]
but after move is L(150,489)@S(0)[0,0 2560x1600]
Possible cause in case you did not touch the mouse while script was running:
Mouse actions are blocked generally or by the frontmost application.
You might try to run the SikuliX stuff as admin.

[debug] Image: loaded: 1500905344904.png (file:/C:/Users/Dan/Dropbox/Gaming/SikuliX/TDAutoManager.sikuli/1500905344904.png)
[debug] Image: cached: 1500905344904.png (7 KB) (# 2 KB 8 -- 0 % of 64 MB)
[debug] Region: find: waiting 20.0 secs for P(1500905344904.png) S: 0.65 to appear in R[0,0 2560x1600]@S(0)

[debug] Region: find: P(1500905344904.png) S: 0.65 has not appeared [1682 msec]
[error] script [ TDAutoManager ] stopped with error in line 30
[error] FindFailed ( can not find P(1500905344904.png) S: 0.65 in R[0,0 2560x1600]@S(0) )
[error] --- Traceback --- error source first
line: module ( function ) statement
57: Region ( wait ) Line 2189, in file Region.java
[error] --- Traceback --- end --------------
[debug] Sikulix: cleanUp: 0

RaiMan (raimund-hocke) said : #5

looks like you are running something in parallel.

Sure, that no other part of your stuff moves the mouse in parallel?

Hilary C (hc360) said : #6

Hi, not sure how to run debugging mode (I use the IDE) but what worked yesterday prior to the windows update, seems to no longer work today, and I get a similar error to above. Have run as admin but no difference. You can see the mouse go to the right location, but then sort of jumps around. Any help appreciated, if I can supply more info let me know what/how.

[error] RobotDesktop: checkMousePosition: should be L(2638,1175)@S(0)[0,0 3000x2000] but after move is L(2546,1140)@S(0)[0,0 3000x2000] Possible cause in case you did not touch the mouse while script was running: Mouse actions are blocked generally or by the frontmost application. You might try to run the SikuliX stuff as admin.

[log] CLICK on L(2638,1175)@S(0)[0,0 3000x2000] (539 msec)

[error] script [ CUSFPred ] stopped with error in line 6
[error] FindFailed ( 1506616110472.png: (301x70) in S(0)[0,0 3000x2000] E:Y, T:3.0 )

RaiMan (raimund-hocke) said : #7

could you please report your Windows 10 build stamp.

RaiMan (raimund-hocke) said : #8

Just tested on my
- Windows 10-64 (version 1703, build 15063.674)
- Java 1.8.0_152

with SikuliX IDE 1.1.1 and could not find any strange behavior

Hilary C (hc360) said : #9

windows 10-64 (version 1709, build 16299.19)
Java 1.8.0_151-b12

I was using SikuliX 1.1.0 and had issues, then updated to 1.1.1 but same issue persisted.

RaiMan (raimund-hocke) said : #10

ok, just started the update to 1709 - will take a while.
I then will test again.

Dan Quimby (daqu40) said : #11

My system:
Win 10 (64 bit) Home Edition - version 1709 build 16299.19
Java 1.8.0_151-b12
Sikuli IDE 1.1.0

RaiMan, looks like you'll need to update to version 1709 to see this error.

I'm also going to upgrade the Java to 1.8.0_152 to see if that will help.

Dan Quimby (daqu40) said : #12

Hmm, Oracle's site hasn't released 1.8.0_152 yet.

RaiMan (raimund-hocke) said : #13

as mentioned: my windows update is running (or crawling LOL ;-)
may be I get a chance the next days - ok, just a joke ;-)

Java version 152: I always use the JDK - this is already at 152
... so JRE 151 is the latest.

hope I will be back soon.

RaiMan (raimund-hocke) said : #14

Just an idea:
what happens if you globally set the mouse move time to 0:
Settings.MoveMouseDelay = 0

Dan Quimby (daqu40) said : #15

Here is a video of the behavior. I slowed the mouse down to 2.0.

https://app.box.com/s/3mixeclboy054a4fts1amx5rfxg983r6

It gets close, but something makes it start jumping around and it eventually misses the mark.

RaiMan (raimund-hocke) said : #16

ok, nice ;-)

... but what about
Settings.MoveMouseDelay = 0

RaiMan (raimund-hocke) said : #17

ok, now on Windows 10 1709:
no problems with the mouse movement.
everything works as expected.

Dan Quimby (daqu40) said : #18

I tried it on MouseMoveDelay - 0 and it was the same bad result (my default setting on this was (0.05)

I tried the opposite with a setting of 6 and here is the video: https://app.box.com/s/rnrbr8q8y4zmx30gjly8zxab7fs120s9

Does that give any clue as to why the mouse is moving so erratically when Sikuli is driving?

Dan Quimby (daqu40) said : #19

I updated the Intel video drivers, but that didn't help either.

RaiMan (raimund-hocke) said : #20

could you just run the following script and report:
Settings.MoveMouseDelay = 2
for n in range(2):
  hover()
  getTopLeft().offset(100,100).hover()
  getTopRight().offset(-100,100).hover()
  getBottomRight().offset(-100,-100).hover()
  getBottomLeft().offset(100,-100).hover()
  hover()
wait(3)

and then the same with
Settings.MoveMouseDelay = 0

RaiMan (raimund-hocke) said : #22

Looks like you have a problem on your system.

I am rather sure, that it has nothing to do with SikuliX nor Java (same as mine).

Any software piece running, that might interfere with the mouse driving of Java Robot?

Stop/inactivate everything and then step by step bring it back to normal until the effect gets visible and you have your cause.

Dan Quimby (daqu40) said : #23

Sigh! I develop in .Net on a Trading platform and run into this sort of things from time to time. An issue that despite my best efforts I can't reproduce in my development environment.

In this case it started after a major Windows update, so I'm speculating that something changed in there that's different on my hardware.

So my platform is:
Dell Inspiron 15 - 7000 Series
Intel i7-6500U (2.5 GHz)
Video from CPU (Intel HD Graphics 520)
Mice devices: (2) HD compliant mouse, (1) PS/2 Compatible mouse
Human Interface Devices: dozens that all look standard.

Anyone else seeing this issuse, please put your hardware configuration in the comments and we'll see if we can find something in common.

Meanwhile, I'll do as you suggest RaiMan and see if I can find the offending piece of software.

RaiMan (raimund-hocke) said : #24

Could you please confirm, that you problem exists also in the case of using
Settings.MoveMouseDelay = 0

In this case, the mouse move is not visible and hence I could not see anything about that on your last video.

So just make a test like this:

Settings.MoveMouseDelay = 0
getTopLeft().offset(100,100).hover()
target = getBottomRight().offset(-100,-100)
target.hover()
print target, Mouse.at()

If the coordinates are the same, then I think you have your workaround.

Dan Quimby (daqu40) said : #25

The last video showed with MouseMoveDelay = 2 and = 0 per your instructions.

At MMD = 0 I can't see any movement of the mouse.

This video is a run of the script you just recommended. I'm not sure if it did what you were expecting or not.

https://app.box.com/s/rmg3ibgkvth2jp2j9ge3c4td0xohz3pr

My automation doesn't work any better on MMD = 0.

I also downloaded the SDK to see if that would help, but no luck there either. Also tried disabling the trackpad so only one device could drive the mouse and that hasn't fixed it either.

RaiMan (raimund-hocke) said : #26

ok, not sure, why I cannot see the output of the print at the end ...
should look somehow like this:
L(1179,699)@S(0)[0,0 1280x800] L(1179,699)@S(0)[0,0 1280x800]

Of course one cannot see the movement ;-)

RaiMan (raimund-hocke) said : #27

... and please no more videos ;-)

Dan Quimby (daqu40) said : #28

I was curious about not seeing the output. Sikuli is running as Admin and in debug mode. After loading my debug class and restarting Sikuli here is the output of your latest test script:

[debug] Debug: setLogger <TDMethods.myLogger instance at 0x2>
[debug] Debug.setLoggerAll: callback
[debug] JythonSupport: findModule: TDMethods.sikuli ([u'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\TDMethods.sikuli', u'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\MouseTest2.sikuli', u'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\Lib', 'C:\\Users\\Dan\\Dropbox\\Gaming\\SikuliX\\sikulix.jar\\Lib', u'__classpath__', u'__pyclasspath__/'])
[debug] JythonSupport: loadModulePrepare: TDMethods in C:\Users\Dan\Dropbox\Gaming\SikuliX\TDMethods.sikuli
[debug] ImagePath: duplicate not added: /C:/Users/Dan/Dropbox/Gaming/SikuliX/TDMethods.sikuli/

[debug] Debug: setLogFile: C:\Users\Dan\Dropbox\Gaming\SikuliX\TDLog.txt
[debug] Debug: setLogger <TDMethods.myLogger instance at 0x3>
[debug] Debug.setLoggerAll: callback
[error] RobotDesktop: checkMousePosition: should be L(100,100)@S(0)[0,0 2560x1600]
but after move is L(0,0)@S(0)[0,0 2560x1600]
Possible cause in case you did not touch the mouse while script was running:
Mouse actions are blocked generally or by the frontmost application.
You might try to run the SikuliX stuff as admin.
[error] RobotDesktop: checkMousePosition: should be L(2359,1399)@S(0)[0,0 2560x1600]
but after move is L(2559,1599)@S(0)[0,0 2560x1600]
Possible cause in case you did not touch the mouse while script was running:
Mouse actions are blocked generally or by the frontmost application.
You might try to run the SikuliX stuff as admin.
L(2359,1399)@S(0)[0,0 2560x1600] L(2559,1599)@S(0)[0,0 2560x1600]
[debug] Sikulix: cleanUp: 0

RaiMan (raimund-hocke) said : #29

Looks like some systematic offset ;-) But that is surely by accident.

top left offset (-100, -100)
bottom right offset (200, 200)

Ok, all this has surely something to do with your environment.

Dan Quimby (daqu40) said : #30

These are correct. I changed bottom right offset to (-200,-200) to keep it from triggering the Windows 10 show desktop feature that happens when the mouse is put in the lower right corner of the screen.

This only started happening after the big Windows 10 release was installed.

Since the error is in RoboDesktop what sorts of things affect that module's behavior?

RaiMan (raimund-hocke) said : #31

As already mentioned: the Java class used finally deep down inside is java.awt.Robot (RobotDesktop is a subclass of it).

If you are interested, I can make a Java only small jar, that only moves the mouse around and reports what happens on a command line.

Dan Quimby (daqu40) said : #32

Let's give that shot RaiMan.

Maybe it will give us a clue what thing in my system is interacting badly with the Java to cause the erratic movement when Java is driving.

RaiMan (raimund-hocke) said : #33

Find a small test jar in the related bug (only there it is possible to use attachments).

Dan Quimby (daqu40) said : #34

been busy on another project. Will get to trying the jar this week.

RaiMan (raimund-hocke) said : #35

no problem ;-)

I am on holidays the next 2 weeks - so take your time.

TestMechanic (ndinev) said : #36

You may try experimenting with AutoIt tool for Windows. There are commands for moving mouse as well. This may give some insight where the problem is

Can you help with this problem?

Provide an answer of your own, or ask Dan Quimby for more information if necessary.

To post a message you must log in.