[research] [Java] Mouse/keyboard choppiness when using concurrent threads

We have to look into the sources, to find a possible reason.

Sure, I'll see if I can built a test program with just the problem spots to explain what's going on.

I've duplicated the issue in a Sikuli project. In my original program, I had 2 threads during many operations, and got Sikuli command lag. In this test program I used 3-4 observers before duplicating my results very noticeably.

The program itself simply has 1 dummy pattern that is looked for by 4 observers. There's an observer for every quadrant of the screen (in graph speak). The program highlights each quadrant, and then begins looking for the dummy pattern. In the almost impossible chance that it finds the pattern (it's pretty random, and 1.00 similarity), the event just passes.

There's an event for each observer to more accurately represent observer usage. In my tests, having multiple events for the observers instead of just one event seems to slow the mouse down even more, but that may not be typical.

While the 4 observers observe, the main thread moves the mouse in a rectangle 3 times to test the responsiveness. After completing 3 rectangles, the observers are stopped, and the mouse makes another 3 rectangles to show the difference in responsiveness.

On my machine, the main thread executes so slowly at times that click/type commands can fail to work properly.

You can download the 4KB .sikuli project here:
http://www.mediafire.com/?oi2pf9gfbbwveai
or just use the source directly below:

#Test program to show that concurrent threads can slow mouse operations
#Jacob Dorman

#dummy pattern
dummy = Pattern("dummy-pattern1.png").exact()
#Create the initial region
firstQuadrant = Region(SCREEN.w/2, 0, SCREEN.w/2, SCREEN.h/2)
firstQuadrant.highlight(1)
#First threaded section (1st quadrant)
def dummyEventOne(event):
    pass

firstQuadrant.onAppear(dummy, dummyEventOne)
firstQuadrant.observe(FOREVER,background = True)

#Second threaded section (2nd quadrant)
def dummyEventTwo(event):
    pass

secondQuadrant = Region(0, 0, SCREEN.w/2, SCREEN.h/2)
secondQuadrant.highlight(1)
secondQuadrant.onAppear(dummy, dummyEventTwo)
secondQuadrant.observe(FOREVER, background = True)

#Third threaded section (3rd quadrant)
def dummyEventThree(event):
    pass
thirdQuadrant = Region(0, SCREEN.h/2, SCREEN.w/2, SCREEN.h/2)
thirdQuadrant.highlight(1)
thirdQuadrant.onAppear(dummy, dummyEventThree)
thirdQuadrant.observe(FOREVER, background = True)

#Fourth threaded section (4th quadrant)
def dummyEventFour(event):
    pass
fourthQuadrant = Region(SCREEN.w/2, SCREEN.h/2, SCREEN.w/2, SCREEN.h/2)
fourthQuadrant.highlight(1)
fourthQuadrant.onAppear(dummy, dummyEventFour)
fourthQuadrant.observe(FOREVER, background = True)

#Main thread, moves the mouse in a rectangle
count = 0
while (count < 3):
    hover(Env.getMouseLocation().above(200))
    hover(Env.getMouseLocation().left(200))
    hover(Env.getMouseLocation().below(200))
    hover(Env.getMouseLocation().right(200))
    count = count + 1
#Stop the threads to show normal Sikuli speed with the same operation
firstQuadrant.stopObserver()
secondQuadrant.stopObserver()
thirdQuadrant.stopObserver()
fourthQuadrant.stopObserver()

count = 0
while (count < 3):
    hover(Env.getMouseLocation().above(200))
    hover(Env.getMouseLocation().left(200))
    hover(Env.getMouseLocation().below(200))
    hover(Env.getMouseLocation().right(200))
    count = count + 1

Thanks for the test script. I will try it too and add some time protocol.

Did you already have a look at the Settings.ObserveScanRate()?

BTW: exact() should not be used currently, since using similar(1.0) might lead to problems (known bug). Use 0.99 instead, which does what you want and always works.

Ah, good to know. I never really used 1.0 in my own programs because it's far too specific for most things, but I was unaware it had a bug. I'll use .99 and explore your recommendation.

It seems like the mouse choppiness is reduced at scan rate of 1, but it's still jumpy enough to miss clicks. Increasing ObserveScanRate smooths the mouse out when the background threads are waiting for the next scan, but the choppiness comes back as soon as they start their next observe cycle.

It's definitely an improvement, and maybe I could use this in my code to make observers viable in my multi-threaded script. I'm just curious as to why concurrent Sikuli search operations can impede Sikuli mouse/keyboard usage.

I appreciate it, RaiMan. I'll look into implementing that in my original project and see if it fixes the issues with clicks failing.

Sorry for the late reply. I couldn't get the program stabilized. Changing MoveMouseDelay to 0 caused some clicks to miss, and 0.1 seemed much more stable, but eventually it too failed due to "thread lag". It's especially noticeable when a thread attempts to use the type method.

For now, I'm going to continue to develop my program without multi-threading. I'm going to mark this as solved. Perhaps someday I'll be able to use multi-threading more when Sikuli is thread safe.

Thanks RaiMan, that solved my question.

Hi RaiMan,

Noticed the same problem while using sikuli-java library and multithreading. Second thread is an observer (implemented either with observer mechanism or a new thread that waits for a match) and first thread does different stuff. In the first thread, mouse clicks are missed and key presses are very slow.

If I run the second thread stand-alone in a new instance of the application, seems to work.
So, a workaround would be single threaded with two OS processes. But this means running two applications.

Any ideas of a better solution?

Thank you!

@Bogdan
With the current version, there is no other chance than to either run only one thread at a time, that uses the mouse or to somehow coordinate the mouse and keyboard usage among the 2 threads yourself (though this is rather complex at the script level).

In the upcoming version 1.1.0 the usage of mouse and keyboard is guarded internally, so that each complex mouse operation (e.g. click) is completely processed, before another thread can use the mouse and do his click. The mouse can even be made exclusive for one thread for some time for critical operations (like for example the short processing in a popup observe-handler).
Furthermore it is detected, that the mouse has been moved by an independent other thread or the user, which can be handled on request (e.g. something like a "rollback" or partial restart).

Though the concurrent handling of the mouse among independent threads is a rather complex challenge, these features will make life a bit easier I guess.

Thank you for your reply RaiMan. Based on what you are saying, I look forward to the new features, would make the things much easier.
Meanwhile I found a more accessible approach: refactored my entire app (I thought easier than to edit sikuli sources to override click) to use the Java built-in Robot class. Until the new features, this works for me.

@Bogdan
... to use the Java built-in Robot class.
... sounds a bit mysteriously, since click internally uses Java Robot as well.

So what do you think might make the difference?

Could you give an example of how you use Robot?

@RaiMan: the robot class is private member in my class, where also i use private methods to do different stuff using sikuli. usually in one private method i make 1-2 actions, like (var names only for example):

Pattern pattern = blablabla;
Location location = region.wait(pattern);
robot.mouseMoveClick(location.x, location.y); //mouseMoveClick is in custom class that extends Robot, basically moves then clicks

//Robot is set to autoWaitForIdle = true and autoDelay = 5ms

Basically sikuli and Robot are running in the same Thread in my case.
Hope this helps.