[request] Randomized mouse route from A to B?

It can be done by dividing A and B into several parts and moving the mouse in several times.
How to move, not the problem of SikuliX, is your task.
Why do you need that?

To randomly click somewhere inside a region, use something like

a = Region(x, y, w, h) # define a as region, either by giving its coordinates, or as a match from a find action, etc.

click(Location(random.randint(a.getX(),a.getX()+a.getW()),random.randint(a.getY(),a.getY()+a.getH())))

Don't forget to initialize random at the beginning, e.g. with random.seed()

I would like to get something similar to this @masuo.
https://youtu.be/VP2cpgXJzvw

Cant seem to find how to make something like that.

@Manfred, thanks alot for your help concerning the region click :D i'll give this a shot if my first question gets answered aswell.

The base feature is mouseMove(aLocation) or hover(), which moves the mouse from the current location (x,y) to the given location.

In the standard a mouse move from a to b takes 0.5 seconds and is a move on a straight line with decreasing steps towards the target.
As such it is already a bit more human-like, then just moving the mouse to b without any delay.

If you want to move the mouse along some curved line, you have to create a function, that calculates the interpolated coordinates of that curve (the more steps, the smoother).
then a series of mouseMove() (with Settings.moveMouseDelay=0) along that list of coordinates finally does what you want.
Be aware: each step on the line itself is a straight move. So a movement like in your example needs an appropriate amount of steps to make it smooth.

I would also like to know, why you think you need that.

Thanks for the reply, i was hoping some kind of base code was already written and usable somewhere for adjustment.
Your explanation is clear but i know i wont be able to write something like that myself without docs or guideline.

The reason im looking for this is i want to mimic human movements as much as possible, i could also live with me recording my own mouse movements from A to B for 100 times and let sikuli choose a random recording to mimic everytime it moves from A to B or vice versa. Would that be an easier way to reach my goal, is it even possible to do that?

I actually found a piece of python which is exactly as i want it to be, looks amazing!
How can i implement pieces of code like this into Sikuli?
https://github.com/xvol/bezmouse

Good finding ;-)

... but it cannot be used as such with SikuliX due to the dependencies and that it is only for Linux.

But the kernel curve features could be moved to a function as mentioned above and then used to move the mouse along that curve.

I will have a deeper look at it the next days because I indeed find it interesting.

Awh what a letdown that couldnt be used, i found similar projects by googling "human mouse movement python".

Im excited by the fact that you are interested in this! Hopefully we could figure something out.

Here you have a first draft to play with:

The code is taken from github-bezmouse and adapted to the use with SikuliX.

from random import randint, choice
from math import ceil

def pascal_row(n):
  # This returns the nth row of Pascal's Triangle
  result = [1]
  x, numerator = 1, n
  for denominator in range(1, n//2+1):
    # print(numerator,denominator,x)
    x *= numerator
    x /= denominator
    result.append(x)
    numerator -= 1
  if n&1 == 0:
    # n is even
    result.extend(reversed(result[:-1]))
  else:
    result.extend(reversed(result))
  return result

def make_bezier(xys):
  # xys should be a sequence of 2-tuples (Bezier control points)
  n = len(xys)
  combinations = pascal_row(n - 1)
  def bezier(ts):
    # This uses the generalized formula for bezier curves
    # http://en.wikipedia.org/wiki/B%C3%A9zier_curve#Generalization
    result = []
    for t in ts:
      tpowers = (t**i for i in range(n))
      upowers = reversed([(1-t)**i for i in range(n)])
      coefs = [c*a*b for c, a, b in zip(combinations, tpowers, upowers)]
      result.append(
        list(sum([coef*p for coef, p in zip(coefs, ps)]) for ps in zip(*xys)))
    return result
  return bezier

def mouse_bez(init_pos, fin_pos, deviation, speed):

  #time parameter
  ts = [t/(speed * 100.0) for t in range(speed * 101)]

  #bezier centre control points between (deviation / 2) and (deviation) of travel distance, plus or minus at random
  control_1 = (init_pos[0] + choice((-1, 1)) * abs(ceil(fin_pos[0]) - ceil(init_pos[0])) * 0.01 * randint(deviation / 2, deviation),
               init_pos[1] + choice((-1, 1)) * abs(ceil(fin_pos[1]) - ceil(init_pos[1])) * 0.01 * randint(deviation / 2, deviation)
               )
  control_2 = (init_pos[0] + choice((-1, 1)) * abs(ceil(fin_pos[0]) - ceil(init_pos[0])) * 0.01 * randint(deviation / 2, deviation),
               init_pos[1] + choice((-1, 1)) * abs(ceil(fin_pos[1]) - ceil(init_pos[1])) * 0.01 * randint(deviation / 2, deviation)
               )

  xys = [init_pos, control_1, control_2, fin_pos]

  bezier = make_bezier(xys)
  points = bezier(ts)

  return points

def mouseMoveTo(point):
    x = point[0]
    y = point[1]
    target = Location(x,y)
    if (SCREEN.contains(target)):
        Mouse.move(Location(x, y))

def mouseMoveRandom(reg, deviation, speed):
    here = Mouse.at()

    x_coord = reg.x + randint(0, reg.w)
    y_coord = reg.y + randint(0, reg.h)

    curve = mouse_bez((here.x, here.y), (x_coord, y_coord), deviation, speed)

    #print curve

    previous = Location(0,0)
    for point in curve:
        lPoint = Location(point[0], point[1])
        if previous.grow(1).contains(lPoint):
            continue;
        previous = lPoint
        mouseMoveTo(point)

# ----------- main script
Settings.MoveMouseDelay = 0
Mouse.move(getBottomRight().offset(-300, -300))
mouseMoveRandom(Region(0+80, 80, 10, 10), 50, 50)

This looks pretty good. I see its indeed working and randomizing the pattern and start and slowdown speed of the mouse. Looks very human like.

I dont really know how I would apply this though. How do I apply this behaviour everytime the mouse wants to move:

##### ABOVE THIS PART IS RAIMANS CODE ##########

a = Region(x, y, w, h) # define a as region, either by giving its coordinates, or as a match from a find action, etc.

# ----------- main script
Settings.MoveMouseDelay = 0
Mouse.move(getBottomRight().offset(-300, -300))
mouseMoveRandom(Region(0+80, 80, 10, 10), 50, 50)

if exists(IMAGE):
 click(Location(random.randint(a.getX(),a.getX()+a.getW()),random.randint(a.getY(),a.getY()+a.getH())))

So i should define the region where it has to click (dont know how)
And i should define the usage of the human mousemovement to get to the location where it needs to click (also dont know how to).

Both region random click and the human mouse movements are new to me.

Is there a way i can share my script with you?
Its done and functioning very well except for the randomization of wait times, random clicking withing a region and mouse movement.
Really looking forward to implementing those 3 pieces.

--- randomization of wait times
use the random features

--- mouse movement
use mouseMoveRandom()
eventually you have to create an appropriate region before like so:
match = find(someImage)
target = match.getCenter().grow(5)

which would randomize the move target within 5 pixels around

--- random clicking
I guess you are talking about click(someImage) and similar which combine an image search, a mousemove and a click.

You have to create your own random click, hover, ... and use it in the script

def myClick(someImageOrPattern):
    match = find(someImageOrPattern)
    target = match.getCenter().grow(5)
    mouseMoveRandom(target, 50, 50)
    click(Mouse.at())

import random
m = find("1595940323624.png")

click(m.offset(random.randint(0,100), random.randint(0,100)))
wait(random.randint(1,10))
click(m.offset(random.randint(0,100), random.randint(0,100)))
wait(random.randint(1,10))
click(m.offset(random.randint(0,100), random.randint(0,100)))
wait(random.randint(1,10))
click(m.offset(random.randint(0,100), random.randint(0,100)))

I know how to do the random waiting times now, aswell as the random click location within a found image (that was what i was looking for).

Just need to learn how to use the human mouse movement and im set!! Exited :D
Still dont understand that particular part. I'll try to figure it out but if you find some time to explain or show me how to implement that in my code i would be very happy, my codeshare is above your reply.

# ----------- main script
Settings.MoveMouseDelay = 0
Mouse.move(getBottomRight().offset(-300, -300))
mouseMoveRandom(Region(0+80, 80, 10, 10), 50, 50)

match = find("1595942809125.png")
target = match.getCenter().grow(5)

use mouseMoveRandom()

if exists(match):
    click(match)

[error] SyntaxError ( "no viable alternative at input 'mouseMoveRandom'", )

I just dont know how to use it.
Once that is clear my problems will be solved :D

import random
m = find("1595940323624.png")

click(m.offset(random.randint(0,100), random.randint(0,100)))
wait(random.randint(1,10))
click(m.offset(random.randint(0,100), random.randint(0,100)))
wait(random.randint(1,10))
click(m.offset(random.randint(0,100), random.randint(0,100)))
wait(random.randint(1,10))
click(m.offset(random.randint(0,100), random.randint(0,100)))

--- click(m.offset(random.randint(0,100), random.randint(0,100))
depending on the width and height of m, the random click point might be outside of m.
... and it is always right of the center of m

You should pack the special features in functions to save repeated code.

With your example:
import random

def clickR(someImageOrPattern, around = 5):
    match = find(someImageOrPattern) # find image
    target = match.getCenter().grow(around) # target area around pixels around center of match
    mouseMoveRandom(target, 50, 50) # move mouse on a randomized track to a randomized point inside target
    click(Mouse.at()) # now finally click there
    return match

img = "1595940323624.png"
clickR(img, around = 100)
for n in range():
    wait(random.randint(1,10))
    clickR(img, around = 100)

Since I am short on time with other priorities, you now have to find your way.

Thanks i tried it out and am googling to find a solution to the following syntax error:

[log] CLICK on L[1139,599]@S(0) (51 msec)
[error] script [ HumanMouseTest ] stopped with error in line 89
[error] TypeError ( range() takes 1-3 arguments (0 given) )
[error] --- Traceback --- error source first
line: module ( function ) statement
89: main ( <module> ) for n in range():
[error] --- Traceback --- end --------------

    curve = mouse_bez((here.x, here.y), (x_coord, y_coord), deviation, speed)

# ----------- main script
Settings.MoveMouseDelay = 0
Mouse.move(getBottomRight().offset(-300, -300))
mouseMoveRandom(Region(0+80, 80, 10, 10), 50, 50)

import random

def clickR(someImageOrPattern, around = 5):
    match = find(someImageOrPattern) # find image
    target = match.getCenter().grow(around) # target area around pixels around center of match
    mouseMoveRandom(target, 50, 50) # move mouse on a randomized track to a randomized point inside target
    click(Mouse.at()) # now finally click there
    return match

img = "1595940323624.png"
clickR(img, around = 100)
for n in range():
    wait(random.randint(1,10))
    clickR(img, around = 100)

Not a sikuli error, but a python error:

for n in range():
[error] TypeError ( range() takes 1-3 arguments (0 given) )

range needs parameter(s)

How often do you want to repeat the block?
If 200 times, then use range(200), etc.

Settings.MoveMouseDelay = 0
Mouse.move(getBottomRight().offset(-300, -300))
mouseMoveRandom(Region(0+80, 80, 10, 10), 50, 50)

import random

if exists("1595917673821.png"):
        click("1595917673821.png")
        wait(random.randint(5,10))

i dont want it to repeat, i just want the "click" animation above this to be a bezier movement instead of a straight line.

I am a bit lost.

My interpretation of your request was twofold:
- If you click on a found image, don't click in the center, but on some random spot in the image
- If you move the mouse from one spot to another one, don't use straight line, but random path.
Unfortunately these two aspects got somehow mixed in the solutions presented so far.

For just the "random" equivalent of

if exists("1595917673821.png"):
        click("1595917673821.png")
        wait(random.randint(5,10))

I suggest

if exists("1595917673821.png"):
        mouseMoveRandom(gestlastmatch(), 50, 50) # move the mouse randomly to a random spot in the image
        click(Mouse.at()) # do a mouse-click where the mouse currently is
        wait(random.randint(5,10))

Thanks alot for your patience Manfred.

I'm going to paste the code im using now to test your piece:

#above this part is the whole bezier human mouse movement code thats incredibly long
# ----------- main script
Settings.MoveMouseDelay = 0
#Mouse.move(getBottomRight().offset(-300, -300)) # i think i need to delete this. <<<<<<<<<<<<<<<<<<<
#mouseMoveRandom(Region(0+80, 80, 10, 10), 50, 50) # and this. <<<<<<<<<<<<<<<<<<<<<<<

getlastmatch = "1595960559184.png"

if exists("1595960559184.png"):
    mouseMoveRandom(getlastmatch(), 50, 50) # move the mouse randomly to a random spot in the image
    click(Mouse.at()) # do a mouse-click where the mouse currently is
    wait(random.randint(5,10))

When i used your code i got an error saying 'getlastmatch' is not defined so i defined it by making it the same as the image it should check for and click (both are the same image in this case).

if i run the above i get this error:
[error] script [ HumanMouseTest ] stopped with error in line 90
[error] TypeError ( 'str' object is not callable )
[error] --- Traceback --- error source first
line: module ( function ) statement
90: main ( <module> ) mouseMoveRandom(getlastmatch(), 50, 50) # move the mouse randomly to a random spot in the image
[error] --- Traceback --- end --------------

what am i doing wrong here?

The Feature getlastmatch is written
getLastMatch()

Thanks, this works!

I made a video of an example of the mouse movement and random clicking. The mouse is acting weird sometimes, i'm not touching mine and you can tell that it sometimes works very well while others it clearly chops the road it has to go up in pieces and makes a couple of very quick and straight lines within the bezier / curve.

https://youtu.be/-JxLqpga4Qo

Any thoughts?

Thanks alot for the help both!