Random Clicking Offset

Asked by EddieWilliams

Is it possible and if so please show me how. This program is great and I read the manule and found clicking offset and it is what i want only I want it random. So it finds the center of the Image were it would click then randomly clicks within 4 pixels of the center for say. Perhaps this is what the offset already does but from what I read it clicked the same pixels over every time. Read through all 200+ questions and was supprised to not have seen this one asked yet. Must be to it being beta.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:

This question was reopened

Revision history for this message
Best RaiMan (raimund-hocke) said :
#1

you have to use the python module random and set your clickpoint.
http://docs.python.org/library/random.html

import random
m = find(image)
center = m.getCenter()
click(center) # clicks center pixel
upperLeft = Location(m.getX(), m.getY())
click(upperLeft) # clicks upper left corner

# now we click10 times randomly within 4 pixels around center
for i in range(0,10):
 click(upperLeft.offset(random.randint(0,8), random.randint(0,8))

Hope it helps.

Revision history for this message
EddieWilliams (deamonrex) said :
#2

import random
m = find(#for this I used a picture of the back button on Internet Explore)
center = m.getCenter()
click(center) # clicks center pixel
wait(5)
upperLeft = Location(m.getX(), m.getY())
click(upperLeft) #clicks upper left corner

#All the code before this works great
# now we click 10 times randomly within 4 pixels around center
for i in range(0,10):
 click(upperLeft.offset(random.randint(0,8), random.randint(0,8))

#the last line a.k.a.Line10 gives me this error msg
#SyntaxError: ("no viable alternative at input '\\n'" ' click(upperLeft.offset(random.randint(0,8), random.randint(0,8))\n'))

If i get this to work I'm going to be really happy,
 Thanks for responding to my noob questions!

Revision history for this message
EddieWilliams (deamonrex) said :
#3

import random
m = find(#for this I used a picture of the back button on Internet Explore)
center = m.getCenter()
click(center) # clicks center pixel
wait(5)
upperLeft = Location(m.getX(), m.getY())
click(upperLeft) #clicks upper left corner

#All the code before this works great
# now we click 10 times randomly within 4 pixels around center
for i in range(0,10):
 click(upperLeft.offset(random.randint(0,8), random.randint(0,8))

#the last line a.k.a.Line10 gives me this error msg
#SyntaxError: ("no viable alternative at input '\\n'" ' click(upperLeft.offset(random.randint(0,8), random.randint(0,8))\n'))

If i get this to work I'm going to be really happy,
 Thanks for responding to my noob questions!

Revision history for this message
RaiMan (raimund-hocke) said :
#4

I hate this really awful message

#SyntaxError: ("no viable alternative at input '\\n'" ' click(upperLeft.offset(random.randint(0,8), random.randint(0,8))\n'))

but it means, that some colon, bracket or other syntax element is missing, but no hint which of them:

in this case a closing bracket is missing (sorry, my fault from beginning :-(

 click(upperLeft.offset(random.randint(0,8), random.randint(0,8)))

Revision history for this message
EddieWilliams (deamonrex) said :
#5

Thanks RaiMan, that solved my question.

Revision history for this message
EddieWilliams (deamonrex) said :
#6

Thanks RaiMan this has saved me a lot of trouble.

I'm confused on what the Varible i is linked to. Also what range(0,10) is reffering to unless its the top corner and bottom right corner. Then I'm like why are we using upperLeft on the next line instead of center. Unless this is just the syntax format the sikuli and python need in order to perfrom this function. Also because I dont understand this I'm unsure how to modify the code above to send one random click instead of Ten.

then the next part that I'm unsure about is what is going to happen when I need to do this for multiple pictures in my loop. Do I use the same upperLeft and center varible names for all the all the pictures in the loop or do I need to have names like centerOne, upperLeftOne, then centerTwo, upperLeftTwo? Were I need to Run my loop bye starting and finding the image and centering it, getting the location vaules then send a random click to the image, Wait (2) , next find the next image and repeate the process. Then this brings another question it I use a loop were it runs 100 times then breaks will it have to do this every time it loops or just the first time and rember those varibles. Also in Python/Sikuli do the varibles in the loop affect the varibles ouside the loop (if there like C then no of course not)

I hate to sound like a real nood here I'm more of a C programmer and this Python stuff is all new to me. I am tying to learn it though. MITopenCourseware computer science videos are what I'm using to learn and there vary helpful. It alwas seemed in C if you wanted to perfrom a function you had to find the correct libray then meets the commands syntax to perform the function which is what you have done here. This is my first time calling any type of GUI/API functions and I still learning. I really apriceate all the help.

Revision history for this message
EddieWilliams (deamonrex) said :
#7

import random
while True:
 setThrowException(False)
 while True:
  wait(random.randint(2,3))
  if find(The Start Button in left cornor of screen):
   wait(random.randint(3,5))
   t = find(first ledge)
   center = t.getCenter()
   click(center.offset(random.randint(1,17), random.randint(1,17)))
   wait(random.randint(7,9))
   l = find(second ledge)
   center = l.getCenter()
   click(center.offset(random.randint(2,10), random.randint(2,11)))
   wait(random.randint(9,11))
   m = find(back to first ledge)
   center = m.getCenter()
   click(center.offset(random.randint(1,7), random.randint(1,7)))
   wait(random.randint(4,6))

This works Wonderfully, The way im calling the random is it clicking the same pixel ever time?

Revision history for this message
EddieWilliams (deamonrex) said :
#8

After waticing it run i changed
 click(center.offset(random.randint(2,10), random.randint(2,11)))
to
click(center.offset(random.randint(-5,10), random.randint(-5,11)))
and this gave a more rounded area of click instead of alwas to on side.

I have answered all my own questions bye reading up on this matter and would like to say,
 RaiMan thank you very much please go enjoy some dersert! You've erened it!

Revision history for this message
EddieWilliams (deamonrex) said :
#9

Thanks RaiMan, that solved my question.

Revision history for this message
RaiMan (raimund-hocke) said :
#10

some answers on some questions and some comments:

--- Loops:
you have < while condition: > and < for variable in iterator: > to define a loop.
The while will run the loop as long as condition is met.
the for loop gets with every run the next element from iterator (e.g. a list of elements (an array in other languages)), as long as elements are there. Since there is no c-like for, you have the range() iterator, that returns the next integer element the next time (a third parameter can define a stepping).

so the following would be equivalent:
i = 0
while i < 10:
   # some code inside loop
   i += 1

for i in range(0,10):
      # some code inside loop

With the second version the intention is more obvious. Since a list with 11 values is produced by range(0,10), it should not be used for big numbers (memory consumption). In these cases you can either use the while construct or the xrange() built-in.

--- scope of variables:
the principle is, that a variable has to be on a left side, before it can be used on the right side (you don't have declarations in python). all variables are accessible everywhere in your script (everything is global by default). the only exceptions are functions (def foo():), modules (to be imported) and classes, which have their own so called namespace (local variables).

def foo(x):
   y = x
   return y

y = 1
print foo(2), y

prints 1 2

insert global y after def foo ... and you will get
2 2

--- Using < while True: > :
this gives an infinite loop. so you have to have some termination code inside the loop.

e.g. the while example above with True:

i = 0
while True:
   # some code inside loop
   i += 1
   if i > 9:
      break

So your example would run for ever. How do you terminate?

--- Exception FindFailed:
There is a function exists(), that returns True if the image is found and otherwise False, without raising FindFailed. So this should be used in if's instead of find().
So you can leave setThrowException() as the default True, giving you the chance to get the script aborted, if some find() or click(image) fails (which normally is a hint for you, that your workflow has to be optimized).

so your example should start:
import random
while True:
 while True:
  wait(random.randint(2,3))
  if exists(The Start Button in left cornor of screen):

This could be optimized as:
while True:
 wait(random.randint(2,3))
 while exists(The Start Button in left cornor of screen):
  # rest of your inner loop
  wait(random.randint(2,3))

and at least your inner loop would have a termination (start button not visible).

--- remembering variables (matches in this case)
if the position of your images (the ledges in this case) is stable after you find your start button, then you could say:

while True:
 wait(random.randint(2,3))
 tCenter = find(first ledge).getCenter()
 # same for l and m
 while exists(The Start Button in left cornor of screen):
  click(tCenter.offset(random.randint(-5,10), random.randint(-5,11)))
  # rest of your inner loop

--- a question:
  wait(random.randint(2,3)) # gives either wait(2) or wait(3)
what are these for? why not wait(3)?

--- another question:
when starting the question, you said, you wanted to click 4 pixels around the center of the screen?
if so, it should be

  click(tCenter.offset(random.randint(-4,4), random.randint(-4,4)))

each call of random.randint(-4,4) returns a random integer between -4 and +4 including the boundaries.

so if you want to make it more flexible and have an option to not click the center:
make a function at the beginning of your script:

def clickAround(center, offset, clickCenter=True):
    x = random.randint(-offset, offset)
    y = random.randint(-offset, offset)
    if clickCenter:
       return center.offset(x,y)
    while x==0 and y==0:
       x = random.randint(-offset, offset)
       y = random.randint(-offset, offset)
    return center.offset(x,y)

so your clicks would be:

click(clickAround(tCenter, 4))

or not clicking the center:

click(clickAround(tCenter, 4, notCenter=True))

You are always welcome with every question.

Revision history for this message
EddieWilliams (deamonrex) said :
#11

The reason I asked to to click withen a 4x4 random offset was to see an example source code that would have taken me hours to find if I would have ever even found it. With you example I was able to answer many Questions. How to wait random intervals of time, How to click random offsets, and How to genrate a random number so that to answer witch path to take this time (I will explan what I mean here later on.) And for this information I am grateful.

The Programming Process
1 Specify the task
2 Discover an algorithm for its solution.
3 Code the algorithm
4 test the code

Part of the Programming process consists of this cycle:
 Edit--------Compile--------Execute
    |---------------------------------|
        Repeat until satisfied with the program performance, the cycle ends.

Using this as my guideline here we go!
1 Runs a course in a 3d game (Most Important is that it is Undetectable)
2 Random wait times with random clicking offset.
3 Still withen the Edit--------Compile--------Execute
4 has been tested 'Once', Editing a new more complex revision

The way I terminate is simple by puting a window over what it is looking for causing it to error out.
The reason for the random wait and clicking is to make it seem more human like.

I have came up with a new algorithm that will make my program perfect! I have a few questions but here is my approach.

The program rolls a number (1-4) [This is what I ment as to "path"]
  Depending on the number will control witch loop is ran.
  All the loops will run the same course but have diffrent wait time and diffrent click positions.

So 1 is rolled, The program selects loop one
  loop one will run its self 5 times then return to the begining and reroll for the possible loop to run.

So 2 is rolled, The program selects loop two
  loop two runs its self 1 time then returns to the begining and reroll for the possible loop to run.
#this is continued for loops 3 and 4 (possibly even more)

  Two advancements to the loops I would like to accomplish:
 1 after each click the program needs to check if it is in the correct possition. There is a possibilty to
    fail causing you to be in a diffrent location then expected if this happens needs to go back were it
    was then continue the loop at the previous step.
 2 If this is possible, the loop to be able to cheak and see witch obstical it is at then start at the
    obstical its at in the loop and contiue from there.

This is going to be a really cool program once its finished and I know there are a lot of people out there wanting to do the same thing. Its really cool to have found this program it's such a powerful tool. Any advice on how to actauly code my algorithm will be great!

Revision history for this message
RaiMan (raimund-hocke) said :
#12

You want to:
- check if it is in the correct position
- check and see witch obstical it is at

I'm not a gamer, but from your words I guess you are running a game, where you have to click somewhere and this brings you further. There are situations, where you are thrown back on your course or you are hindered somehow to proceed.

Since you are simulating human behavior, to not be detected as a program, every check has to be done by looking on the screen and evaluate the presence/absence of visual objects.

So you somehow need a list of possible steps and information about the visual representation of each step.

So you could build up a repository of visual objects, that come up during the course together with information, when and where on the screen they appear or disappear.
(the means of Sikuli would be using a VDict(), that gives you a storage and the ability, to search for images in this repository)

to be able to go back, you have to track your steps.The feature to use would be a list (array in other languages), that you extend with each step or go back if needed.
To make decisions, that are based on visual checks, you have to somehow define your expected situations after an action. this again could be a list.
If your steps could be named (e.g. entrance, back door, living room, kitchen ...) a dictionary would be the thing to use.

Some Sikuli:

using the VDict():

v = VDict() # makes up a new dictionary

v["some-captured-picture"] = (1, (2, 3, 7), (x,y,w,h)) # makes up a new entry

what does this mean:
"some-captured-picture" is the key of the entry and used to search if the VDict() contains an image.

the assigned value could be a list that contains information about the picture.
to make it simple I code the steps with numbers:
1 is the current step, where the picture should be visible
(2, 3, 7) are the possible next steps
(x,y,w,h) is the position/size on the screen where the picture should be

To build up this VDict() is a bit tricky. I would make a helper Sikuli, that interactively gives a chance to capture the images and assign the values:

r = selectRegion("select an image typical for this step")
pic = capture(r)
# pic is in temp dir and has to be moved to a named folder and renamed to be persistent
# going on with temp here
# m = find(pic) could be used to check if it is found
v[ pic ] = ( 1, (2,3,7), (r.x, r.y, r.width, r.height) )

we need a cross reference:
vRef = {} # an empty dictionary
for k in v: # we go thru our VDict
   vRef[ v[ k ][ 0 ] ] = k

after this we know e.g. which entry in the VDict corresponds to step 1

To make all this persistent with a program, additional python programming is necessary to fill a .py, that can be recalled with excefile().

I will stay with this workflow, since it is the basis for something like a Sikuli based MacroRecorder.

If you need further help you may come back. If it makes sense, you can mail me directly with my address in https://launchpad.net/~raimund-hocke. So it could be easy, to send zipped .sikuli's forth and back.

Revision history for this message
RaiMan (raimund-hocke) said :
#13

Sorry, I have to correct:

for k in v: # we go thru our VDict
--- has to be
for k in v.keys(): # we go thru our VDict

the value in a VDict is stored as a list, even if its only one value (don't ask me why, I report a bug).
So to get the value, you have to write:
v[ k ][ 0 ] instead of v[ k ]
in our case:
   vRef[ v[ k ][ 0 ][ 0 ] ] = k

Revision history for this message
EddieWilliams (deamonrex) said :
#14

Once I get around to rewriteing my code I email it to you to check out. I'm working a lot right now so it takes me a bit.

Revision history for this message
RaiMan (raimund-hocke) said :
#15

don't hurry ;-) be happy.

have a nice time. always welcome.

Revision history for this message
EddieWilliams (deamonrex) said :
#16

v = VDict() # makes up a new dictionary
v["some-captured-picture"] = (1, (2, 3, 7), (x,y,w,h))

Ac = VDict()
Ac[ledge one] = (1, (2), (Pic of Rigeon))
Ac[ledge two] = (2, (1), (Pic of Rigeon))

#the list I have is much larger than this any ways I tried

click (1)

it runs then ends but dosnt click the first ledge, am sure I'm doing something wrong. I have an entire VDict created Im just not sure how to use it in my program now.

Now that I'm looking at it would it be-
  vRef[ Ac[ k][ 0 ][ 0 ] ] = k

is there an exaple of how to call and use the VDict images in a sikuli other than the tut on the main page? Thank you

Revision history for this message
RaiMan (raimund-hocke) said :
#17

The idea of the VDict() is to have a storage (since its key/value, its called a dictionary in python) with pictures as keys and some assigned values.
So when you have a given picture, you can lookup the dictionary, wether it contains this picture and get the assigned value. So e.g. you can have a bunch of symbols and you can check, wether it's already there or it's a new one.

So saying click(1), you tried to say, that Sikuli should click on that picture, whose value in Ac has a 1 in its first place.

For this approach, a normal dictionary would be more usable. Try this:

Ac = {} # a new normal dictionary

Ac[1] = (ledge one, (2), (Pic of Rigeon))
Ac[2] = (ledge two, (1), (Pic of Rigeon))

click(Ac[1][0])