Help with making some Sikuli image recognition?

With just the basics so far, I've gotten it to look at the area where the first arrow is, and if it sees the left arrow, types L. (Im not sure how make it actually proc the left arrow key yet....)

As soon as I figure that out, I think I could messily make a newb one that does what I want....

Help with things such as the keyboard arrow keys and repeating would still be appreciated~

Ok so like everything that seems like it should be simple enough. It turns out its insanely convoluted....

Things im baffled on -

 - Defining regions and defining which region to watch.

 - moving on to the next region.

As im looking around trying to learn, people always make their examples as complicated and as hard to understand as they can. You find out about one thing, observing regions for example, but they fail to show you anything about the actual region in the coding....

Some help would be really appreciated

Well playing around with it more, and apparently it recognizes all the arrows as eachother, so not matter what arrow I put in to look for it sees it immediately.

So I can't even get to the point of anything else I was asking about. Its down to the basic point of no matter what I use with
 If exists()
if it sees even a part of the image, it continues.

Anyway to fix this.... I know I have to be missing something. Because if Im not, it might as well be a program that would mistake a tree for a frog.

With what version of Sikuli are you playing around?

For a newbe you selected a problem situation, that needs some scripting skills and experience.
I doubt, that you find any helpful example about that in the net.

Looks like some game situation, you want to tackle ??

--- setting up regions
most people have a rather fixed setup of their playground and hence can live with some absolute region definitions.

I myself always do and recommend to do that: define there regions relative with respect to something, that was already found (logos, headers, fixed visuals, window frame, ...).
there are many features to support this relative definition of regions.
the docs tell more about that:
http://sikulix-2014.readthedocs.org/en/latest/region.html

--- if it sees even a part of the image, it continues.
the magic with Sikuli is based on 2 principles:
- define and use a suitable search region
- have optimal screenshots, that reliably allow, to distiguish good matches from unwanted matches (false positives)

To tell you more about possible approaches in your case:
Either setup some screenshots in the net or send me the stuff zipped together to my mail at https://launchpad.net/~raimund-hocke (top left)

I am always interested in such more tricky situations.

Its the 1.1.

I sent you that mail~

Thank you kindly

ok, indeed a bit tricky:
I had to use the triangular head of the arrows, to get a consistent behaviour.
I used an external preview tool to get the exact shots (using magnification).
... and the 2 right arrows are slightly different (as you can see with the scores).

The script can be downloaded here (unzip), to get the images as well:
https://dl.dropboxusercontent.com/u/42895525/arrows.sikuli.zip

it contains the relevant part of the shot you gave me showing the part, that should be selected as search region.
To run it, the arrow bar must be visible on the screen.
The possible result highlights are inactivated.

Of course I could not test the type, might be that a short wait(0.5) might be needed after it.

this is the bare script:
# for doc and test purpose the sample screenshot
# with the recommended selectRegion() area
"original.png"

# the arrows
aL = "aleft.png"
aR = "aright.png"
aU = "aup.png"
aD = "adown.png"

# what should be typed in case
arrows = {aL : Key.LEFT, aR : Key.RIGHT, aU : Key.UP, aD : Key.DOWN}

# get the search region
reg = selectRegion("select arrow bar")
reg.highlight(2)

# for test only: a sample find for each arrow type
for a in arrows:
  reg.find(a)
# reg.highlight(-1)

# the concrete solution
reg.setCols(5) # segment the region horizontally into 5 segments

while True: # loop until finished
  # loop through the segments
  for i in range(5):
    col = reg.getCol(i)
  # col.highlight(1)
    # loop through the arrow types
    for a in arrows:
      # check the arrow type in segment
      if col.exists(Pattern(a).similar(0.8), 0):
        print i, a, col.getLastMatch().getScore()
        type(arrows[a]) # type the associated key
        break # leave the inner loop
  if not popAsk("repeat?"):
    exit()

Oh wow~ Thank you so much, Probably the most help I've received with any type of question online ever!

I am going to check this out immediately and maybe ask you so more questions if thats ok~

always welcome.

as mentioned: interesting enough, to motivate me.

... and I am always interested in learning what new or revised features might be of value.

Ok so I ran it without changing anything, it runs for a moment and then asks me if I want to repeat. If I click yes, it asks me again immediately.

When Im done, I notice you have it printing to the program itself. (Im guessing that I would just change the code slightly for keyboard input?)

Though I noticed this after some more testing
[log] highlight R[341,358 129x25]@S(0)[0,0 1440x900] E:Y, T:3.0 for 2.0 secs

[error] script [ arrows ] stopped with error in line 20
[error] FindFailed ( can not find aleft.png in R[341,358 129x25]@S(0) )

ok, for production you might reduce it to something like that:

# the arrows
aL = "aleft.png"
aR = "aright.png"
aU = "aup.png"
aD = "adown.png"

# what should be typed in case
arrows = {aL : Key.LEFT, aR : Key.RIGHT, aU : Key.UP, aD : Key.DOWN}

# get the search region
reg = Region(x, y, w, h) # insert the real coordinates you might have got from the selectRegion()
# or use any other suitable method to define the region.

# the concrete solution
reg.setCols(5) # segment the region horizontally into 5 segments

while True: # loop until finished
  # loop through the segments
  for i in range(5):
    col = reg.getCol(i)
  # col.highlight(1)
    # loop through the arrow types
    for a in arrows:
      # check the arrow type in segment
      if col.exists(Pattern(a).similar(0.8), 0):
        print i, a, col.getLastMatch().getScore()
        type(arrows[a]) # type the associated key
        break # leave the inner loop
  if not popAsk("repeat?"):
    exit()

You might of course use other means, to trigger the next loop (some image appeared or vanished, many options ;-)

the FindFailed surely comes from this:
# for test only: a sample find for each arrow type
for a in arrows:
  reg.find(a)

because at crash, the arrow bar did not show a left arrow at all

Ok Ok, ya I found out the test chunk there and played around with it some more, I can only seem to get it to recognize a max of 4 out of 5 arrows, it usually misses number 1 or 2.

0 adown.png 0.93724489212
2 aright.png 0.934350371361
3 adown.png 0.813036680222
4 aup.png 0.884007751942

(with 1 being an up arrow, which is strange because it got number 4 which was also an up arrow)

yep, this problem I already noticed:
the arrows do not seem to be pixel-identical in all cases

In your sample that I used:
... and the 2 right arrows are slightly different (as you can see with the scores).

Since the check of possible arrows is rather fast, you might add some variants:

# the arrows
aL = "aleft.png"
aR = "aright.png"
aU = "aup.png"
aU1 = "aup1.png"
aD = "adown.png"

# what should be typed in case
arrows = {aL : Key.LEFT, aR : Key.RIGHT, aU : Key.UP, aU1 : Key.UP, aD : Key.DOWN}

here I added an alternative image for the up arrow.
The rest stays the same (very flexible ;-)

To refine it, you need some viewer, that can magnify without aliasing (showing the pixels as squares when enlarged).

Ok, Ill try to adjust it for a while. (And I might fall asleep) and get back to you again.
Thank you SO MUCH for your help. I really am learning a lot and I really appreciate it!

ok, fine.

as mentioned in the beginning:
as a first example for Sikuli usage you really selected a tricky case.
So be strong ;-)

Where would one add some wait time inbetween the key presses?

Nvm, I see it now... I need some sleep hahaha

    if col.exists(Pattern(a).similar(0.8), 0):
        print i, a, col.getLastMatch().getScore()
        type(arrows[a]) # type the associated key
        wait(0.5) # waits 1/2 second <-----
        break # leave the inner loop

ya adding some variations, helped it A LOT.
still fails on an arrow now and then though, I apparently need to add more.
(I saw what you were talking about, there are like 3-5 different versions of each arrow)

Now assuming I can get that running a little more smoothly.
My only other 2 things I can think of would be -

What the best repeating option would be.

and you mentioned using the disappearance of an image to use as a trigger.
what would the context of that be like if I wanted to use the 'Arrow Bar' disappearing as a trigger to click something else?

Ive been playing with it.

Right now Im trying to get it to
1 - Repeat automatically when the last arrow disappears (the inside of the arrows disappear when pressed, leaving only the outline.) - Is there a GOTO command or something, where I can easily send it back to a certain point? I've been trying to figure out how to make it loop when the last arrow changes, but the while/ifnot has me a bit stumped.

2 - The 'whoops code' - If it has an accident and something goes wrong, the whole arrow bar will disappear. Im trying to get it to watch for the disappearance of the bar - when it does, click a button, and start the arrow section again. (Another reason why Im trying to figure out if GOTO works.)

--- repeat after 5th arrow done

referring to my sample I made for you, I would do it this way:

instead of the final
  if not popAsk("repeat?"):
    exit()

I would wait for the 5th arrow to change.
For that I would store the arrow, that matched a bit earlier:
     if col.exists(Pattern(a).similar(0.8), 0):
        print i, a, col.getLastMatch().getScore()
        type(arrows[a]) # type the associated key
        lastArrow = a # store the match for later use <-----
        break # leave the inner loop

now after the loop finished with the 5th arrow, we know, which arrow matched there and wait for it to vanish:

  if reg.getCol(4).exists(Pattern(lastArrow).similar(0.9), 0):
    wait(0.5)

as we already experienced with the similarity challenge, again here you have to set it reasonably to distinguish from the "naked" arrow (usually it should even be Pattern(lastArrow).exact().

To not get an endless loop here, one should add a max counter, to limit the wait time and either exit or simply assume it can repeat.

  max = 10
  if reg.getCol(4).exists(Pattern(lastArrow).similar(0.9), 0):
    wait(0.5)
    max -= 1
    if max == 0:
      continue # loop again
      # exit() # to terminate in this case (recommended while testing)

BTW:
Modern programming or scripting languages do not have a GOTO anymore. Everything has to be setup in some nested blocks of code. This sometimes make things a bit more complicated depending on the language, but one gets used to it after a while.
This design principal usually leads to the intensive use of modules/functions, to keep the main workflow clear and transparent.

In your case this could be like that:

def handleOneArrow(cell):
    global lastArrow # to signal an outside used variable
    # loop through the arrow types
    for a in arrows:
      # check the arrow type in segment
      if cell.exists(Pattern(a).similar(0.8), 0):
        print i, a, cell.getLastMatch().getScore()
        type(arrows[a]) # type the associated key
        lastArrow = a
        return True # leave the inner loop
    return False # no arrow matched

def decideRepeat(reg, pattern):
  max = 10
  if reg.exists(pattern, 0):
    wait(0.5)
    max -= 1
    if max == 0:
        return False # signal wait exceeded
  return True

as you might realise here: this approach heavily supports the DRY principle (don't repeat yourself): implement functions with some variables, that might be useful in more than one situation.

# all the stuff before the main while loop

while True: # loop until finished
  # loop through the segments
  for i in range(5):
    col = reg.getCol(i)
    if not handleOneArrow(col):
       pass # here we could handle the situation, that no arrow was found
    if not decideRepeat(reg.getCol(4), Pattern(lastArrow).similar(0.9)):
       exit() # or some other action in case of vanishing arrow not detected

now the main workflow is clear and rather fixed. adaptions will only be needed in the functions eventually.

... and now we can see, where we could add some 'whoops code' (nice wording - hope it is not copyrighted ;-)
It does not make sense to check an arrow, when the bar is no longer there:

while True: # loop until finished
  barImage = capture(reg) # save the initial state of the bar
  # loop through the segments
  for i in range(5):
    if not barExists(reg, barImage): <---------
       pass # decide what to do here
    col = reg.getCol(i)
    if not handleOneArrow(col):
       pass # here we could handle the situation, that no arrow was found
    if not decideRepeat(reg.getCol(4), Pattern(lastArrow).similar(0.9)):
       exit() # or some other action in case of vanishing arrow not detected
    barImage = capture(reg) # save the current state of the bar for next existence check

... and we add another def to the top:

def barExists(reg, barImage):
    if not reg.exists(barImage):
        return False
    return True

now we have the last problem to solve:
after having handled the bar-vanished situation we want to start allover again: but we are on the second loop level and want to proceed with the first level.
Python only knows
continue - instantly go to the current loop head and continue
break - leave the current loop at this point and continue with the next statement after the loop

other scripting languages have a feature like
continue label
to tell what outer loop should be continued and/or
break label
to tell what outer loop level should be left

In Python in doubt we need an indicator and an additional decision, but in our case when breaking the inner loop, the next statement will be the While True anyway:

while True: # loop until finished
  barImage = capture(reg) # save the initial state of the bar
  # loop through the segments
  for i in range(5):
    if not barExists(reg, barImage):
       # do everything needed to start again with while True:
       break <---------
    col = reg.getCol(i)
    if not handleOneArrow(col):
       pass # here we could handle the situation, that no arrow was found
    if not decideRepeat(reg.getCol(4), Pattern(lastArrow).similar(0.9)):
       exit() # or some other action in case of vanishing arrow not detected
    barImage = capture(reg) # save the current state of the bar for next existence check

BE AWARE:
nothing tested here - only some thoughts written as real code how your situation could be brought further.

whew, thats gonna take me a while to wrap my head around all that info you just gave me!! hahah
One question before I start studying that~

Instead of lastArrow, what would you use if you wanted to watch the arrow before the last one?
(once the last arrow is matched with the keyboard key, it immediately switches over to a new set of arrows, so only the first four arrows actually have their insides disappear.)

It seems like it would still work unless the new last arrow happens to be the same as the previous one.

It is really fun, to help you: good perception and intelligent questions with good explanations.

In this case, I would simply add a container, that saves the arrows we found in sequence.

just at the beginning of the loop add
foundArrows = [] # empty list

... and when you have decided to type (an arrow was found)
foundArrows.append(a)

... and then when waiting for the bar to reappear:

max = 10
  if reg.getCol(3).exists(Pattern(foundArrow[3]).similar(0.9), 0):
    wait(0.5)
    max -= 1
    if max == 0:
      continue # loop again
      # exit() # to terminate in this case (recommended while testing)

this of course only works, if all arrows are found in a row.

depending on your final solution, it might be necessary to use
foundArrows.append(None)

in the right moment, to keep the foundArrows list consistent (always has 5 elements and each found arrow is at the right place in the list)

Like I mentioned before, you are probably the single most helpful person I've talked to on the internet, hahahah!
I've stopped by a few handfuls of forums that were dedicated to learning things such as PHP in the past. They all were extremely rude and pompous, it was always frustrating and quite baffling.

I seem to have caught a bit of an illness so I still haven't quite finished looking through all the chunks you've shared with me so far, but I was working on them. I first tried the lastArrow just to see what it would do. It would run the first set of arrows just fine, but on the second set it would mostly fail on the first one, sometimes it would get the first 1 or 2 but it could never complete the second set of arrows.

I tried storing the arrows in a container like you suggested in your latest post. I couldn't seem to get the CONTINUE to be nested correctly.

If I just add continue to the bottom instead of popASK and a quick wait() near the start. It ran for about 7 minutes before failing (not sure if it failed to recognize an arrow or the wait() I put in wasnt long enough and it eventually ran too fast and looked at the same set twice.

So ignoring the 'whoops-code' for the moment, does the below look correct if I were adding in to look at the second to last arrow before continuing?

----------
# the concrete solution
reg.setCols(5) # segment the region horizontally into 5 segments

while True: # loop until finished
  foundArrows = [] # empty list
  # loop through the segments
  for i in range(5):
    col = reg.getCol(i)
  # col.highlight(1)
    # loop through the arrow types
    for a in arrows:
      # check the arrow type in segment
      if col.exists(Pattern(a).similar(0.8), 0):
        print i, a, col.getLastMatch().getScore()
        wait(.2)
        type(arrows[a]) # type the associated key
        foundArrows.append(a)
        break # leave the inner loop
  max = 10
  if reg.getCol(3).exists(Pattern(foundArrow[3]).similar(0.9), 0):
    wait(0.5)
    max -= 1
    if max == 0:
      continue # loop again
      # exit() # to terminate in this case (recommended while testing)
----------

should work (takes the 4th arrow to wait for the change of the bar).

Only one problem (typo):
 if reg.getCol(3).exists(Pattern(foundArrows[3]).similar(0.9), 0):

the variable is called foundArrows not foundArrow

Ok~ lol.

One thing that is sorta throwing a wrench into things....
Ill be working on it, and suddenly itll completely stop working.

So Ill think that its something I did. So Ill undo things and run it again. Still doesn't work.
This has happened 3 times now.

The first 2 times, the only way I got it working correctly again, is opening up that original one that only displayed which arrows it saw, and getting new coordinates for the region and re-entering them. Doesn't seem to make sense, so I'm not sure whats going on.

But now, even the original one isnt picking up the arrows...
very frustrating!

frustrating indeed if such things happen, but that is what beginners have to pay due to their lack of experience which bad things might happen when fiddling around with these nasty snippets of code.

The only thing, that helps in the long run is to have a reliable versioning and rollback concept, isolate code, that works form code that is changed or added and finally separate code from data (in the case of Sikuli usually the images).

All these things are not well supported by Sikuli IDE and need some extra investment (concepts and code) to achieve this by the users of Sikuli.

With the visual, pixel based approach one challenge is added: if the base environment changes even only slightly, the script might not work any more.

Might not be comforting, but from time to time I face similar problems with the development of Sikuli myself, because I did not obey these rules enough strictly. There where cases, where it took 2 - 3 days to get back to where I was before.

So in your case, you have to make the decision to go further and learn or leave Sikuli and try another approach.

At this moment, my means to help you efficiently are limited, since I cannot see what you are doing.

Any chance for me to get this arrow bar live on my screen with little effort?

And you can send me what you have currently all zipped together to my mail at https://launchpad.net/~raimund-hocke (top left on the personal page)

I had several working versions, but they all stopped functioning at the same time. hmmmmmm

ok, then you have to check, what has changed with the target app.

I was trying for the longest time, to figure out what went wrong....

The 'target app' had gone through an update and apparently automatically resized the window with all the graphics in it, and because of the fact they are set to 'Stretch to fit' squashed the black lines of the arrows just enough.

The strange thing is that it was working with the arrows in this squashed state for quite a while, but just randomly decided enough was enough.

Of course all that irritation, turns out I only had to resize the graphics window about 2 millimeters larger. and Poof, problem solved......

yes, what I already thought:
With the visual, pixel based approach one challenge is added: if the base environment changes even only slightly, the script might not work any more.

I know, that people with more Sikuli experience at startup first check with one or more key visuals, wether all is as expected.

keep on running. all the best.

I ended up getting pretty ill after I last spoke to you.
I just wanted to thank you for your assistance again.

I much enjoy Sikuli and appreciate the help and patience with such a beginner!

I will continue to learn and stop by again if I get stuck as in it was such a pleasure speaking with you.

thanks for feedback.

If you ever come back with questions, just start new ones on each specific topic.
all the best.