'findAll(): 'NoneType' object is not iterable' due to inconsistent image recognition?

Asked by Martin Roth on 2017-02-18

Hi there,
I have written a script in SikuliIDE 1.1.0 nightly on an up-to-date Arch Linux system. The script works, well, kind of. It is supposed to run indefinitely but every now and then it stops and throws a 'findAll(): 'NoneType' object is not iterable'.
The script is meant to check the screen for random numbers (between 75-108) in a specific location, recognizing the numerical value, clicking one button on the screen ('store') if it happens to be the highest number as of yet, and keep clicking on another button ('reroll') on the screen to generate a new random number. The idea is to keep going until the user stops the script while storing the highest number along the way.
The script is using cropped images of the numbers as they might appear on the screen. A loop checks for the respective image of every number, then uses findAll() to see how many occurances it can find, then sorts them according to their X coordinate and finally puts them together and converts them to an integer. Then it starts over again.

Settings.MinSimilarity = 0.95

numbPattern = [(0,"num0.png"),(1,"num1.png"),(2,"num2.png"),(3,"num3.png"),(4,"num4.png"),(5,"num5.png"),(6,"num6.png"),(7,"num7.png"),(8,"num8.png"),(9,"num9.png")]

    for i in range(0, 9):
        if rollRegion.exists(numbPattern[i][1]):
            currentRollRaw = list(rollRegion.findAll(numbPattern[i][1])) #error appears here (happens occasionally)
            y = i
            for n in range(len(currentRollRaw)):
                coordRoll = currentRollRaw[n].getX()
                x.append((y,coordRoll))
        else:
            pass

Sometimes numbers don't get recognized correctly (due to my script, not due to Sikuli); eg. 84 on the screen would be recognized as 804 by my script. However, this only happens from time to time. Usually the image recognition is consistent. Numbers above 108 are simply omitted by the script (a not so elegant solution, but still).
I suspect that sometimes 'exists' in the for loop finds a certain number, but then findAll() fails to find it (one or more occurances) and thus 'findAll(): 'NoneType' object is not iterable' terminates the running script. This is due to Sikuli which doesn't recognize the number images consistently.

How can this be avoided? I hope that my explanation is understandable. My apologies in advance if this code makes any veteran programmer nauseous.

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
Martin Roth
Solved:
2017-03-08
Last query:
2017-03-08
Last reply:
2017-02-26
RaiMan (raimund-hocke) said : #1

-- 1: your code as such is ok - no need for being nauseous ;-)

-- 2: use of exists()
in your case, supposing that the number is already visible at the time the loop starts, use exists(..., 0), which restricts each search to only one search trial.

-- 3: exists() does not seem to be necessary at all:
 foundNumbers = []
 minScore = 0.95
 for i in range(9): # range starts with 0 anyway
            matches = rollRegion.findAll(numbPattern[i][1])
            if matches.hasNext():
                 while matches.hasNext()
                     match = matches.next()
                     if match.getScore() > minScore:
                         foundNumbers.append((i, match.x))

-- 4: quality of number shots
you should rework your number shots, especially that of 8 and 0, which seem to have possible scores, that might lead to false positives (as little background as possible around the numbers).

-- 5:
Then I recommend to use individual scores for each number, that you have evaluated before:
numbPattern = [(0,Pattern("num0.png").similar(individualScore)), ..., ... ]

now you can code:
 foundNumbers = []
 for i in range(9): # range starts with 0 anyway
            matches = rollRegion.findAll(numbPattern[i][1])
            if matches.hasNext():
                 while matches.hasNext()
                     match = matches.next()
                     foundNumbers.append((i, match.x))

-- 6: check if a number slot has already been identified
after having implemented step 5, you should sort the rollRegion manually, so that the number with the highest score comes first, than the next with the next lower score and so on.
Now you can implement another check: if any number later is matched again for the same slot (x value), then just forget it.

matchInFoundNumbers():
   for found in foundNumbers:
       if (match.x > found.x - 2 and match.x < found.x +2): return true # +-2 need possibly be adjusted
       return false

 foundNumbers = []
 for i in range(9): # range starts with 0 anyway
            matches = rollRegion.findAll(numbPattern[i][1])
            if matches.hasNext():
                 while matches.hasNext()
                     match = matches.next()
                     if (not matchInFoundNumbers()):
                         foundNumbers.append((i, match.x))

RaiMan (raimund-hocke) said : #2

BTW: I recommend to use version 1.1.1 (http://sikulix.com)

Martin Roth (captain-rage) said : #3

Thank you kindly for the feedback. Being a very novice hobby programmer, I still have trouble understanding parts of the code that you posted (although now I picked up a few new useful commands!).

The old code looks as following:

Settings.MinSimilarity = 0.95

rollRegion = Region(Region(536,570,49,23))

numbPattern = [(0,"num0-1.png"),(1,"num1-1.png"),(2,"num2-1.png"),(3,"num3-1.png"),(4,"num4-1.png"),(5,"num5-1.png"),(6,"num6-1.png"),(7,"num7-1.png"),(8,"num8-1.png"),(9,"num9-1.png")]
bestRoll = 0
count = 0

while True:

    hover(Location(614, 626)) #Reroll
    mouseDown(Button.LEFT)
    wait(0.1)
    mouseUp(Button.LEFT)

    x = []
    for i in range(0, 9):
        if rollRegion.exists(numbPattern[i][1]):
            currentRollRaw = list(rollRegion.findAll(numbPattern[i][1])) # naughty
            y = i
            for n in range(len(currentRollRaw)):
                coordRoll = currentRollRaw[n].getX()
                x.append((y,coordRoll))
        else:
            pass

    sortedRoll = sorted(x, key=lambda mlem: mlem[1])

    finalRoll = []
    for i in range(len(sortedRoll)):
        finalRoll.extend([sortedRoll[i][0]])

    stringRoll=[str(i) for i in finalRoll]
    endString = ''.join(stringRoll)

    endResult = int(endString)

    if endResult > 108:
        print('Omitting: ', endResult)
        endResult = 0
    else:
        pass

    if endResult > bestRoll:
        hover(Location(321, 624))
        mouseDown(Button.LEFT)
        wait(0.1)
        mouseUp(Button.LEFT)
        bestRoll = endResult
    else:
        pass

    count += 1
    print('Last roll: ', endResult)
    print('Best so far: ', bestRoll)
    print('Attempt number: ', count)

Martin Roth (captain-rage) said : #4

Whoops, that got posted before I was done editing, now I don't know how to fix it. As for the Sikuli version, I am now using 1.1.1-nightly. The images of the number patterns are already cropped, like you described (not sure what I can do about the backgrounds, but the number images are rather small, 7x11 pixels, and I tried to crop their images maximally with only the numbers visible).

-- 5:
I tried reading the documentation regarding the similar() command but I am not sure that I am following. Your strategy would be to make a list, based on the similarity of the number patterns (a list with all numbers, very similar or not) and then choose the one with highest similarity? The I would run into the problem with false positives, or atleast getting positive checks but not knowing which magnitude they have (ones, tens, or the single possible hundred).

-- 6:
Do you mean that, if we already have a list with all number matches (according to similarity), then they should be checked for their location using their x coordinate, and only select the most similar match based on its x coordinate for the end result?

Thank you kindly for the example. The problem now is that it will require alot of processing from my side. :) I'll keep thinking about it.

RaiMan (raimund-hocke) said : #5

ok, might all be a little bit complex for a newbee.
I revised the situation and possible solutions and found that the following is the most effective one.

the steps:
- I manually created 10 images of 8x13 for the possible digits (_numN.png) (the _ is a defined precaution for images never be deleted automatically)
- I evaluated the region of number in question: max 6 digits in my case
- I defined a cell width and the steps to walk through the number from left to right with this cell and evaluate each digit using an exists() loop
- to adjust accuracy, I made some tests, to know the minimum similarity needed, to identify each digit (e.g. 0, 8, 6,9 are very similar)

the script can be downloaded from here: (it is zipped)
http://snippets.sikuli.de/examples/nums.sikuli.zip

I used a webpage as testcase, that generates random numbers - in my case between 1 and 1000000
http://www.randomnumbergenerator.com

as a reference I use the click button "Generate number" to evaluate the region that contains the 6-digit number.

To run a test, just prepare the website as I did and run.
If you comment out the lines with the highlight()'s, it runs rather fast.

******* the script text *******
# the manually created digit images
nums = []
for n in range(10):
  nums.append("_num%d.png" % n)

# the website http://www.randomnumbergenerator.com
# must be the frontmost window in a browser
# evaluate the region of the number
imgRef = "imgRef.png"
number = find(imgRef).right(1).right(171).right(1).right(60)
number.highlight(3) # the region that contains the number

# the first cell from left containing a digit
number = find(imgRef).right(1).right(169).right(15)

result = ""
# walk through the possible 6 cells
for n in range(6):
  number.highlight(1) # the cell
  # try the digit, first match wins
  for k in range(10):
    if number.exists(Pattern(nums[k]).similar(0.81), 0):
      print n, k, number.getLastMatch().getScore()
      result += str(k)
  number.x += 10 # switch to the next cell

# the evaluated number
print "the number:", result

Martin Roth (captain-rage) said : #6

Wow, this is alot more than I bargained for. Thanks alot, RaiMan!! I think I picked up quite a few useful tricks from your code. This should lead me to my goal and I will experiment with it more once I get the time (so far I tried it with my images and it is working well, even though they are 9x11 px and white on a black background).
The only issue I suspect might occur in my example is that the cells might get shifted depending on whether a two or a three digit number appears, but it is tricky to be sure since I have never seen a three digit number when manually generating them (since the probability to get one is low). In your example the cells that the numbers can appear in are static. However, now I understand how you can use a cell approach, since the numbers will always have the same relative distance to each other. The tricky thing would be to know the absolute location of the first cell (derived from the left-most number), since it might be different depending on if the number that is generated has two or three digits.

I am very grateful for the example code. I will try to see what I can conjure up once I get a moment to spare. :-)

Regarding the original error (as stated in the title), I realize now that is was a human error and not a malfunction in Sikuli that was the underlying cause.

Much obliged, RaiMan! I owe you one.

RaiMan (raimund-hocke) said : #7

thanks for kind feedback.

Through the years I have come to the decision to not bootstrap any script solution for other people. I do not have the time to do that and once started, it usually does not take an end.

But from time to time, there are scripting challenges, that might lead to a knowledge gain about possible new features or at least enhancements.
Your example is such one: decide which one out of a set of images is the best match in a specific location and the support for creating such image sets.
Another example is the decision, wether a button is active, selected or inactive.