multiple screenshot analysis in a script

Asked by arminius on 2017-08-01

I've been trying for months to screenshot an alarm on a stock webpage, but the background and foreground are so dyanimc that a screenshot I take get's it's match scaled down to below 30% and then so many false positives crop up.

I've been trying to figure out a solution for months, I think I'm close.

I think I need to paraphrase " if exists(Pattern("image seen as of monday").targetOffset(-16,-58)):
                                                             print('found 1st image')
                                                             else:
                                                                  break
                                                          if exists(Pattern("image seen as of tuesday").targetOffset(-16,-58)):
                                                                print('found 2nd image')
                                                          else:
                                                                   break
                                                          if exists(Pattern("image seen as of wednesday").targetOffset(-16,-58)):
                                                                print('found 3rd image')

        hover(getLastMatch())

I need it to look for the first one, if it doesn't find it, it looks for the 2nd, and so on and so forth, when it finds one the loop breaks and it moves on the the hover(getlast match) if it doesn't find one it returns to the top of the loop.
If I come along hours later and find it hung then I add a thursday screenshot, just keep adding to the loop till it will always find a match.

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
Manfred Hampl
Solved:
2017-08-03
Last query:
2017-08-03
Last reply:
2017-08-02
arminius (arminius75) said : #1

it occurs to me I'll probably be mocked for that abridged version of my code up top, was just trying to give an example of what I want it to do.
here's the best I could really do

"def search():
                while True:# 1st attempt to find the image with a loop
                    print('Searching......at the 1st loop.')
                    if exists(Pattern("screenshot1.png").targetOffset(-15,-52)):
                        print('Image1 located!')
                        # break loop.
                        break
                    else:
                        break
                    if exists(Pattern("screenshot2.png").targetOffset(-15,-52)):
                        print('Image2 located!')
                        # break loop.
                        break
                    else:
                        break
        search()"

didn't do what I wanted it to though, it just kept displaying Searching......at the 1st loop over and over again.

keith Fischer (fischertech00) said : #2

If you see the target image but sikuli does not find it consider this: If not already identify if these images appear in a consistent region of the web page. Create the web page sub region as the search region. This makes matches many times more accurate without all the other image noise. Doing this will provide the search accuracy and confidence in your tool. Also, make sure your match png's are pixelated and not lots of solid pixel color. I have sikuli running reliably using these techniques.
Because the target is a web page on a browser, consider using selenium with sikuli. selenium can return the coordinates of the image and use sikuli to verify.

TestMechanic (ndinev) said : #3

arminius,

Your loop is correct - it is only one loop. Seems like 2 images doesn't exists on the screen
If there a way you to provide few images of the whole web page with the alarm(s) inside?

Manfred Hampl (m-hampl) said : #4

It seems to me that your code snippet in comment #1 is nonsense.
In the first if you have "break" in both the "true" and the "else" branches, meaning that the loop will be terminated in any case and the next if will never be reached.

Please describe:
What should happen if the first image is found and
What should happen if the first image is not found?

I understood that you are trying to identify certain signals on stock quote graphics, isn't it?
Due to the fact that there can be an enormous variation in the visual appearance, I have doubts whether it will easily be possible to use Sikuli's image comparison functionality for this purpose.
Maybe an approach to download a certain amount of historical data and doing calculations on the numbers might work better.

arminius (arminius75) said : #5

Keith Fischer
Thanks Keith, I wasn't aware that Selenium could work together, figured it was one or the other, I'll look into that, thanks.

Hey Dinev, I've had about 101 different kinds of loops trying everything and anything to get the result I want.

I think I've had loops that came closer then what I posted, that's just what I had when I finally threw my hands up in frustration and turned to the forums, so if it seems internally inconsistent that's just because it's not finished yet.

But since none of the other's ever did what I wanted it never can be finished, not with my current python programming level.

When I get home I can fish out some of the older loops I worked on and see if any of them are better at conveying what I'm trying to do.

Sorry Manfred, right after posting the 1st one I knew it was a mistake.
Was just supposed to be a crude example to explain what I was trying to do.

I have other loops I worked on, when I found "class FinderFinder(path-to-imagefile)" I figured maybe I could link it to just a folder of screenshots and have it go through all of them till a match was found, didn't work.

If the first image is found the loop should break and it should take that match down to hover(getLastMatch())
If it does not find the first image in say 10 seconds then it looks for the second image and so on and so forth.
I was hoping if it gets to the last image it would just go back to the start and look for the first image again, eventually they should find a match, but if it's taking to long then I can add new images to the bottom of the loop until I might have ten or a hundred.

I know it can work with Sikuli because it does work fine when the screenshot is new, works fine at match levels higher then 80%

But as time goes by I have to lower it to under 60% then under 40% then it starts clicking on the wrong things.

So I figure if instead of lowering the match percentage I just add new screenshots to the bottom of the loop, eventually if I have say a dozen, then one of them will click into place.
And the first screenshot will move in and out of the match range.

Best Manfred Hampl (m-hampl) said : #6

If you have limited number of images, you could use something like

while True:
  if exists(Pattern("image seen as of monday").targetOffset(-16,-58)):
      print('found 1st image')
  elif exists(Pattern("image seen as of tuesday").targetOffset(-16,-58)):
      print('found 2nd image')
  elif exists(Pattern("image seen as of wednesday").targetOffset(-16,-58)):
      print('found 3nd image')
  else:
      print('none of the images found')
      break # retry from the beginning
  hover(getLastMatch())
  #do the real action

With a varying list of images for checking, the case is a bit more complicated.
There are possibilities to loop through all files in a directory, and you would then have to use them in sequence in a exists() statement in a loop until you have found a match (or no match at all).
See python documentation for os.path.walk or os.walk (depending on your python version)

Manfred Hampl (m-hampl) said : #7

Oops, wrong,
instead of "break" you have to use "continue" in case that no image was found to go back to the beginning of the loop.

TestMechanic (ndinev) said : #8

I was with impression we are trying to solve tho ovoid using tons of images for one slightly different element. So my suggestions are in this direction.

One unorthodox solution may be saving the html on every XX seconds(or minutes) and then searching in the html. For example it should look like this:

1. press ctrl+s, type name and enter
2. open saved htm file and analyze it
3. if alarm found do something and exit
4. wait XX seconds/minutes
5. goto 1

keith Fischer (fischertech00) said : #9

Actually, all this could have been done with selenium and even on a headless browser and get all the screenshots you need all run in the background. Based on events detected you could even scrape the various textual facts you need off the page into the run report log.
If you still need sikuli, you would ref the selenium jar from Jython. I have not done from jython but from java to ref the sikuli api jar.

arminius (arminius75) said : #10

Thanks Manfred Hampl, that solved my question.

arminius (arminius75) said : #11

Thanks Manfred you are the man.

This came extremely close "while True:
  if exists(Pattern("image seen as of monday").targetOffset(-16,-58)):
      print('found 1st image')
  elif exists(Pattern("image seen as of tuesday").targetOffset(-16,-58)):
      print('found 2nd image')
  elif exists(Pattern("image seen as of wednesday").targetOffset(-16,-58)):
      print('found 3nd image')
  else:
      print('none of the images found')
      break # retry from the beginning
  hover(getLastMatch())
  #do the real action"

I slotted a screenshot into the 1st position, it hung, I thought that's odd that the first one would pass out of favor so fast.

In the console it was printing found 1st image over and over, so I knew it was finding it fine.

I added break under every instance of print and it worked perfectly.