[request] findAll() should only return the most probable matches

Bug #880861 reported by Andreas Balogh
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
SikuliX
Fix Released
High
RaiMan

Bug Description

--- request (based on the primary bug situation)
valuable test information below in comment #5 +

findAll(): "most probable" means, that only those matches are returned, whose score differ only slightly from that of the best match (e.g. max 0.3 or even less). This surely is the expectation of most people, when using findAll().

Additionally it might make sense, to implement a findAny(), that returns ALL matches having a score above the given similarity, because this might be the expected result in some situations, when using findAll() in its current shape.

-------------------------------------------------

1. Sikuli 1.0 rc3
2. Windows XP SP3 32-bit
3. Enclosed code loops over all "Apply" boxes. Once apply has been clicked it wait for a confirmation box to appear, confirms, waits for the confirmation box to disappear. The final match does not point to the bitmap.
4. Required bitmaps attached.

# setFindFailedResponse(PROMPT)
setROI(Screen(0))

applies = findAll("Apply.png")

for apply in applies:
    click(apply)
    confirm="Doyoureallyw-1.png"
    wait(confirm)
    click(Pattern("YesNo.png").targetOffset(-31,-1))
    waitVanish(confirm)

Revision history for this message
Andreas Balogh (baloand) wrote :
Revision history for this message
RaiMan (raimund-hocke) wrote :

I do not think, that this is a bug, but ...

To analyze the situation pls. attach a screenshot too. This makes it possible to see, what really happens.

There might be similar things on the screen, that are matched with findAll(), but with a lower similarity.

so you might try:

applies = findAll(Pattern("Apply.png").similar(0.9))

Other general things:
- capture as little background as possible. The red apply button should only contain the text "Apply".
- If possible narrow the search region to the area, where the apply buttons appear and use reg.findAll()

Revision history for this message
Andreas Balogh (baloand) wrote :

Well, I got proper recognition working after extensively playing by
a) reducing pattern size, and
b) adding .similar(0.97) - 0.96 and 0.98 do not work.

In Sikuli 0.10.2 pattern recognition (same code without similar and pattern consisting of the complete pushbutton) worked without tweaking - much easier to handle.

Compared to Sikuli 0.10.2 pattern recognition is DIFFICULT to get right.

So I'd consider this a regression bug. Does it make sense to restore 0.10.2 recognition in 1.0?

Much of the fun using Sikuli is gone if recognition becomes unreliable and works only after spending a lot of time on tweaking the pattern and .similar(x).

Revision history for this message
RaiMan (raimund-hocke) wrote :

--- So I'd consider this a regression bug
You might get this impression based on your current situation. But the VisionEngine of X is much faster and much more precise and reliable, than the one from 10.2.
And it might be, that this improvement has some impact on finding images having much background of a solid color like in your case, compared to the content, that makes it distinct. This is an aspect that should be evaluated.
... but the similarity is not a tweak, it is a feature, to tell Sikuli, what results you expect.

--- Does it make sense to restore 0.10.2 recognition in 1.0?
simply no, because this would really be a big regression.

--- Much of the fun using Sikuli is gone if recognition becomes unreliable
No, this should not be.
The difference between 10.2 and X might be, that the score of a match is higher in average, so using findAll() without a similarity value returns more (and not wanted) matches than in 10.2 using the default of 0.7.
This is worth an evaluation: Should the standard similarity be raised above 0.7 (e.g. 0.8)?

Sorry for not thinking about that possibility when answering your post: Your script might run without modification if you just set the minimum similarity to e.g. 0.8 (see: http://sikuli.org/docx/globals.html#Settings.MinSimilarity)

--- conclusion
I think, in most cases we would expect from findAll, that it only returns the most probable matches (as in your case). This means, all matches with a score near the one of the best match. If you analyze your "wrong" result, you will find, that the wrong match's score is very different.

I will change this bug into a request bug, to get an improved findAll().

summary: - findAll() with multiple matches returns invalid last match
+ [request] findAll() should only return the most probable matches
Changed in sikuli:
importance: Undecided → Wishlist
RaiMan (raimund-hocke)
description: updated
Revision history for this message
Andreas Balogh (baloand) wrote :

Thanks for your feedback. To further facilitate improving findAll() I added my testcase:
1. apply-grid.gif
2. run code below (slightly modified code attachment):

setROI(Screen(0))
applies = findAll("Annly.png")
for i, apply in enumerate(applies):
    print i, apply.getTarget(), apply.getScore()

3. enclosed the result on my box. What I'd expect are 13 matches with score 1.0. When using "Matching Preview" I can see matches which are completely unrelated (no border, no text, different color). Looks to me like the non-matching scores are much too high. I am especially irritated that borders without text match better than identical patterns with only different color (none of the yellow or green buttons match). I'd expect the match to contain at least some text. Enclosed as well matching preview screenshot.
"0 (219,354) 1.0
1 (219,678) 1.0
2 (219,696) 1.0
3 (219,732) 1.0
4 (219,750) 1.0
5 (219,822) 1.0
6 (219,372) 1.0
7 (219,390) 1.0
8 (219,408) 1.0
9 (219,426) 1.0
10 (219,804) 0.999999821186
11 (219,606) 0.999999821186
12 (219,588) 0.999999582767
13 (198,228) 0.953331410885
14 (302,134) 0.833787679672
15 (302,210) 0.830605745316
16 (857,362) 0.81985527277
17 (121,58) 0.814565420151
18 (105,362) 0.795119285583
19 (73,362) 0.795119285583
20 (46,362) 0.795119285583
21 (223,877) 0.793206393719
22 (192,877) 0.792263448238
23 (38,451) 0.791314184666
24 (96,58) 0.788552463055
25 (610,299) 0.788552105427
26 (586,299) 0.788552105427
27 (881,223) 0.786840677261
28 (828,603) 0.775286316872
29 (852,603) 0.775285959244
30 (145,58) 0.775285601616
31 (68,590) 0.768583893776
32 (44,590) 0.768583893776
33 (640,71) 0.761338293552
34 (665,71) 0.758514881134
35 (326,134) 0.748186349869
36 (908,603) 0.741286456585
37 (556,603) 0.741286456585
38 (532,603) 0.741286456585
39 (883,603) 0.7412860989

57
40 (610,1005) 0.737122774124
41 (562,1005) 0.737122774124
42 (112,286) 0.727324426174
43 (634,299) 0.727323651314
44 (618,590) 0.723185300827
45 (507,58) 0.713712573051
46 (799,147) 0.713135540485
47 (912,223) 0.710099637508
48 (917,514) 0.709267377853
49 (468,603) 0.705952942371
50 (799,514) 0.705952942371"

Revision history for this message
Andreas Balogh (baloand) wrote :
Revision history for this message
Andreas Balogh (baloand) wrote :

Result of same code run wih Sikuli 0.10.2:
0 (129,788) 0.948552966118
1 (129,554) 0.948552310467
2 (129,320) 0.948552310467
3 (129,374) 0.948552310467
4 (129,662) 0.948552310467
5 (129,716) 0.948552966118
6 (129,770) 0.948552310467
7 (129,356) 0.948552310467
8 (129,392) 0.948552310467
9 (129,644) 0.948552310467
10 (129,698) 0.948552310467
11 (129,338) 0.948552966118
12 (129,572) 0.948552310467

-> matches what I'd expected as result.

This is why I'd consider this a regression bug.

Regards, Andreas

Revision history for this message
RaiMan (raimund-hocke) wrote :

Great, thanks for the documentation. This will really help and supports my request towards findAll().

The "problem" is due to the drastic improvement of the vision engine.
The score in 10.2 is below .95 and is .99+ (which means exact match) in X. This difference means worlds in pattern recognition.
But I agree, that this was implemented in X without notice. And everyone, who works with findAll() should be aware of this situation.
That you only got the expected matches in 10.2 was due to the fact, that all other matches in 10.2 had a score below .7. And again I agree, from your sight, this was what you expected and did not work then same in X.

And there might still be problems in the new vision engine together with solid colors as background in images (what you stated above analyzing the Preview result).

description: updated
RaiMan (raimund-hocke)
Changed in sikuli:
status: New → In Progress
assignee: nobody → RaiMan (raimund-hocke)
importance: Wishlist → Medium
tags: added: fkt-region
removed: findall match
RaiMan (raimund-hocke)
Changed in sikuli:
importance: Medium → High
milestone: none → x1.1
RaiMan (raimund-hocke)
tags: added: todo
RaiMan (raimund-hocke)
Changed in sikuli:
milestone: 1.1.0 → 1.2.0
RaiMan (raimund-hocke)
Changed in sikuli:
status: In Progress → Fix Released
milestone: 2.0.0 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.