Accuracy of image recognizers and actions sikuli.

Asked by born

Hello!
How to get sikuli with 100% probability to recognize patterns?
Used a variety of ways, but every time sikuli recognizes a pattern in different ways - that do not successfully successfully.
I hope video will help you understand what I'm doing.
Link below:

http://narod.ru/disk/8089334001/sikuli.avi.html

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

Interestingly people in many cases try to use Sikuli to automate menu-clicking, without being aware, that it would be much easier to use the existing keyboard-shortcuts (in this case ctrl-N). Menus of an application with a windows standard GUI can always be easily automated using the keyboard (this is how many other automation tools do their job).

a sequence
switchApp("part of window title of app")
type('n',KEY_CTRL)

is cheaper, faster and more secure.

In your video, the one situation where it did not work, was caused by the fact, that (I don't know why) the app was inactive (not focused) and so the first click only activated the window, but did not open the menu - so the find of the second click failed.

So just put a switchApp() or App().focus() before the first click. I always give some additional time to the app, to show its window.

So this would be my solution:

switchApp()
wait(0.5)
type('n',KEY_CTRL) (or the clicks here ;-)

One more thing:

It is waste of time and resources to use the whole screen. The search operations should be restricted at least to the app window:
myApp = App("part of window title of app")
reg = myApp.window()
reg.focus()
reg.highlight(1) # a nice alternative to a dumb wait(1)
with reg:
   click()
   ....

Revision history for this message
born (born127) said :
#2

Yes, I understand that c better use something like: type ('n' key_ctrl) but in my case, not all menu items can be so called.
You can certainly use alt + (underlined letter in the menu) but the problem I described above is found not only in the menu. Sikuli not always accurately determine where to click. Little bit helps design similar () and targetoffset () but in the case of the menu, these tricks did not work. By the way, I'm not sure that it has focus, is seen in 35 seconds video - an application receives the focus when you click on the main menu.
Or I do not understand?
Thanks for the tip on the use of the region, but I did it here is such a problem:

Code:

switchApp ("notepad")
wait (1)
myApp = App ("Untitled - Notepad")
reg = myApp.window ()
reg.focus ()
reg.highlight (1)
with reg:
 click(img)
Error:

[log] App.focus notepad (0) # 0
[debug] getWindow: null [error] Stopped [error] Error occurs on line 5 [error] error message: Traceback (most recent call last): File "C: \ DOCUME ~ 1 \ SEMENO ~ 1 \ LOCALS ~ 1 \ Temp \ sikuli-tmp6006180189247011497.py ", line 5, in reg.focus () AttributeError: 'NoneType' object has no attribute 'focus'
[debug] setTargetOffset: (-58,0)
[debug] setTargetOffset: (-21, -27) [debug] setTargetOffset: (-43, -13)

I apologize for my English.

Revision history for this message
RaiMan (raimund-hocke) said :
#3

Sorry, it is always a risk, to just type script code here, without having it tested ( not reg.focus() but myApp.focus() )

Here you have a tested snippet, that you have to adapt to your english environment (I think: Editor = Notepad, Datei = File).

And you get an idea of an alternative approach for your menu problem.

myApp=App("Editor")
myApp.focus()
reg = myApp.window()
reg.highlight(2)
menubar = reg.above(1).below(30).below(20)
menubar.highlight(2)
mDatei = menubar.find("Datei") # text search
mDatei.highlight(2)
click(mDatei)
wait(3) # to give a chance to see the effect

--- you say:
 I'm not sure that it has focus, is seen in 35 seconds video - an application receives the focus when you click on the main menu. Or I do not understand?

Yes, should normally be so. But I had cases with Sikuli clicks, where it did not work and either a (Sikuli) doubleclick or a switchApp() was needed. (This problem is also dicussed here from time to time)

--- you say:
 but the problem I described above is found not only in the menu. Sikuli not always accurately determine where to click

per definition and design - Sikuli cannot be "accurate" when searching, since this process is based on similarity. If something is found, Sikuli tells you, how similar it is. Only 1.0 normally means that the match is pixel by pixel identical with the search pattern. So if you cannot be sure, you have to check the result of a search. So e.g., if you expect something to be exactly matched, it is not wise, to allow the search with a similarity below 0.9 or even 0.99.

--- my experience and recommendation
the best way to assure your searches is to restrict the search region to the area, were you expect the visual object to appear. Next is setting a high similar() value and check the match.
I normally use m = exists(image) and check m before going further, e.g.
menu = exists(menu_image)
if not menu:
   print "menu not found"; exit()
click(menu)

Can you help with this problem?

Provide an answer of your own, or ask born for more information if necessary.

To post a message you must log in.