Click on text captured from Screen().text()

Asked by Hina on 2018-03-07


I have got text using string1= Screen().text() function.
Can we extract coordinates of some sub-string ? and then by using these coordinates is it possible to click(or apply any operation)on the sub-string?

e.g. I have text on screen with button Yes.
Are you sure you want to continue?
No Yes
will return
Are you sure you want to continue?
No Yes

the simple OCR text search approach isn't successful in my application case :(

I want to click on Yes button without using image.

Question information

English Edit question
Sikuli Edit question
No assignee Edit question
Last query:
Last reply:
RaiMan (raimund-hocke) said : #1

you can try using
someRegion.find("some text")

which searches for a text and returns a match region, that can be clicked.

... but be aware of all the weaknesses of the text feature implementation.

Hina (hina-amir) said : #2

The region.find("text") won't work as expected.
I tried to print text from region and it displayed
"No Administrator account found " as "fldmin istrator account found "

If I search for Administrator it will throw Find failed error.
I tried with "account found" and result is same.
might be text feature weakness :(

RaiMan (raimund-hocke) said : #3

... might be text feature weakness :(
surely true

... but if you want to step into the gory details:

Hina (hina-amir) said : #4

I used Screen().text() instead it gives you better results ... it had difficulty reading first letter of each sentence on CMD but otherwise it worked well. The only issue is I can't narrow down sear to specific coordinates on screen. It will take the monitor extents which makes my script slower. Let me know if ther's any way to change focus on particular the screen coordinates.

RaiMan (raimund-hocke) said : #5

instead of
Screen().text() (BTW: Screen(). not necessary - simply text() is enough for whole screen)

you might use

after before having evaluated someRegion with other SikuliX features
simpliest as
someRegion = Region(x, y, w, h)

someRegion = selectRegion() # interactive

or with the many other features, that aloow to define a region relative to other regions or locations on the screen.

Can you help with this problem?

Provide an answer of your own, or ask Hina for more information if necessary.

To post a message you must log in.