text search problem when .text() proves it is there

Asked by Donald Hines

I realize there are text search problems with sikuli currently. but I've got one that really baffles me.

I've selected a region of the screen, and then displayed the text in it:
print(scnAct3.text())

this shows:

A
 Add Work

I'm not certain where the "A" is coming from, but clearly the "Add Work" string is present in the region and the ocr is processing it just fine. The VERY next command in my script is:

scnAct3.click("Work")

which results in the following error:
org.sikuli.script.FindFailed: FindFailed: can not find Work on the screen.

Like I said, I understand the OCR is a bit flaky right now, but once the OCR has actually correctly identified some text, shouldn't I be able to click on it? Am I missing some obvious and important step here?

Sikuli is a really cool tool, but I seem to be stuck on this point. Any help will be greatly appreciated.

thanks!

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Donald Hines (donald-c-hines) said :
#1

I did further reading on 710586 and this exact issue is already mentioned in the comments there:

----RaiMan (raimund-hocke)
1. The OCR can recognize the text but the click fails.
    If a screen has text "Search" and if I try click("Search") the click returns failure. But when I try to get the text in the screen using the text() api and print the text, it will print all the strings including the string "Search".
    May be I think we need some improvement in searching the string of text returned by OCR.
----

so, I guess this is a known issue and the answer is "wait" ?

Revision history for this message
RaiMan (raimund-hocke) said :
#2

--- additional A at beginning of Region.text()
Might be, that the region scnAct3 contains some grafic information left of the text "Add Work", that is interpreted as an "A"

--- scnAct3.click("work")
I tried this with the text in your above question like this

r = selectRegion()
print r.text()
r.hover("Work")

and this works.

But if I do the same with "Overview" at the top of this page, it fails with the click, but
r.hover("verview")
works.

Since the current OCR is very dependent on the fonts and fore-/background colors, I guess it is a problem with the capital letter at the beginning of the search text.

Revision history for this message
Donald Hines (donald-c-hines) said :
#3

I don't get success with hover or click, and even when just using "ork" instead of "Work"
The text I'm trying to find is on a colored background, and I would understand if the OCR was missing it. But the OCR is CORRECTLY interpreting the text, as demonstrated by doing the scnAct3.text(). It just can't find the exact SAME text when I do a click, hover, or even a find. I don't understand how the ocr can give the the correct text on a text() command but not on any of these other commands. Very strange.

Revision history for this message
RaiMan (raimund-hocke) said :
#4

The difference is based on the fact, that Region.text() (reading text from a given region on the screen) internally works totally different than searching for some given text somewhere on the screen (or inside a smaller given region).

So if your search need not be on text, because it varies (in content, font, background, ...), it is easier, to either use captured images or even calculated offsets to click on.

e.g. since you want to click on "Work" and already know the region (scnAct3), why not click on an offset from the center of the region?

Revision history for this message
Donald Hines (donald-c-hines) said :
#5

---why not click on an offset from the center of the region?
That might work, I will give it at try.

The reason we are using text search instead of graphic search is because we set up a nice script that ran perfectly on my computer. But as soon as someone with a different screen resolution and theme tried to run the same script it couldn't find anything. So we were hoping to replace most of the graphic searches with text searches that would work with more flexibility across different machines.

Thanks!

Revision history for this message
Donald Hines (donald-c-hines) said :
#6

Well, I'm going to have to give up on this.

I can handle the flaky ocr. It's usually possible to find strings that the ocr can identify fairly reliably. But the bizarre inconsistency of the text search is just too difficult to work around. It seems to work differently from day to day, and code that I had working yesterday will suddenly decide on the next day that it will NOT find strings that region.text() shows the ocr has correctly translated. The same strings that it WOULD find yesterday.

I was able to get around the bugs in the wait("text") function by simply writing my own. But for wait I only needed to identify that a string EXISTS in region.text(). For finding places to click, I MUST be able to figure out where those strings are on the screen. I could write a function to break the screen up into smaller and smaller regions and locate the text that way, but it would be TERRIBLY slow.

I think the way we will have to deal with this is to use graphic searches and either insist that anyone using the tool have the same theme and screen resolution, OR we might run the code in a Virtual Machine.

Sikuli is an AWESOME tool, we just happen to need functionality that isn't fully functional yet.

Revision history for this message
Gayatri (sangasangasanga) said :
#7

#6 is this problem still going on? As i am new to sikuli, not sure if it is still the ocr flaky or i am coding wrongly

Can you help with this problem?

Provide an answer of your own, or ask Donald Hines for more information if necessary.

To post a message you must log in.