Best way to read a combobox, or any other control with unselectable content

First of all, I hope I understood what you are asking.

I wouldn't recommend using tesseract for menu items detection. As you have mentioned yourself, it is not reliable. There are 2 more or less reliable options:

1. Create a screenshot (pattern) of every item in your combobox and then just detect it as any other Sikuli pattern.

2. In most cases, the items are more or less static, meaning that they are not expected to disappear or their order to be changed on a regular basis. So what you can do is just use keyboard actions to navigate to a specific list items. For example, if you know that a certain value is coming second in your list, you can simulate DOWN button to be pressed twice:

if item == "itemName1":
    type(Key.DOWN)
elif item == "itemName2":
    type(Key.DOWN)
    type(Key.DOWN)

and so on...

Of course, first you will have to select the combobox itself by clicking somewhere on it and making the items menu active.

Euegene

Thanks for the answer.

1. Overkill, moreless 5570 items.
2. Yes Im doing that, but as I said the content of the combobox is not exactly typed 100% as the postal service (where I got the city, defines it), I need to choose, I need to read and I need to decide if I stick with it.

Something easier than I said, would be process those cities that I found inside the source, don't do anything with that, only search it what I want is there (and if I don't find it, then work with matching ratios) then simply type it inside the combobox (spaces are typed with shift+space).

But as I said, its luck to be able to extract the source from the combobox from a jar file, in other programs I will not have that luck.

As far as I understand:
you are typing some characters in a field of a combobox, which selects an entry and at some time of typing, you want to verify that you want to use the selected entry.

If this is right:
If Region.text() does not work reliable (or can be made reliable, by dealing with some regular exceptions), then you have no chance with Sikuli.
even if you could use images, then you first need a routine, that maps text entries to images and for this you need to know the text entries (here the loop starts ;-)

The only idea:
be satisfied with what you can decide using Region.text() and proceed.
Is there any chance some steps later to verify your choice?
If yes, then go back and choose another entry.

Thanks Eugene Maslov, that solved my question.

I did not used that solution, I ended up processing the source list of cities and made a translation table to simply type them inside the combobox, anyway what you suggested seems interesting.

@Eugene

Just curious how fast/slow does that approach works for you. I tried something similar but it was painfully slow and I decided to drop it.

Thanks

@Eugene S
It's slower than search of one simple image, but not really too slow.
The following problems can slow it down:
 - Old Sikuli 1.0.0 - it couldn't reduce AutoWaitTimeout less than 0.3 sec, in 1.1.0 it's well solved.
 - Many words in the screen, having the same beginning: it has to check many times many items and removes wrong ones from consideration quite late, so the search time increases. In this case, e.g. if many words are sorted alphabetically, it's better to start checking from the end of the string.
 - Words starting from I, 1, or l : findAll understands some long vertical line as many first symbols, and then the check of second symbols takes long.
 - Low similarity used. If it's 0.7 or less, the first pass takes too many items, and then the items are removed at longer stages because of false positive detections, therefore the time grows. I usually use 0.85-0.95 for characters and 0.99 for the space, it works well if the background is the same as in the screenshots of the characters. If different background requires reducing the similarity, then yes, then the time will grow.

I understood that Edmundo has just a few list boxes in the screen. I also usually work with screens where the text is not very dense - just textual items, menus, popups and buttons... But if there is a full page of small text, like in a book, then, I think, having found many first letters, the procedure can search for necessary string much longer than just a ready screenshot of a word.

@Eugene Maslov

Thanks a lot for sharing the experience!
This is very interesting.

I don't know how you use it, but I just tested it. I restrict the region I want to recognize the text (its a line), do a findAll with a similarity of 0.99 for all the characters and sort the matches by position, the result is a phrase. It takes 4 seconds. The worst thing happens if I let a very big empty area at the right, then the space matches 30 times and it takes 10 seconds.

Updating, to who want to know how long it takes. It takes moreless 0.3 seconds to recognize a phrase of a combobox with 100% of accuracy.