Possible OCR improvement through screen capture change?

Asked by Mike Bedwell on 2018-03-14

I was comparing and researching a solution for a text grab from the screen. I know, from reading the conversations here on launchpad that ocr is a hit or miss thing. I captured a png from my screen using the mac screen capture tool, and also captured the same area using sikuli. Looking at the results, and the tesseract results, the difference is noticeable. I tried to look through the api code and the robot hooks, and came to the conclusion that maybe the robot library is the limitation, or that i just can't find a capture implementation that might suffice. below is text from a terminal session comparing the two screen captures of the same image.

I'm happy to poke at this further, but do you have any guidance on where(either in the capture interface or the robot library) to look at? I did attempt a patch that i thought might achieve a higher resolution capture, but was unable to test because I broke your code in too many other places attempting it (I mucked with abstractions but didn't update everywhere).

the image captured from the mac is a bit larger-
skexis:imagecaptures.sikuli$ ls -lh 013689.png maccapt.png
-rw-r--r-- 1 mb staff 29K Mar 13 21:52 013689.png
-rw-r--r--@ 1 mb staff 43K Mar 13 21:55 maccapt.png

imagemagick shows that the pixel count is significantly higher in the mac capture (haven't tested this on linux or windows)
skexis:imagecaptures.sikuli$ magick identify 013689.png maccapt.png
013689.png PNG 236x50 236x50+0+0 8-bit sRGB 29618B 0.000u 0:00.009
maccapt.png PNG 476x102 476x102+0+0 8-bit sRGB 43778B 0.000u 0:00.000

tesseract has hugely different degrees of success on the two images. Both of these are without creating any training info for tesseract to use on the odd font that the capture uses.

skexis:imagecaptures.sikuli$ tesseract 013689.png stdout
Warning. Invalid resolution 0 dpi. Using 70 instead.
YEAR < AKHzrxao 652AM
MEROE Dzsm 7350 76961
OWNED Ev KUSH

skexis:imagecaptures.sikuli$ tesseract maccapt.png stdout
YEAR 1. AKHET I-éo. 6:55AM
MEROE DESERT: 380. 6961
OWNED BY: KUSH

Question information

Language:
English Edit question
Status:
Solved
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Solved by:
Mike Bedwell
Solved:
2018-03-14
Last query:
2018-03-14
Last reply:
2018-03-14
RaiMan (raimund-hocke) said : #1

A first thing to note:
You are working on a Retina display, which doubles the pixels both in width and height when showing the image. The Mac image capture tool returns the shown pixels and not the original image pixels. Hence Mac captures have first to be downsized to 0.5, to use them with the SikuliX search feature.
With newer java releases, internally with Robot capture this "Retina effect" is compensated: as a result you get the "real" pixels.
your case:
--- Mac Retina capture: 476x102
--- SikuliX capture (Java AWT Robot): 236x50

The slight differences I guess are because of 2 different manual captures.

RaiMan (raimund-hocke) said : #2

Tesseract:
When using the tesseract command from commandline, then as a guideline from Tesseract you have to take care, that the given image has a resolution of at least 200 dpi better 300 dpi. Another guideline is to turn it to greyscale or even black/white for Tesseract processing.

This means, an image captured with SikuliX usually has a resolution of 72 - 96 dpi (depending on the screen's definition).
This means, a normal screen capture has to be enlarged with factor 3 - 4 before giving it into Tesseract (one might use ImageMagick for that).

In the SikuliX implementation internally the image is enlarged and turned to greyscale before giving it to Tesseract.

The current weaknesses of Region.text() in SikuliX are caused by the fact, that the implementation is from 2011 (unchanged since then written in C++) and based on the Tesseract 2 features. Only a few standard fonts shown on a clean monochrome background with high contrast can be read with higher reliability. There are big problems with variable spacing and special characters.
Tesseract 3 has tons of options, that can be used, to optimise the results including the learning features. All this cannot be used in a simple way in the current implementation (see faq 2709).

Mike Bedwell (bedwell2012) said : #3

Makes sense. I appreciate your time and explanation.