SikuliX

how to detect when screen changes - java

Asked by John Henckel

I'm trying to determine the refresh rate of an animation so I need to test when the region changes. I wrote this code in Java to measure how frequently the screen is changing.

Region reg = new Region(500,500,10,10); // small region of the screen
reg.waitVanish(Screen.getPrimaryScreen().capture(reg), 10);

But this crashes with below.

[error] Region: doFind: invalid parameter: org.sikuli.script.ScreenImage@dc67e
[error] ***** Terminating SikuliX after a fatal error 999 *****
It makes no sense to continue!
If you do not have any idea about the error cause or solution, run again
with a Debug level of 3. You might paste the output to the Q&A board.

I set debug=3, but it didn't show anything else. I changed the code to be

reg.waitVanish(new Pattern(Screen.getPrimaryScreen().capture(reg).getFile()), 10);

and this WORKS! but it is very slow, because of writing disk files.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
John Henckel (henckel-jonathan) said : 		#1

Bottom line: what I really want to know is, what is the best way to detect when anything in a region has changed. I want to write code such as

boolean ok = region.waitForChange(10);

in which ok=true means that at least one pixel in that region changed during the 10 seconds.

How would you recommend to implement that?

thanks,

Revision history for this message
Best RaiMan (raimund-hocke) said : 		#2

All image related features of SikuliX currently are based on OpenCV's function matchTemplate, that simply uses some statistical functions, to find out which area of one (at least equal sized but usually larger) (base) image has the most equal pixels as the given (search) image. This means, that from top left at each pixel, the area with the same size as the search image is compared pixel for pixel, to evaluate the so called similarity that ranges from 0 (no pixel is equal) to 1 (all pixels are equal).

the basic find op makes a shot of the given region as base image and runs a matchTemplate using the search image. The area showing the highest similarity is returned as the match region.

There is an implicit standard minimum similarity of 0.7, leading to a FindFailed exception, if the evaluated highest similarity is below this threshold. There are some different ways to define this wanted minimum similarity up to exact (meaning the evaluated similarity must be >0.99).

... and we have exists() that does not throw FindFailed, but returns false (null) and we have the observe() feature, that might even run in background and check different appear, vanish or change (which actually internally uses different OpenCV features) events (look at the docs to understand the features).

Especially for observe it depends on the SikuliX version you are using, how this works on the Java level (not documented).

One more thing: the scan rate: since we are looking for an image in a screenshot, the search operation has to be repeated during the given waiting time with a refreshed screenshot from time to time.
As a good average standard value this scan rate is defined as 3 per second, but can be changed globally or per region if needed (lower or higher). But how many repeats are actually done per second depends on the individual search time for one repeat step.
So if the search time is larger than 0,333 secs (searching in large regions or even big screens), then we do not have our 3 per second and the machine is constantly in the number crunching mode.

In your case, this is the easiest solution (Python code):

observedRegion = Region(500, 500, 10, 10)
regionContent = Pattern(capture(observedRegion)).exact() # to detect differences of some pixels
changed = True
maxTime = 10
endTime = time.time() + maxTime * 1000 # when should it stop
while observedRegion.exists(regionContent, 0): # runs one check lasting some milli seconds
    if time.time() > endTime:
        changed = False
        break
    wait(0.1) # this influences the scan rate

If you need that in Java code: come back later, I want to go to bed now ;-)

Revision history for this message
John Henckel (henckel-jonathan) said : 		#3

Thanks. I'm going to try robot.getPixelColor. I think that will be adequate for my needs, and much faster. Also I could do a screenCapture and compute some kind of CRC against the raw pixel data. I'll try that also.

Revision history for this message
John Henckel (henckel-jonathan) said : 		#4

Thanks RaiMan, that solved my question.

Revision history for this message
RaiMan (raimund-hocke) said : 		#5

--- getColor is available in the Sikuli API 1.0.1+
org.sikuli.script.Location.getColor()

which returns a Java Color object.

I made some tests last year with it: for only one pixel it is rather fast, but if one decides to check some more pixels the time consumed is simply time-for-one-pixel * number-of-pixels, which does not make sense for more than some ten pixels, since the check of an image 10x10 against an even sized or slightly larger region is only some milli-seconds.

--- some kind of CRC against the raw pixel data
this is similar, to what OpenCV matchTemplate does.
 I just made a test with version 1.1.0:

for an observed region of about 20x20 pixels it takes on average about 3 - 5 milli-seconds for a check.
So you have a scan rate of about 200+ per second, which should not be necessary, about 50 should be sufficient.

this is my test script:

img = Pattern("1412875178325.png").exact()
reg = exists(img)

i = 0
start = time.time()
while True:
  if reg.exists(img, 0):
    wait(0.01)
    i += 1
  else:
    print i, (time.time()-start)/i
    break