Sikuli doesn't find images that definitely exist on screen due to clock differences between host and guest in VM

Asked by Andrew P

Here's the problem I have been trying to figure out for a while now. I wrote quite a few tests in Sikuli and plugged them into our continuous integration system. Every 30 minutes the battery of Sikuli scripts is executed.

Unfortunately, 1 out of 3 times I get a FindFailed error on an image that clearly exists on the screen. To clarify, the same exact script sometimes fails and most of the time doesn't. When it failed, I launched the Sikuli IDE, opened the script, found the offending image and then confirmed that there was indeed a match (in Pattern Settings->Matching Preview) on the screen at that time. The IDE finds it, but the script doesn't. There are no subtle changes in the displayed image on screen between tests, including the background.

I am using the wait() function with a relatively large timeout (30-45 seconds). When I was able to catch the failure, I noticed that the Sikuli just sat there for that length of time and then threw the FindFailed exception.

A few more things that might or might not be important. I am testing a Flash/Flex application in the full screen mode. There might be a few elements on the screen that nave an animated pulsating glow around them (although the image to be matched is always static.)

Any ideas?

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:

This question was reopened

Revision history for this message
RaiMan (raimund-hocke) said :
#1

- try to restrict the relevant search region to the area around the object (always the best idea)
- do not capture any background if possible

print out the match: what is the score?

Send me a screenshot and your captured image.

Revision history for this message
Roman Podolyan (podolyan-roman) said :
#2

I had similar situation sometimes, when setting high similarity, 0.95 , to avoid "false positives", and solved it by loosening similarity for error-prone images to 0.85 .

Revision history for this message
Andrew P (a-ap-u) said :
#3

I did resolve the problem, but not sure that I found the cause or the solution. I was running Sikuli on a VMWare ESXi 5.0 machine (Windows 7). Someone recently changed the graphics adapter on the machine without reinstalling ESXi. Once I reinstalled ESXi, these random failures disappeared. What's interesting is that the virtual machine was exactly the same as before (I didn't overwrite the partition just to see what would happen.)

Thank you for your suggestions, though.

Revision history for this message
Andrew P (a-ap-u) said :
#4

Sorry, I re-opened the question because the problem is still happening. These random failures disappeared for a while and then came back. Moreover, when I was writing some scripts in a virtual machine very similar to the ones that Sikuli tests run on, I encountered the following situation. Suddently a script stopped working while I was in the IDE and suddenly it stopped finding the image on the screen. It wouldn't even find it via Pattern Settings->Matching Preview. I had to exit the IDE and start it again to get the script to work again.

RaiMan, thank you for the offer to take a look at the screenshots. I'll forward them to you. Maybe I don't understand correctly, but how can I print out a match if Sikuli simply doesn't find the image (there is no match, right?)

Roman -- my images use default similarity (0.7), so this isn't the issue.

Revision history for this message
RaiMan (raimund-hocke) said :
#5

@ Andrew
You sent me 2 probes each had a screen shot and the captured images you use.

--1. "web search" area (white font on blue solid background)
Having a look at the images on the pixel level, there are differences at the borders of the white letters between the screen shot and the capture. Even if I reduce the captured image to have as little background as possible, the score is only 0.99.
If I take the capture from the given screen shot, I get a score of 1.0

So my question is: Since there are differences - can it be, that there are situations, where the web search text area really looks different. Do you have screen shots from the failing situations?

--2. the "my stuff" tab
this is a light blue font and has a background with diagonal darker blue lines. the capture seems to be exactly the same as on the screen shot.

So here is the question: might there be situations, that this special background is somewhat displaced or not yet ready, when the search is done. Do you have screen shots from the failing situations?

For now I have no idea, how to optimize your situation.
I would try to find out, what really is on the screen, when the FindFalied situations happen.

Revision history for this message
RaiMan (raimund-hocke) said :
#6

to not let it expire

Revision history for this message
Andrew P (a-ap-u) said :
#7

RaiMan,

Thank you for helping out. Good news is that I figured it out and the image recognition mechanism was not the problem. I was actually hugely relieved that it wasn't the core feature of Sikuli that didn't work well.

The virtual machine that ran Sikuli scripts was being restored automatically from a screenshot. It turned out that the clock on the VMWare ESXi hypervisor was set 4 hours behind by mistake. Windows 7, on the other hand, was set to synchronize time with a time server automatically. It would do it on its own schedule and, as a result, the clock would jump 4 hours forward while Sikuli scripts were running. Sometimes the time change would happen while the wait(image.png, 45) statement was scanning for the target image. Sikuli would then decide that 45 seconds have gone by and throw the FindFailed exception prematurely.

I know that this problem is not going to affect that many people, but when it does, it will drive people crazy, since it is very hard to debug. Sikuli seems to be checking the local time to determine whether the necessary waiting period has gone by. One could implement it using a system timer event that counts the seconds instead of periodically looking whether a target time has been reached.

I haven't tested them, but here are possible scenarios that will most likely cause Sikuli scripts to fail on wait():
- Change from standard to daylight savings time and vice versa
- Manual time change
- NTP time synchronization to respond to clock drift.

Revision history for this message
RaiMan (raimund-hocke) said :
#8

User found out himself.