vnc: flaky Pattern matching

Using the latest SikuliX 1.1.2 ?

I'm using 1.1.1

I'm now trying to build 1.1.2 from git, but I'm getting failure (unfortunately, I'm not a java wiz):

$ mvn clean install
... snip...
[INFO] Reactor Summary:
[INFO]
[INFO] sikulix1 .......................................... SUCCESS [0.321s]
[INFO] sikulixapi ........................................ SUCCESS [8.460s]
[INFO] sikulixsetupAPI ................................... SUCCESS [4.809s]
[INFO] sikulixsetup ...................................... SUCCESS [2.682s]
[INFO] sikulix ........................................... FAILURE [1.675s]
[INFO] sikulixsetupIDE ................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.100s
[INFO] Finished at: Wed Feb 21 07:42:31 EST 2018
[INFO] Final Memory: 33M/454M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project sikulix: Compilation failure: Compilation failure:
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[6,1] package java.awt.desktop does not exist
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[8,39] cannot find symbol
[ERROR] symbol: class AboutHandler
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[8,53] cannot find symbol
[ERROR] symbol: class PreferencesHandler
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[8,73] cannot find symbol
[ERROR] symbol: class QuitHandler
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[8,86] cannot find symbol
[ERROR] symbol: class OpenFilesHandler
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[23,25] cannot find symbol
[ERROR] symbol: class OpenFilesEvent
[ERROR] location: class org.sikuli.idesupport.IDEMacSupport
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[32,27] cannot find symbol
[ERROR] symbol: class AboutEvent
[ERROR] location: class org.sikuli.idesupport.IDEMacSupport
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[37,33] cannot find symbol
[ERROR] symbol: class PreferencesEvent
[ERROR] location: class org.sikuli.idesupport.IDEMacSupport
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[42,37] cannot find symbol
[ERROR] symbol: class QuitEvent
[ERROR] location: class org.sikuli.idesupport.IDEMacSupport
[ERROR] /home/atmdev/Desktop/sikulix/SikuliX-2014/IDE/src/main/java/org/sikuli/idesupport/IDEMacSupport.java:[42,50] cannot find symbol
[ERROR] symbol: class QuitResponse
[ERROR] location: class org.sikuli.idesupport.IDEMacSupport
[ERROR] -> [Help 1]

no need to build it.

download from the nightly page:
https://raiman.github.io/SikuliX-2014/nightly.html

... sorry, I just realized, that the link on sikulix.com is broken :-(

I think I successfully compiled 1.1.2 (after deleting the Mac code)

Finding patterns via vnc is still flaky.

ok, then I cannot help you - sorry.
I can only test some basics locally.
... and VNC is not my priority.

you have to implement some debugging code, to find out, wether the internal screen snapshot is not correct.

Ok, I'm pretty certain now that Sikuli works only with the initial screen capture, and does not get updates when the screen changes.

I'm going to try to explain. Assume strings are actually bitmaps.

Suppose I have this on the vnc screen: "Button1 Button2 Button3"

v = vnsStart(....)
wait(3.0)
use(v)
click("Button1")
# Assume after the click, the screen changes to: "Button1 Button4 Button5"
click("Button3")
# "Button3" is no longer on the screen, but sikuli goes and click on "Button5"

So, it looks like updates to screen are not being received by sikuli.

If I close, and then vncStart again between each operation, then it works with the most up to date screen.

Not sure, what you are talking about.

click("Button1")
means, that locally an image "Button1.png" is loaded from bundle-path.
If this file does not exist, SikuliX switches to text-search "Button1" (if text search is switched on).

If you are talking about that, than you have to be aware about the weaknesses of the text feature implementation, which might find Button3 as Button5.

If you are not talking about text search, then I do not understand:
Assume strings are actually bitmaps.

Do you mean the strings are image file names?

Sorry, let's try a different notation:
[ButtonX] is a notation to mean a bitmap containing the text ButtonX
"buttonX.png" is a file containing [ButtonX]

The files do exist.

Code version 1:
# Screen has [Button1 Button2 Button3]
v = vncStart(...)
wait(3.0)
use(v)
click("button1.png") # Changes screen to [Button1 Button4 Button5]
click("button3.png") # Clicks on screen where Button5 is, even though button3.png is a bitmap of [Button3]. Click would have failed if sikulix had the most up to date screenshot

If I change the code to version 2:
# Screen has [Button1 Button2 Button3]
v = vncStart(...)
wait(3.0)
use(v)
click("button1.png") # Changes screen to [Button1 Button4 Button5]
i = capture()
print i.getFile() # output file shows [Button1 Button2 Button3] - so, sikulix does not have the latest screencap
v.close()
v = vncStart(...)
wait(3.0)
use(v)
click("button3.png") # throws error, because [Button3] is not on the screen. restarting vnc picked up the correct screencap.

So, vncStart does get a correct initial screen cap representing the current state of the display, but any update to the display after that are not seen by sikulix. Only stopping and restarting vnc picks up the up to date screen cap.

Hopefully this is clearer.

ok, finally understood.
I will check tomorrow.

ok, made some tests.

It principally works, but there is indeed a problem, which is implemented ;-)

Especially with web-content some image might already be visible on the screen, but is not yet in its final place.
Same goes for the other situation, where something should change, but is still visible in that moment.

Since the frame buffer is constantly working in the background refreshing the content from remote and the taking of screenshots is not synchronized with a somehow "stable" frame buffer, these effects might happen.
Only inserted wait()'s can help, to give some time for getting stable/changed.

so in your case:
click("button1.png")
wait(1) # adjust the waitime as needed - shorter or longer
click("button3.png")