VNCScreen.wait() does not catch image change

Asked by Pavel on 2018-12-24

Hi.

I'm using sikuliapi-1.1.4.jar (and also tried on 1.1.3-SNAPSHOT). The system is on remote server and I'm using build in VNC client.
Seems there is an issue with screen refresh while waiting for set image, which appears on screen after wait() method start.
Here an example:
Assume, there is a system with response delay. Script clicks on button and waits for new images. The wait() method is started and checking screen for a few seconds when expected image appears on screen (new form). After image appears on screen it is still not visible for wait() and finally FindFailed exception is thrown. The screenshot which is taken in catch clause saves screen without appeared image, the same as it was at wait() method start.
As I wrote before, It seems like wait() by itself does not recapture screen.
Is that correct, or that is and issue in new version?

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2018-12-27
Last reply:
2018-12-31
RaiMan (raimund-hocke) said : #1

just checked - works as expected (when obeying the special aspects of VNCScreen and dedicated regions/locations)

please post some offending snippet.

Pavel (pe-test) said : #2

I tried a couple options, both have the same result.
option 1:
public Match waitForImage(VNCScreen vnc_old, String image, Region reg, double waitTimeout, float score) {
  System.out.println("Waiting for image: " + image + " in a specific region: " + reg.getX() + ", " + reg.getY() + " " + reg.getW() + "x" + reg.getH());

  vnc_old.setRect(reg.x, reg.y, reg.w, reg.h);

  Pattern pat = new Pattern(image);
  pat.similar(score);
  Match found = null;

  try {
   found = vnc_old.wait(image, waitTimeout);
   System.out.println("Match: " + found.getScore());
   System.out.println("Found image region: " + found.x + "," + found.y + " " + found.w + "x" + found.h);
  } catch (FindFailed e) {
   e.printStackTrace();
   screenShot(vnc_old, reg);
   throw new RuntimeException("Failed to find a match for image and click it. " + image);
  }

  vnc_old.hover(); // printing found region coordinates
  return found;
 }

option 2:
public Match waitForImage(String image, Region reg, double waitTimeout, float score) {
  int port = Integer.parseInt(5950);
  VNCScreen vnc = connectVNC(HOST, port, PASSWORD, 10, 5000);

  System.out.println("Waiting for image: " + image + " in a specific region: " + reg.getX() + ", " + reg.getY() + " " + reg.getW() + "x" + reg.getH());

  vnc.setRect(reg.x, reg.y, reg.w, reg.h);

  Pattern pat = new Pattern(image);
  pat.similar(score);
  Match found = null;

  try {
   found = vnc.wait(image, waitTimeout);
   System.out.println("Match: " + found.getScore());
   System.out.println("Found image region: " + found.x + "," + found.y + " " + found.w + "x" + found.h);
  } catch (FindFailed e) {
   e.printStackTrace();
   screenShot(vnc, reg);
   throw new RuntimeException("Failed to find a match for image and click it. " + image);
  }

  vnc.hover(); // printing found region coordinates
  return found;
 }

Commenting vnc.setRect(reg.x, reg.y, reg.w, reg.h); also does not bring any changes to final result.

RaiMan (raimund-hocke) said : #3

I have no idea, what your problems are.

My test with the latest build 1.1.4: (transscripted to my habits)

The image to search for is seen near the top of the remote screen.

      //public Match waitForImage(VNCScreen vnc_old, String image, Region reg, double waitTimeout, float score) {
      //System.out.println("Waiting for image: " + image + " in a specific region: " + reg.getX() + ", " + reg.getY() + " " + reg.getW() + "x" + reg.getH());

      VNCScreen.startUp(0); //do not wait after start (standard would be 3 secs)
      VNCScreen vs = VNCScreen.start("192.168.2.121");

      Region reg = new Region(0, 0, 1600, 300);
      vs.setRect(reg.x, reg.y, reg.w, reg.h);

      String vimg = "vncChrome";
      Pattern pat = new Pattern(vimg);
      //pat.similar(score); //match score is > 0.99 ;-)
      Match found = null;

      try {
        found = vs.wait(pat); //standard waiting 3 secs
// System.out.println("Match: " + found.getScore());
// System.out.println("Found image region: " + found.x + "," + found.y + " " + found.w + "x" + found.h);
        p("found: %s", found); //my formatting print function
      } catch (FindFailed e) {
        log(-1, "not found: %s", pat); // class Debug
// e.printStackTrace();
// screenShot(vs, reg);
// throw new RuntimeException("Failed to find a match for image and click it. " + image);
      }
      //vs.hover(); // printing found region coordinates
      //return found;
      // }

The printout when using setRect():
found: M[836,3 74x88]@S(NonLocal, 1280x300) S:1,00 C:873,47 [158 msec]

the search-time will be about 1 sec with fullscreen (no setRect())

BTW: setRect() is not recommended for restricting the search region for one search.
Better is to use the class Region:

Region reg = vnc.newRegion(givenRegion); // dedicates the Region to the given VNCScreen
found = reg.wait(image);

Pavel (pe-test) said : #4

RaiMan, are there any log records, which could help? And which level if so?

RaiMan (raimund-hocke) said : #5

Debug.on(3); //switch on normal debug level
Debug.off(); //switch off again

Pavel (pe-test) said : #6

yeah, I've found this in documentation, but does debug has records about screen capture?

RaiMan (raimund-hocke) said : #7

With the latest build from today, this snippet: (build#167)

      Debug.on(3);
      VNCScreen vs = VNCScreen.start("192.168.2.121");
      Region reg = vs.newRegion(0, 0, 1500, 300);
      reg.find("vncChrome");
     System.exit(0);

gave this log:

CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8
[4662 debug] VNCScreen: start: R[0,0 1920x1200]@S(VNC 192.168.2.121:5900)
[4733 debug] Image: loaded: vncChrome.png (file:/Users/raimundhocke/IdeaProjects/sikulix1/API/src/main/resources/ImagesAPI/vncChrome.png)
[4733 debug] Image: cached: vncChrome.png (19 KB) (# 1 KB 19 -- 0 % of 64 MB)
[4733 debug] Region: find: waiting 0 secs for vncChrome.png to appear in R[0,0 1500x300]@S(VNC 192.168.2.121:5900)
[4737 debug] VNCScreen: capture: (0,0) 1500x300 on R[0,0 1920x1200]@S(VNC 192.168.2.121:5900)
[4743 debug] RunTimeAPI: libsExport: folder exists: /Users/raimundhocke/Library/Application Support/Sikulix/SikulixLibs (1.1.4 - 201812311549)
[4746 debug] RunTimeAPI: loadLib: libopencv_java342.dylib (success)
[4748 debug] Finder2: makeMat: INT_RGB (1500x300)
[4762 debug] Finder2: makeMat: 3BYTE_BGR (74x88)
[4764 debug] Finder2: doFind: start (stdDev: 161,6029 mean: 317,441032)
[4814 debug] Finder2: doFind: in original: %100,0000 (?70) 48 msec
[4815 debug] Finder2: doFind: end 49 msec
[4818 debug] Region: find: vncChrome.png appeared (M[836,3 74x88]@S(VNC 192.168.2.121:5900) S:1,00 C:873,47 [83 msec])
[4819 debug] RunTimeAPI: ***** final cleanup at System.exit() *****
[4819 debug] VNCScreen: stopping all
[4819 debug] VNCScreen: stopping: R[0,0 1920x1200]@S(VNC 192.168.2.121:5900)
[4834 debug] FileManager: deleteFileOrFolder:
/var/folders/pc/t43tqvr56xx7kxz3m6z5g4rw0000gn/T/Sikulix_694096369

Can you help with this problem?

Provide an answer of your own, or ask Pavel for more information if necessary.

To post a message you must log in.