TightVNC: click into VNC window does not work

Asked by CHEN

I am facing an issue while using SikUli function Region.Click(). I am using this function to detect and click on a Region in a Window. But this fails always. The weird thing is when i do Region.highlight(), it hightlights the particular region correctly. Immediately after that i do Region.Click() which is not clicking on the Region. But in logs it shows it had clicked on it, actually it did not.
The Region is in a window inside "TightVNC" App. When the click happens, the cursor just moves outside the TightVNC, but in logs the clicked X & Y dimensions shows a value which correctly maps to the Region to be clicked. I dont understand what is wrong here. Can someone please help me?

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

Did that ever work for you somehow?

If yes, what is the difference now?

If not: suure, that the VNC window is ready to accept clicks at that moment? (activated, selected, ...)

Revision history for this message
Julian (kj88) said :
#2

I'm still encountering this issue.
I'm connecting through VNC to a machine. When I do the first region click, it clicks on my machine, and then I do another region click, clicks on the connected machine.
VNCScreen vnc = org.sikuli.script.Sikulix.vncStart("192.168.0.11", 5900, 10, 1000);

vnc.getRow(3, 4).getCol(1, 3).click(); - this clicks on my machine
vnc.find("Unlock").click(); - this clicks on the remote machine

Revision history for this message
RaiMan (raimund-hocke) said :
#3

Uuups, you touched a critical area ;-)

I am sorry, but not all Region methods are "VNC-aware", meaning, that a resulting Region, might "not know", that it is on a VNC screen.

... and I had a look into this group of methods for rastering a region: you might find bugs (result not as expected) besides the mentioned not-VNC-aware problem.

so
vnc.getRow(3, 4).getCol(1, 3)
might be calculated correctly, but is not flagged as being on the vnc-screen

What you can do:

make a method:
public Region asVNC(Region reg, IScreen vnc) {
    reg.setOtherScreen(vnc);
    return reg

and use it:
asVNC(vnc.getRow(3, 4).getCol(1, 3), vnc).click();

ugly, but works.

if the function asVNC is in the same class file as your code:
public Region asVNC(Region reg) {
    reg.setOtherScreen(vnc);
    return reg

and
asVNC(vnc.getRow(3, 4).getCol(1, 3)).click();

less ugly, but now depends on the global variable vnc.

I am sorry, but I doubt, that I get to that problem before beginning of next year.

So you have to live with it for now.

Revision history for this message
RaiMan (raimund-hocke) said :
#4

At least as a quick fix I added these object methods to VNCScreen (latest nightly 1.1.2):

// flag a Region Location as being on this VNCScreen
Region set(Region element)
Location set(Location element)

// get new Region/Location on this VNCScreen
Location newLocation(int x, int y)
Location newLocation(Location loc)

Region newRegion(int x, int y, int w, int h)
Region newRegion(Location loc, int w, int h)
Region newRegion(Region reg)

so it now gets less ugly:
- no extra method needed

vnc.getRow(3, 4).getCol(1, 3).click()

write as:
vnc.set(vnc.getRow(3, 4).getCol(1, 3)).click()

Just watch the nightly 1.1.2 during the next time about VNCScreen related fixes.

Revision history for this message
Julian (kj88) said :
#5

Thank you very much, what worked.
But what about clicking?

If I'm doing something like:

asVNC(vnc.getRow(3, 4).getCol(1, 3), vnc).click(42, 80);

it will still click on my machine instead of the remote one.

Any thoughts on that?
Thank you again!

Revision history for this message
RaiMan (raimund-hocke) said :
#6

a feature:
click(42,80) is not defined in SikuliX

Interesting, that it does something at all - have to check.

The general rule for your situation:

regOrLoc = <some region or location evaluation that is vnc aware>
regOrLoc.click()

Please work with comment #4

Revision history for this message
RaiMan (raimund-hocke) said :
#7

--- about click(42, 80)
at least with 1.1.2 this silently does nothing and returns 0 to signal "no success".

In 1.1.1 it does not behave the same (this was accepted as a bug and fixed in 1.1.2):
It somehow moves the mouse and clicks somewhere and returns 1 (signalling success)

So again: if ever possible use the latest 1.1.2 nightly.

Revision history for this message
Julian (kj88) said :
#8

Thank you! Worked like a charm!

Can you help with this problem?

Provide an answer of your own, or ask CHEN for more information if necessary.

To post a message you must log in.