capture command wrong using vnc

Asked by Christoph Fetzer on 2019-01-24

Hello,

I managed to use sikulix on a vnc connection and are writing due to two points:
- I'd like to share my experience making it work
- I'd like to report an issue.

Point 1, my way:
I have a Windows 7 machine running in a virtual box that I'd like to control with sikulix (1.1.3 BTW) using vnc. My host also is Windows 7.
The first vnc server I tried to use was UltraVNC. Installation procedure worked really smooth, connection using vncviewer from the host worked but with sikulix I was not able to read screen content. capture only showed black screens (yes, I played around with wait(1-5)).
The next vnc-server I played around was tiger vnc because I read success messaged here and in the docs.
Unfortunately the tiger vnc installer let me alone with the firewall settings and did not tell me. Since I found that out and opened the firewall for vnc, I was able to capture the screen and work with sikulix.
My script currently is looking like this:

client = vncStart(ip="192.168.56.104", port=5900 )
#wait( 5 )
img = client.capture()
client.click( client.wait("StartButton.png",10) )
client.type( "Explorer" )
client.click( client.wait("Windows-Explorer.png") )

client.stop();

Point 2:
when I use the capture(filename) variant as capture( client, "Test.png" ) my screenshot file shows a screenshot from the host instead of the virtual box, so I guess there's something like a 'feature' somewhere in there.

Thank you for listeing!

Regards,
Christoph

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2019-02-01
Last reply:
2019-02-18

This question was reopened

RaiMan (raimund-hocke) said : #1

--- capture( client, "Test.png" )
what screen should be captured is taken from the object, that the capture command is issued against.
At script level all un-dotted methods are bound to the quasi-constant object SCREEN, which is Screen(0) by default.
the first parameter (a region - client in your case) only say, which part of the captured screen should be the result.

solution:
use
client.capture( client, "Test.png" )

or say
use(client)
capture( client, "Test.png" )
....
use()

all un-dotted commands between executing use(client) and executing use() (resets to Screen(0)) are going to the VNC screen.
Not guaranteed, that it is reset in all possible cases. So be aware, since the use(some-other-screen) sets the target screen for this IDE session.

Christoph Fetzer (christest) said : #2

Hi!

Thank you very much for the information. I think, your first point (client.capture( client, "Test.png" )) won't help, because docs say: only available in Python scripting (MUST be used as such undotted)
That's why I didn't even try. I'll give it a chance....
But I am very happy with your second solution because I only intend to work with the mut using vnc and that will also help with all other undotted commands.

Thank you very much again!

Regards,
Christoph

RaiMan (raimund-hocke) said : #3

--- because docs say: only available in Python scripting
Yes, you are right.
The capture implementation at the script and Java level differ since the beginning of Sikuli.

... and without breaking the backwards compatibility, there is no easy solution.

For a workaround you have to step down to the Java level, but that is not well documented ;-)
Come back with problems on this.

Christoph Fetzer (christest) said : #4

Thanks RaiMan, that solved my question.

Christoph Fetzer (christest) said : #5

Hm, sorry, i have to come back here.

That's what I do:
client = vncStart(ip="192.168.56.102", port=5900 )
use( client )
wait( 1 )
type( "R", Key.WIN )
wait( 1 )
capture( client, "1.png" )

That's what I get:
[error] script [ Install ] stopped with error at line --unknown--
[error] Error caused by: Traceback (most recent call last): File "D:\x\anovis2\Funktionstest\sikuli\Install.sikuli\Install.py", line 7, in <module> capture( client, "1.png" ) File "C:\Users\CKl\AppData\Roaming\Sikulix\Lib\sikuli\Sikuli.py", line 57, in capture shot = SCREEN.cmdCapture(args) AttributeError: 'org.sikuli.vnc.VNCScreen' object has no attribute 'cmdCapture'

Christoph Fetzer (christest) said : #6

Some more questions, as I am going the first steps inside sikulix now and I'm wondering about some oddities:

Documentation recommends entering text via "paste()". Ever tried this in a cmd-Window?
(Hint: the result you get looks "^V")
Documentation says, you can enter text using type() only on US keyboards and instead it recommends paste(). Ever tried to enter a path in a cmd-shell of a german windows?

Christoph Fetzer (christest) said : #7

Sorry, forgot about one thing: type( "R", Key.WIN ) doesn't seem to work. Also a combination of keyDowns doesn't work.

RaiMan (raimund-hocke) said : #8

--at comment #7:
it must be
type("r", Key.WIN)

your version is
r - SHIFT - WIN

RaiMan (raimund-hocke) said : #9

--- at comment #6
yep, that surely does not work.
Currently I am not really available, but will try to find a workaround during weekend, when I have access to Window system.

RaiMan (raimund-hocke) said : #10

--- at comment #5:
Tut mir leid: Mein Projekt, das unbekannte Wesen ;-)

I have to check and will fix if needed.
I will come back her asap.

RaiMan (raimund-hocke) said : #11

--- at comment #5:
I have checked the situation. None of the mentioned variants will work against a VNC-Screen object at the script level.

this is the only way currently:
image = client.capture() # will create a ScreenImage object
image.saveInBundle("1.png") # will store the shot in the script bundle as _1.png

... the _ underscore saves the image against being deleted on script save.

or short:
client.capture().saveInBundle("1.png")

I see, that the whole thing needs a revision, which will take some time (... but it is on the list now)

Christoph Fetzer (christest) said : #12

Thank you very much for your investigations!

Some experiences from me:

For me it seems, I accidentally used my machine under test using the virtual box GUI instead of the vnc interface. As I am consequently using client.blabla hardly anything is working.

My script:
client = vncStart(ip="192.168.56.102", port=5900 )
wait( 1 )

client.type( "r", Key.WIN )
client.capture().saveInBundle("1.png")

client.keyDown( Key.WIN )
client.keyUp()

wait( 10 )
client.capture().saveInBundle("2.png")

client.wait("Ausfuehren.png",10)
client.capture().saveInBundle("3.png")

Capture 1 shows only the clean desktop (OK, I should wait but there really happens nothing).
Capture 2 shows an incomplete Start menu that doesn't contain what I'd like to search for in the next step (is there a way to give you the file?).
In the virtual box GUI I see a complete Start menu.
When I remove the client. before the wait(), everything s working but not the intended way. (it's supposed to run on a jenkins controlling virtual test machines on vSphere).

By the way I updated to 1.1.4 in the meantime.

A - surprise! With that typing paths suddenly works.

RaiMan (raimund-hocke) said : #13

With VNC understanding the timing of a GUI is even more important than with local screens:

client.type( "r", Key.WIN ) # action
client.capture().saveInBundle("1.png") # check

the action takes some milliseconds to complete, and triggers something on the VNC screen, that in turn might take some time, to be completely reflected in the vnc frame buffer on the local machine (the one running the script)

in this case:
action: will open the run dialog
check: a few milliseconds later will not yet see anything of this.

So if you want to use capture, to afterwards check the content of the VNC screen after an action, you have to wait some time (duration depends on the connection speed).

in this case as an example, the average time to update the frame buffer completely is assumed to be 2 seconds:
client.type( "r", Key.WIN )
wait(2) # might have to be adjusted
client.capture().saveInBundle("1.png")

the intermediate
client.keyDown( Key.WIN )
client.keyUp()
does not make sense for me, since it surely changes, what is seen on the screen.

so assuming that Ausfuehren.png is a shot of the run dialog, this should make more sense:

client = vncStart(ip="192.168.56.102", port=5900 )
wait( 5 )

client.type( "r", Key.WIN )

wait(2)
client.capture().saveInBundle("1.png")
wait( 5 )
client.capture().saveInBundle("2.png")

client.wait("Ausfuehren.png",10)
client.capture().saveInBundle("3.png")

if you then compare the images 1.png, 2.png, 3.png you will get a feeling about the timing of your vnc connection.

Christoph Fetzer (christest) said : #14

Hi!

I mixed two issues, sorry for making this not clear enough:
client.type( "r", Key.WIN )
-> will never show any result, regardlesss how long I wait. Neither on the screen of the virtual box, nor over the vnc connection. Instead I hear some sound out of the virtual machine but I can't relate the source.

client.keyDown( Key.WIN )
client.keyUp()

# wait for the start menu instead of the ausf├╝hren window is a workaround for failing step 1.

client.type( "cmd.exe" )

Issue two:
this wait for the screenshot will fail regardless how long I wait (I went up to 20s). Cause: the captures taken from the vnc connection show an incomplete startmenu with the parts missing, I am searching for ( the entry field at the bottom with the text "Programme/Dateien durchsuchen").
If I use the wait undotted and instead search on my host system in the virtual box window, everything works fine.

Christoph Fetzer (christest) said : #15

Any news on this?
Can I help by explaining more detailed what the problem is for me? I might take some screenshots and upload or make a short video and upload to youtube if it helps.

RaiMan (raimund-hocke) said : #16

Sorry, lost the track.

Just make a new question with the remaining problem.

please give details on environment ( I can test against a Windows 10 machine only).

Can you help with this problem?

Provide an answer of your own, or ask Christoph Fetzer for more information if necessary.

To post a message you must log in.