after sikulix script completes my mouse clicks do not work right via TightVNC

Asked by Doug Symes on 2019-01-09

I'm launching my jar from java within a windows batch file that is called by windows task scheduler. However, after the scheduled task completes and the Windows CMD Prompt closes, then if I try to click on a specific task in Windows Tasks list, all the tasks I click on highlight. Its as if I've anchored the first task and am using SHIFT+Click to select several tasks. If I close TightVNC and reopen it all is well. I checked the Keyboard mapping and it is the normal US English (00000409). Please advise.

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2019-01-11
Last reply:
2019-01-12
RaiMan (raimund-hocke) said : #1

should first be a question

RaiMan (raimund-hocke) said : #2

So it seems you are using VNCScreen?

Or are you running SikuliX scripts on a remote machine, that is managed via a TightVNC session?

Doug Symes (dsymes) said : #3

I am running SikuliX scripts on a remote machine, that is managed via a TightVNC session.

Windows Task Scheduler calls the batch file that has this code:

start /i /b /wait c:\users\biuser\desktop\sikuli\runsikulix -f c:\users\biuser\desktop\sikuli\sikuli-log.txt -r c:\users\biuser\Documents\QMG_Live.sikuli
findstr /i /g:c:\users\biuser\desktop\sikuli\log_string_search_list.txt c:\users\biuser\Desktop\Sikuli\sikuli-log.txt
if %errorlevel% == 0 (
   GOTO SENDMAIL
) ELSE (
   GOTO NOTHING
)
:SENDMAIL
 aws ses send-email --to <email address hidden> --from <email address hidden> --source-arn arn:aws:ses:us-east-1:<acct>:identity/qualbe.com --subject "Data-Mining BI Error" --text "Please login to data-mining.qmg.local and check on PBI Desktop to make sure its still running okay. Look at the log file from the automated Sikuli Windows Scheduled Task. The log is located here: c:\users\biuser\Desktop\Sikuli\sikuli-log.txt"
:NOTHING
exit
------ EOF -----

1) What do you make of the VNC issue?
2) Is the above batch file a typical use for SikuliX / Acceptable use?

RaiMan (raimund-hocke) said : #4

Ok, I understand:
- there is a SikuliX environment on the remote machine
- there is a script .sikuli on the remote machine
- also on the remote machine the scriptrun is started via Task Scheduler
- ... and might send an email after termination based on an evaluation of the SikuliX log info

--- What do you make of the VNC issue?
the SikuliX run does not know anything about the fact, that you look at the remote screen using a VNC session.
Hence your problem does not have anything todo with SikuliX, but maybe with your script.

Does the script use the type() feature of SikuliX or even keyUp/keyDown?

--- Is the above batch file a typical use for SikuliX / Acceptable use?
Typical: I do not know.
Acceptable: yes. From SikuliX's point of view it simply runs on a capable machine.

Doug Symes (dsymes) said : #5

I only use type() for hitting ESC b/c at one point when I'm clicking on things in the application, the screen switches as if someone hit the Windows key (this could happen in any of several places) so I inserted a check for it and hit ESC to return to the application.
Here is the whole PY file from SikuliX - as you can see I'm almost always looking for images exclusively. I will tack on the log I'm getting periodically for your perusal:

#Launch PowerBI and open the QMG-Live PBIX file.
doubleClick(Pattern("1544653159352.png").similar(0.82))
wait(Pattern("1544656197122.png").similar(0.61),60)
click(Pattern("1544215649288.png").similar(0.80))
if exists(Pattern("Server_Desktop.png").similar(0.80)):
    type(Key.ESC)
if exists(Pattern("1544215829534.png").similar(0.74),5):
    click(Pattern("1544215896229.png").similar(0.89),5)
if exists(Pattern("Server_Desktop.png").similar(0.80)):
    type(Key.ESC)
waitVanish(Pattern("1544216116787.png").similar(0.80),80)
if exists(Pattern("Server_Desktop.png").similar(0.80)):
    type(Key.ESC)
wait(Pattern("1544216171532.png").similar(0.57),20)
if exists(Pattern("Server_Desktop.png").similar(0.80)):
    type(Key.ESC)
click(Pattern("1544216198461.png").similar(0.80),20)
if exists(Pattern("Server_Desktop.png").similar(0.80)):
    type(Key.ESC)
if exists(Pattern("1544824408767.png").similar(0.80),10):
    click(Pattern("1544826489006.png").similar(0.80),5)
else:
    if exists(Pattern("Server_Desktop.png").similar(0.80)):
       type(Key.ESC)
    if exists(Pattern("1544718781015.png").similar(0.80),1):
       click(Pattern("1544718781015.png").similar(0.80),1)
if exists(Pattern("Server_Desktop.png").similar(0.80)):
    type(Key.ESC)
Found = False
i=0
while i < 5 :
    i = i+1
    if i>1:
       if exists(Pattern("1544718936612.png").similar(0.80),5):
          click(Pattern("1544718936612.png").similar(0.78),5)
    if exists(Pattern("1544719057841.png").similar(0.82),5):
        Found = True
        break
if exists(Pattern("Server_Desktop.png").similar(0.80)):
    type(Key.ESC)
if Found:
    click(Pattern("1544719165835.png").similar(0.78),6)
    if exists(Pattern("Server_Desktop.png").similar(0.80)):
        type(Key.ESC)
    click(Pattern("1544719189664.png").similar(0.86),6)
    if exists(Pattern("Server_Desktop.png").similar(0.80)):
        type(Key.ESC)
    if exists(Pattern("1544824573671.png").similar(0.80),6):
        click(Pattern("1544719214036.png").similar(0.82),6)
    if exists(Pattern("Server_Desktop.png").similar(0.80)):
        type(Key.ESC)
    wait(Pattern("1544719259439.png").similar(0.78),180)
    if exists(Pattern("Server_Desktop.png").similar(0.80)):
        type(Key.ESC)
    click(Pattern("1544719321513.png").similar(0.75),5)
    if exists(Pattern("Server_Desktop.png").similar(0.80)):
        type(Key.ESC)
    if exists(Pattern("1544719633417.png").similar(0.80),5):
        click(Pattern("1544719633417.png").similar(0.80),5)
    if exists(Pattern("Server_Desktop.png").similar(0.80)):
        type(Key.ESC)
    if exists(Pattern("1544826115697.png").similar(0.79),3):
        click(Pattern("1544826115697.png").similar(0.79),3)
if exists(Pattern("1544719633417.png").similar(0.80),3):
    click(Pattern("1544719633417.png").similar(0.80),3)
if exists(Pattern("1544826115697.png").similar(0.79),3):
    click(Pattern("1544826115697.png").similar(0.79),3)
#the end

=================== Sikuli Log File (usually there are not errors) ===============================
[error] script [ C:\Users\biuser\Documents\QMG_Live.sikuli ] stopped with error at line --unknown--
[error] Error caused by: Traceback (most recent call last):
  File "C:\Users\biuser\Documents\QMG_Live.sikuli\QMG_Live.py", line 2, in <module>
    doubleClick(Pattern("1544653159352.png").similar(0.82))
  Line 2761, in file Region.java

 at org.sikuli.script.Region.wait(Region.java:2761)
 at org.sikuli.script.Region.find(Region.java:2326)
 at org.sikuli.script.Region.getLocationFromTarget(Region.java:3220)
 at org.sikuli.script.Region.doubleClick(Region.java:3767)
 at org.sikuli.script.Region.doubleClick(Region.java:3752)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
org.sikuli.script.FindFailed: FindFailed: 1544653159352.png: (68x89) in S(0)[0,0 1920x1080] E:Y, T:3.0
  Line 2761, in file Region.java

RaiMan (raimund-hocke) said : #6

I am sorry, but I have no idea, what is happening in your case.

BTW: are you using the latest build of 1.1.4?

Doug Symes (dsymes) said : #7

1.1.3 SikuliX
Java 1.8.0.191

What is the upgrade process?

Doug Symes (dsymes) said : #8

In addition to finding out the upgrade process to go from SikuliX 1.1.3 to 1.1.4;
Can you tell me how you interpret this log data?

[error] script [ C:\Users\biuser\Documents\QMG_Live.sikuli ] stopped with error at line --unknown--
[error] Error caused by: Traceback (most recent call last):
  File "C:\Users\biuser\Documents\QMG_Live.sikuli\QMG_Live.py", line 2, in <module>
    doubleClick(Pattern("1544653159352.png").similar(0.82))
  Line 2761, in file Region.java

 at org.sikuli.script.Region.wait(Region.java:2761)
 at org.sikuli.script.Region.find(Region.java:2326)
 at org.sikuli.script.Region.getLocationFromTarget(Region.java:3220)
 at org.sikuli.script.Region.doubleClick(Region.java:3767)
 at org.sikuli.script.Region.doubleClick(Region.java:3752)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
org.sikuli.script.FindFailed: FindFailed: 1544653159352.png: (68x89) in S(0)[0,0 1920x1080] E:Y, T:3.0
  Line 2761, in file Region.java
========================== EOF =================================

Is it as simple as Whatever is on line 2 had an error? Is it that the image listed on the last line of the log was just not found on the screen and the code timed out?

RaiMan (raimund-hocke) said : #9

There is no update process - simply follow the steps on the download page.

--- FindFailed: exactly

You might read the docs, to finally understand and see other possibilities.

Doug Symes (dsymes) said : #10

I'll lean in and see if I can't figure this out. Thanks much.

Can you help with this problem?

Provide an answer of your own, or ask Doug Symes for more information if necessary.

To post a message you must log in.