SikuliX

[2.0.5] IDE: macOS 11: problems with screen capture and mouse actions

Asked by Pedro Pablo

Hello!!!
my environment is:

S.O: Big Sour versión 11.2.3
MacBook Pro (13-inch, 2017, Two Thunderbolt 3 ports)
Pantalla retina 13,3 pulgadas (2560 × 1600)
Resolución utilizada 1680 X 1050 and 1440 X 900
Java 1.8

execution log

***** show environment for 2.0.5 API
user.home: /Users/pedropablo
work dir: /Users/pedropablo
running on macOS 10.16 from a jar
running Java Oracle Corporation 1.8.0_281-b09
java.io.tmpdir: /var/folders/pc/xyqpf_hn7yd2r_z3fbybt1b00000gn/T
app data folder: /Users/pedropablo/Library/Application Support/Sikulix
executing jar: /Users/pedropablo/tagui/src/sikulix/sikulix.jar

When you run:
java -jar /Users/pedropablo/tagui/src/sikulix/sikulix.jar -v -c -v 3

The IDE starts without problems, however when testing the options I have as a result:

.- capture image --> does not work (

.- add imagen --> work ok
.- region --> work ok
.- location --> work ok
.- offset --> work ok

Image capture was working, but not anymore. At the moment of pressing capture screen, I select the section of the screen and then the IDE disappears, I attach log:

[debug] ButtonCapture: capture started
[debug] ScreenImage.store: /Users/pedropablo/Library/Application Support/Sikulix/SikulixStore/lastScreenShot.png
[debug] CapturePrompt: started at (948,865) as L(948,865) on 0
[debug] CapturePrompt: finished at (1181,953) as L(1181,953) on 0
[debug] ButtonCapture: finished
[debug] Image: BufferedImage: (234, 89)

As a test start a project saved a couple of days ago, where I could make the screenshot without problems, when pressing execute, the mouse moves to the image of the project, but no longer clicks, I attach log:

[debug] Jython: use as default region: R[0,0 1680x1050]@S(0)
[debug] Image: BufferedImage: (1680, 1050)
[debug] Image: reused: 1614647849963.png (file:/Users/pedropablo/tagui/flows/comisaria_virtual/BBB.sikuli/1614647849963.png)
[debug] Region: wait: waiting 3,0 secs for 1614647849963.png to appear in R[0,0 1680x1050]@S(0)
[debug] Finder2: makeMat: INT_RGB (1680x1050)
[debug] Finder2: makeMat: 3BYTE_BGR (189x59)
[debug] Finder2: doFindImage: start (stdDev: 194,8939 mean: 454,082683)
[debug] Finder2: doFindImage: in original: %72,3654 (?70) 155 msec
[debug] Finder2: doFindImage: end 156 msec
[debug] highlight M[360,887 189x59]On(0) S 7237 for 2.0 secs
[debug] Region: wait: 1614647849963.png appeared (M[360,887 189x59]@S(0) S:0,72 C:454,916 [292 msec])
[debug] CLICK on L[454,916]@S(0) (2842 msec)
[debug] RunTime:***** running cleanUp *****

Check the settings in accessibility and screen recording. Delete them and add again, I attach LOG:

sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db 'SELECT * from access' | grep kTCCServiceAccessibility

kTCCServiceAccessibility|/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/Support/AEServer|1|2|4|1|||0|UNUSED||0|1615567531
kTCCServiceAccessibility|com.apple.Terminal|0|2|4|1|||0|UNUSED||0|1615837645
kTCCServiceAccessibility|/usr/bin/java|1|2|4|1|||0|UNUSED||0|1615838761

sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db 'SELECT * from access' | grep kTCCServiceScreenCapture

kTCCServiceScreenCapture|com.apple.Terminal|0|2|4|1|||0|UNUSED||0|1615827537
kTCCServiceScreenCapture|/usr/bin/java|1|2|4|1|||0|UNUSED||0|1615838774

Also download the latest version again, deleting the previous one, allowing it to run again and nothing.

The strange thing is that it works with the image of the project that you can save from the previous project.

I need to take screenshots because options like screenshot from OSX do not work, despite changing the image to 100% with 72 pixels.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
Pedro Pablo
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said : 		#1

from a discussion including the sqlite commands, this is my conclusion:

--------------
I now found out:
If you add an app or an executable using the + in the SysPreps, then the entry shows up immediately, when using the sqlite command.

So I guess, that the best approach is, to actively add the respective app or executable using the + in the SysPreps, because any changes done using - or + button in SysPrefs instantly show up using the sqlite command.

So the steps are the following:

in any case stop the Terminal app completely (not only windows!)

when you want to run the SikuliX IDE from a Terminal using Java 8

first in SysPrefs remove all entries with the name java (using - button)
then use the + button to add java again: in the file dialog navigate to the respective java executable (..../bin/java)
do this for Accessibility and Screenrecording
start a Terminal window
run the Java command to start the SikuliX IDE
-------------

See the complete discussion: https://github.com/RaiMan/SikuliX1/issues/429

You use as executable /usr/bin/java, which is a generic executable, that selects the concrete current active Java version from /Library/Java/JavaVirtualMachines/...

My experience is, that the java entries should point to the .../bin/java executable in the respective folder there.

Another option is to use Java 11 from AdoptOpenJDK, which does not need an extra java entry in SysPrefs.

Revision history for this message
Pedro Pablo (pp082) said : 		#2

I did this:

first in SysPrefs remove all entries with the name java (using - button)
then use the + button to add java again: in the file dialog navigate to the respective java executable (..../bin/java)
do this for Accessibility and Screenrecording
start a Terminal window
run the Java command to start the SikuliX IDE

But it didn't work, the IDE keeps disappearing after the screenshot

now I'm going to try:
first in SysPrefs remove all entries with the name java (using - button)
to uninstall java
install the version that you have recommended AdoptOpenJDK

Revision history for this message
Pedro Pablo (pp082) said : 		#3

now

uninstall OK
java -version
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

Install OK
java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.10+9, mixed mode)

then use the + button to add java again: in the file dialog navigate to the respective java executable (..../bin/java)

run the Java command to start the SikuliX IDE
java -jar /Users/pedropablo/tagui/src/sikulix/sikulix.jar -v -c -v 3

***** show environment for 2.0.5 IDE
user.home: /Users/pedropablo
work dir: /Users/pedropablo
running on macOS 10.16 from a jar
running Java AdoptOpenJDK 11.0.10+9
java.io.tmpdir: /var/folders/pc/xyqpf_hn7yd2r_z3fbybt1b00000gn/T
app data folder: /Users/pedropablo/Library/Application Support/Sikulix
executing jar: /Users/pedropablo/tagui/src/sikulix/sikulix.jar
*** classpath dump sikulix
  0: /Users/pedropablo/tagui/src/sikulix/sikulix.jar
*** classpath dump end
***** show environment end
[490 debug] IDE: running with Locale: es
[DEBUG]startUp: IDE: --- Sikuli parameters ---
[DEBUG]startUp: IDE: 1: -v
[DEBUG]startUp: IDE: 2: -c
[DEBUG]startUp: IDE: 3: -v
[DEBUG]startUp: IDE: 4: 3

[519 debug] HotkeyManager: add Capture Hotkey: ⌘+⇧ 2 (50, 5)
[676 debug] HotkeyManager: add Abort Hotkey: ⌘+⇧ C (67, 5)
[702 debug] IDE: Init ScriptingSupport
[705 debug] IDESupport: enter
[2878 debug] Runner: added: AppleScript [script] text/applescript
[2879 debug] Runner: added: PackedSikulix [zip] application/zip
[2879 debug] Runner: init: class org.sikuli.script.runners.AbstractLocalFileScriptRunner: warning: not possible
[2880 debug] Runner: added: SikulixExecutableJar [executablejar] text/jar
[2887 debug] Runner: added: Jython [py] text/jython
[2888 debug] Runner: added: Sikulix [] directory/sikulix
[2889 debug] Runner: added: SikulixJar [jar] text/jar
[2889 debug] Runner: added: PackedSikulix [skl] text/skl
[2889 debug] Runner: added: NetworkRunner [] NET
[2890 debug] Runner: added: Text [txt] text/text
[2892 debug] IDESupport: added: Jython
[2892 debug] IDESupport: added: Text
[2892 debug] IDESupport: exit: defaultrunner: Jython (py)
[2896 debug] IDE: Reading Preferences
[2896 debug] IDE: Adding components to window
[2926 debug] IDE: [I18N] menuRecent
[2948 debug] IDE: [I18N] menuToolAndroid
[2954 debug] IDE: creating tabbed editor
[3065 debug] IDE: creating message area
[3216 debug] IDE: creating combined work window
[3229 debug] IDE: Putting all together
[3229 debug] IDE: Putting all together - after main pane
[3267 debug] IDE: Putting all together - after toolbar
[3269 debug] IDE: Putting all together - before layout
[3971 debug] IDE: Putting all together - after layout
[3972 debug] IDE: Putting all together - Restore last Session
[3972 debug] IDE: Restore scripts from last session
[3972 debug] IDE: Restore Python script: BBB.py
[3982 debug] EditorPane: created 1615924769689
[3988 debug] EditorPane: loadfile: /Users/pedropablo/tagui/flows/comisaria_virtual/BBB.sikuli/BBB.py
[3988 debug] JythonRunner: starting initialization
[4010 debug] EditorPane: InitTab: (text/jython)
[4011 debug] SikuliEditorKit.read
[4027 debug] ImagePath: new BundlePath: /Users/pedropablo/tagui/flows/comisaria_virtual/BBB.sikuli
[4028 debug] EditorPane: setFiles: for: /Users/pedropablo/tagui/flows/comisaria_virtual/BBB.sikuli/BBB.py
[4031 debug] EditorPane: checkSource: started (/Users/pedropablo/tagui/flows/comisaria_virtual/BBB.sikuli/BBB.py)
[4068 debug] Image: loaded: 1614647849963.png (file:/Users/pedropablo/tagui/flows/comisaria_virtual/BBB.sikuli/1614647849963.png)
[4068 debug] Image: cached: 1614647849963.png (32 KB) (# 1 KB 32 -- 0 % of 64 MB)
[4075 debug] Image: reused: 1614647849963.png (file:/Users/pedropablo/tagui/flows/comisaria_virtual/BBB.sikuli/1614647849963.png)
[4081 debug] Screen: initScreens: starting
[4081 debug] Screen: Accessing: GraphicsEnvironment.getLocalGraphicsEnvironment()
[4081 debug] Screen: Accessing: GraphicsEnvironment.getLocalGraphicsEnvironment().getScreenDevices()
[4082 debug] Screen: ScreenDevice 0 has (0,0) --- will be primary Screen(0)
[4082 debug] Screen: Monitor 0: (0, 0) 1680 x 1050
[4084 debug] Screen: initScreens: monitor mouse check

This message does not show it in the IDE
[4084 debug] Screen: initScreens: monitor mouse check
[error] Mouse: not useable (blocked).

I take screenshot , but it didn't work, the IDE keeps disappearing after the screenshot.

---- here disappearing ---
debug] ButtonCapture: capture started
[debug] ScreenImage.store: /Users/pedropablo/Library/Application Support/Sikulix/SikulixStore/lastScreenShot.png
[debug] CapturePrompt: started at (1575,916) as L(1575,916) on 0
[debug] CapturePrompt: finished at (1652,1024) as L(1652,1024) on 0
[debug] ButtonCapture: finished
[debug] Image: BufferedImage: (78, 109)

---- here disappearing ---

check permission in terminal, sqlite format, is OK

sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db 'SELECT * from access' | grep kTCCServiceScreenCapture
   kTCCServiceScreenCapture|com.apple.Terminal|0|2|4|1|||0|UNUSED||0|1615827537
   kTCCServiceScreenCapture|/usr/bin/java|1|2|4|1|||0|UNUSED||0|1615924523

sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db 'SELECT * from access' | grep kTCCServiceAccessibility
   kTCCServiceAccessibility|com.apple.Terminal|0|2|4|1|||0|UNUSED||0|1615837645
   kTCCServiceAccessibility|/usr/bin/java|1|2|4|1|||0|UNUSED||0|1615924512

I am frustrated, The screenshot was working a while ago. when executing a click on a saved image (when it worked ok) it is able to go, without problems, to the image on the screen and press.

please help!! :)

Revision history for this message
Pedro Pablo (pp082) said : 		#6

it's horrible... jajaja..

 it worked when i add sudo

sudo java -jar /Users/pedropablo/tagui/src/sikulix/sikulix.jar -v -c -v 3.

I think the problem is the running user.

thanks !!