How do i import python modules into sikuliX script ?

Asked by ONG YI CHONG on 2019-01-07

I know this question has been asked many times, I have read the posts but i am still confused as to what i should do.

How do i import other python libraries like cv2, matplotlib for use in my sikuliX script ? I am using the SikuliX IDE

Question information

Language:
English Edit question
Status:
Answered
For:
Sikuli Edit question
Assignee:
No assignee Edit question
Last query:
2019-01-10
Last reply:
2019-01-10

This question was reopened

RaiMan (raimund-hocke) said : #1

Python libs can only be imported into SikuliX, if they are completely written in plain Python.
If they rely on any C-based components (native libraries), they cannot be imported.

The reason behind: the scripting language is Python, but we use the Java based Jython interpreter (language level 2.7).

cv2 definitely and matplotlib most probably are candidates for being non-importable (C-based stuff contained).

What is your intention?

ONG YI CHONG (calveeen) said : #2

I would like to use opencv's contour detection to identify UI elements and write a sikuli script to click on those elements.

Is there any other way around it ? I thought it would be possible since sikuliX uses opencv libraries as well.

RaiMan (raimund-hocke) said : #3

The OpenCV API is available in SikuliX at the Java Level.

Since we run the Jython interpreter, Java classes can be accessed easily from the Python script level after import:

the principle:

import org.opencv.core.Mat as CVMat
aMat = CVMat();

... would create a Jython object representing an empty Mat

the JavaDocs: https://docs.opencv.org/3.4/javadoc/index.html

If you have any Python or even C++ examples from the net, it is rather easy, to transcript them to the Java-eco-system.
Surely you will find some Java based examples too.

ONG YI CHONG (calveeen) said : #4

Hi RaiMan thanks so much for your reply ! Does that mean that sikuliX can accept both python and java syntax ?

if i call print "hello world" and system.out.println("hello world") both would be valid ?

RaiMan (raimund-hocke) said : #5

--- print "hello world" and system.out.println("hello world")
are principally the same (though there might be String encoding challenges ;-)

... but every used Java class must be imported before usage.

my preferred technique:
import java.lang.system.out as JSysOut #once per script
JSysOut.println(...)
JSysOut.println(...)
JSysOut.println(...)
JSysOut.println(...)
...

but this also
import java
java.lang.System.out.println("hello world")

more can be found in the net.

ONG YI CHONG (calveeen) said : #6

So even if the sikuliX directory contains .py file, java classes can still be imported and used ?

RaiMan (raimund-hocke) said : #7

Classes bundled in the SikuliX jar can just be imported.

For Classes in other jar's, these jars can be put into the <app-data>/extensions folder, from where they would automatically be put on the classpath.

ONG YI CHONG (calveeen) said : #8

What do you mean by "Classes bundled in the SikuliX jar can just be imported." ?

Is there a place where I can find the list of classes bundled in the SikuliX jar ?

ONG YI CHONG (calveeen) said : #9

Hi RaiMan, I cant seem to import org.opencv.imgcodes.Imgcodecs and org.opencv.highgui.HighGui though.

RaiMan (raimund-hocke) said : #10

--- having in the IDE:
import org.opencv.highgui.HighGui as HG
import org.opencv.imgcodecs.Imgcodecs as IC
print HG
print IC

--- works and prints:
<type 'org.opencv.highgui.HighGui'>
<type 'org.opencv.imgcodecs.Imgcodecs'>

ONG YI CHONG (calveeen) said : #11

Oh i was trying to run sikuliX from the command line. And it gave me an import error. Do i have to append the jar file path to sys.path.append ?

RaiMan (raimund-hocke) said : #12

does not matter.

really using latest 1.1.4?

ONG YI CHONG (calveeen) said : #13

nope i am using 2018-12-14 one. I shall go and download the new one !

ONG YI CHONG (calveeen) said : #14

I have downloaded and added the sikulixapi.jar file into intellij's library module. I was wondering if there is any advantage in running sikuliX's code in java rather than using the SikuliX IDE ?

RaiMan (raimund-hocke) said : #15

--- advantage in running sikuliX's code in java
if you prefer to program in Java and you have more skills than with Python scripting, yes then it might make sense.
The disadvantage: you have to have another concept for capturing/storing/naming your images (image path feature)
But you might always use the SikuliX IDE for image handling only and simply put the .sikuli folder(s) on the image path.

ONG YI CHONG (calveeen) said : #16

I am getting another error when i try to use any of the org.opencv libraries in the sikuliX IDE. Do i need to append sys.path or sth ?

ONG YI CHONG (calveeen) said : #17

import java.lang.System.out as jSysOut
import org.opencv.core.Mat as CVMat
import org.opencv.imgcodecs.Imgcodecs as IC
import org.opencv.highgui.HighGui as HG

print(HG)
print(IC)
## Load image
aMat = CVMat()

output:
<type 'org.opencv.highgui.HighGui'>
<type 'org.opencv.imgcodecs.Imgcodecs'>
[error] script [ automateScreenShotV3.0 ] stopped with error in line 11
[error] java.lang.UnsatisfiedLinkError ( java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat()J )

Im getting an unsatisfied link error when i try to assign aMat() = CVMat()

RaiMan (raimund-hocke) said : #18

ok, this is because the native libraries are not loaded.

I did not forsee, that someone might use the OpeneCV features such isolated as you do.
In a "normal" SikuliX usage the needed libraries are loaded lazily, when respective features are first used in a script/program.

For you as a workaround put this together with the imports:
import org.sikuli.script.RunTime as RT
RT.loadLibrary(RT.libOpenCV);

If you finally succeed with your approach, you can come back and show me your initialization sequence.
I would then add a "makeOpencvReadyForUse" feature.

Can you help with this problem?

Provide an answer of your own, or ask ONG YI CHONG for more information if necessary.

To post a message you must log in.