Building Large-scale Testing framework Using Sikuli 1.0 - Mac and Windows

Asked by Libo Cao

Things to think about before building your system, you might think the following requirements and think of corresponding solution:

1) You want your testing system to be able to run automatically -- either by a website trigger, or a simple cron job that runs at certain times.
Proposed Solution: This means you want to Sikuli in command line mode, which is easier to run from script and capture all the logs

2) You want your testsuites to be built once and never have to change again, or have very low maintenance. This is because your test suties will keep growing, and if you have to update your testsuites all the time, it is hard to maintain.
Proposed Solution: Writing your test cases inside the testsuite by calling common library APIs, instead of using Sikuli directly.

3) When building your common library API, you want to call sikuli methods, however you do not want to update your images that often.
Proposed Solution: Build an image library first, then start writing your common library API.

4) More logging might be needed, especially when a test case fail and you need to know the exact screen flow sequence.
Proposed Solution: Build a screen recorder library that runs before each testsuites, and save after a testsuite is done. Along with your unit testing log, this screen log will help you figure out why a test failed.

So with the above requirements, here would be a feasible design (sorry I cannot draw graph here):

Top layer:

Folder with all the scripts that runs the testsuites, those scripts can be triggered from your designed website interface, or just cron job.

Middle Layer:
a) An image library that defines all the images that would be used, make sure you separate them in product lines and software versions etc, so that you can in the future support or switch among them
b) A bunch of libraries that includes the following:
    * A set of common libraries that implements all the routines that can be repeatability called by your testsuites
    * A recorder library that can triggers a screen record, this can be triggered at the beginning of each of your testsuites and save the log at the end of the testsuites.
   * A logging library if needed to record all test results
   * A notice library for you to notify developers (with links to test results, screen log etc. ) testresults after each test is done.
c) Test suites that calls common library routine

Bottom Layer:
Sikuli libraries, I'd like to keep different version of sikuli in different folders so that it is easier to upgrade and roll back if needed.

One simple example to implement this (without the recording, logging and notifier part), here I am using Mac OS, if you want to use windows, just refer to the method in this post (relatively much easier on windows)

Step 1) Download Sikuli 0.10 and put the .app file into /Application folder

Step 2) Setup your test folder, which from here on I will refer it to TEST_DIR

Step 3) Copy /Applications/ folder to TEST_DIR
then run this following script under TEST_DIR/Frameworks

for lib in *.*lib
   for ref in `otool -L $lib | grep executable_path | awk '{print $1'}`
      install_name_tool -change $ref @loader_path/../Frameworks/`basename $ref` $lib

After this, use otool -L *.*lib to make sure all @executable_path are replaced by @loader_path for all the libraries
Step 4) make a tmplib directory under TEST_DIR, then do the following

a) Extract this jar file from somewhere /Applications/
b) mv the libScreenMatchProxy.jnilib to TEST_DIR tmpdir from META-INF/lib that obtained from the above extraction
c) do "otool -L TEST_DIR/tmplib/libScreenMatchProxy.jnilib" and make sure all the path that were pointing to /opt/local/lib/*.dylib are changed into @loader_path/../Frameworks/*.dylib

If you have any problem doing the above step 3 and 4 (or if you get unsatisfied link error later), you can also download those two folders here (thanks to Tsung-Hsiang Chang):

Step 5) Extract the Lib directory from /Applications/ and put this folder under TEST_DIR, then put sikuli-script.jar also under TEST_DIR

Step 6) download this patch , then apply it by doing:
cd TEST_DIR/Lib/sikuli
patch < sikuli-mod.patch

you will be prompt with which file to patch, and they should be

This is to make sure the import sikuli can be used by python modules that are not main program

Step 7) Make a folder called test_suites, and underneath which make a file named and put it under folder TEST_DIR/test_suites/test.sikuli/ :

import sys
import unittest
import recorder

class Test(unittest.TestCase):
    def setUp(self):

    def tearDown(self):

    def test_recorderl(self):
        recorder_obj = recorder.Recorder()
suite = unittest.


Step 8) Make a class called (make sure you put path to this class to your and put it under TEST_DIR/myLib

from sikuli.Sikuli import *

class Recorder:
    def __init__(self):

    def find_and_click(self):
        if a:

By now your TEST_DIR should contain the following folders


     test_suites -- contains all the test suites

     myLib -- your libraries like
Step 9) Generate a run test script named with content like this
java -d32 -Dpython.path=Lib/ -jar sikuli-script.jar test.sikuli

If you have ton's of them, put them into a folder like Batch_Test

Now you can run your and see all your test pass or fail.

Hope the above helps, the platform is Mac, Sikuli version is 0.10

You can find a lot of trouble shooting and solution finding here:
Look for the part with Libo and Chang's back and forth troubleshooting section.

Question information

English Edit question
SikuliX Edit question
No assignee Edit question
Solved by:
Libo Cao
Last query:
Last reply:
Revision history for this message
Libo Cao (libocao) said :

Just reply to make this problem solved

Revision history for this message
Tsung-Hsiang Chang (vgod) said :

Tsung-Hsiang Chang converts this article to an FAQ. (also replaces Sikuli 1.0 with Siuli 0.10.)

FAQ #1110: “Building Large-scale Testing framework Using Sikuli 0.10 - Mac example”.

Revision history for this message
Libo Cao (libocao) said :

I have people who work in well-know public company contacting me because of this post. They would also like to start evaluating developing automated tools/system using Sikuli.

My presentation on the usage of this system to both client team and server team are well-received. The system will only grow bigger.

Compare to current available tools on the market, for example "device anywhere", it received much better feedback on all aspects.

Now I am trying to build a similar system on windows, the problem is that what worked for old 0.9 version sikuli command line mode won't work for 0.10.1 -- for example I cannot find the win32 folder that was needed to run command line mode.

Revision history for this message
Libo Cao (libocao) said :

Just tested, if you want to use the same method to build on windows XP, you can use the same method here

It still works great using sikuli 0.10.1

Revision history for this message
Tsung-Hsiang Chang (vgod) said :

We provide command line options for Sikuli IDE since 0.10.1. The usage is documented at the official download page.
This should be able to run the test cases mentioned above.

Command line usage:

usage: Sikuli-IDE [--args ] [-h] [-r ]
    --args specify the arguments passed to Jython's sys.argv
 -h,--help print this help message
 -r,--run run .sikuli or .skl file

Example - Run a xxxx.sikuli with three arguments:
For Windows users, type "PATH-TO-SIKULI/sikuli-ide.exe -r xxxx.sikuli --args a1 a2 a3" in the command prompt(cmd.exe).
For Linux users, type "PATH-TO-SIKULI/ -r xxxx.sikuli --args a1 a2 a3".
For Mac users, type "/Applications/ -r xxxx.sikuli --args a1 a2 a3".

Revision history for this message
Gregor Kopka (mailfrom-launchpad-net) said :

How to capture the output of the script when running it via -r ?

Is it possible to direct output to stdout in case -r flag is present?

Because at least for my usecase i would need access to what is dumped into messages tab in IDE (would also help when testing, eg. to be able to see where it crashed).

Revision history for this message
Tsung-Hsiang Chang (vgod) said :

You can redirect the output to stdout by appending -Dsikuli.console=false to java.
On Windows or Linux, just modify the .bat/.sh to do this.
On Mac, it's more complicated that you need to add a key/value pair "sikuli.console" and "false" under the "Properties" in

But of course, the redirection should be done automatically when -r is present. I will fix this.

Revision history for this message
vishal bhat (crab19) said :

Any pointer on how to buid test suites using java