Building Large-scale Testing framework Using Sikuli 1.0 - Mac and Windows

Things to think about before building your system, you might think the following requirements and think of corresponding solution:

1) You want your testing system to be able to run automatically -- either by a website trigger, or a simple cron job that runs at certain times.
Proposed Solution: This means you want to Sikuli in command line mode, which is easier to run from script and capture all the logs

2) You want your testsuites to be built once and never have to change again, or have very low maintenance. This is because your test suties will keep growing, and if you have to update your testsuites all the time, it is hard to maintain.
Proposed Solution: Writing your test cases inside the testsuite by calling common library APIs, instead of using Sikuli directly.

3) When building your common library API, you want to call sikuli methods, however you do not want to update your images that often.
Proposed Solution: Build an image library first, then start writing your common library API.

4) More logging might be needed, especially when a test case fail and you need to know the exact screen flow sequence.
Proposed Solution: Build a screen recorder library that runs before each testsuites, and save after a testsuite is done. Along with your unit testing log, this screen log will help you figure out why a test failed.

So with the above requirements, here would be a feasible design (sorry I cannot draw graph here):

Top layer:

Folder with all the scripts that runs the testsuites, those scripts can be triggered from your designed website interface, or just cron job.

Middle Layer:
a) An image library that defines all the images that would be used, make sure you separate them in product lines and software versions etc, so that you can in the future support or switch among them
b) A bunch of libraries that includes the following:
    * A set of common libraries that implements all the routines that can be repeatability called by your testsuites
    * A recorder library that can triggers a screen record, this can be triggered at the beginning of each of your testsuites and save the log at the end of the testsuites.
   * A logging library if needed to record all test results
   * A notice library for you to notify developers (with links to test results, screen log etc. ) testresults after each test is done.
c) Test suites that calls common library routine

Bottom Layer:
Sikuli libraries, I'd like to keep different version of sikuli in different folders so that it is easier to upgrade and roll back if needed.

One simple example to implement this (without the recording, logging and notifier part), here I am using Mac OS, if you want to use windows, just refer to the method in this post https://answers.launchpad.net/sikuli/+question/100436 (relatively much easier on windows)

Step 1) Download Sikuli 0.10 and put the .app file into /Application folder

Step 2) Setup your test folder, which from here on I will refer it to TEST_DIR

Step 3) Copy /Applications/Sikuli-IDE.app/Contents/Frameworks folder to TEST_DIR
then run this following script under TEST_DIR/Frameworks
---

#!/bin/sh
for lib in *.*lib
do
   for ref in `otool -L $lib | grep executable_path | awk '{print $1'}`
   do
      install_name_tool -change $ref @loader_path/../Frameworks/`basename $ref` $lib
   done
done

---
After this, use otool -L *.*lib to make sure all @executable_path are replaced by @loader_path for all the libraries
Step 4) make a tmplib directory under TEST_DIR, then do the following

a) Extract this jar file from somewhere /Applications/Sikuli-IDE.app/Contents/Resources/Jave/sikuli-script.jar
b) mv the libScreenMatchProxy.jnilib to TEST_DIR tmpdir from META-INF/lib that obtained from the above extraction
c) do "otool -L TEST_DIR/tmplib/libScreenMatchProxy.jnilib" and make sure all the path that were pointing to /opt/local/lib/*.dylib are changed into @loader_path/../Frameworks/*.dylib

If you have any problem doing the above step 3 and 4 (or if you get unsatisfied link error later), you can also download those two folders here (thanks to Tsung-Hsiang Chang):
http://people.csail.mit.edu/vgod/sikuli/sikuli-libs.zip

Step 5) Extract the Lib directory from /Applications/Sikuli-IDE.app/Contents/Resources/Jave/sikuli-script.jar and put this folder under TEST_DIR, then put sikuli-script.jar also under TEST_DIR

Step 6) download this patch http://people.csail.mit.edu/vgod/sikuli/sikuli-mod.patch , then apply it by doing:
cd TEST_DIR/Lib/sikuli
patch < sikuli-mod.patch

you will be prompt with which file to patch, and they should be
TEST_DIR/Lib/sikuli/Screen.py
TEST_DIR/Lib/sikuli/Sikuli.py

This is to make sure the import sikuli can be used by python modules that are not main program

Step 7) Make a folder called test_suites, and underneath which make a file named test.py and put it under folder TEST_DIR/test_suites/test.sikuli/ :

import sys
import unittest
import recorder

class Test(unittest.TestCase):
    def setUp(self):
        setThrowException(False)

    def tearDown(self):
        setThrowException(True)

    def test_recorderl(self):
        recorder_obj = recorder.Recorder()
        recorder_obj.find_and_click()
suite = unittest.

TestLoader().loadTestsFromTestCase(Test)
unittest.TextTestRunner(verbosity=2).run(suite)

Step 8) Make a class called recorder.py (make sure you put path to this class to your test.py) and put it under TEST_DIR/myLib

from sikuli.Sikuli import *

class Recorder:
    def __init__(self):
        pass

    def find_and_click(self):
        a = find("PATH_TO_AN_IMAGE_DEFINED_IN_IMAGE_LIB")
        if a:
            click(a)

By now your TEST_DIR should contain the following folders
     Frameworks
     Lib
     sikuli-script.jar

     tmplib

     test_suites -- contains all the test suites

     myLib -- your libraries like recorder.py
Step 9) Generate a run test script named run_test.sh with content like this
java -d32 -Dpython.path=Lib/ -jar sikuli-script.jar test.sikuli

If you have ton's of them, put them into a folder like Batch_Test

Now you can run your run_test.sh and see all your test pass or fail.

Hope the above helps, the platform is Mac, Sikuli version is 0.10

Reference:
You can find a lot of trouble shooting and solution finding here:
https://answers.launchpad.net/sikuli/+question/108878
Look for the part with Libo and Chang's back and forth troubleshooting section.