Building Large-scale Testing framework Using Sikuli --- to be revised
- Keywords:
- unit test large-scale
- Last updated by:
- RaiMan
*** This FAQ is outdated and should be revised
-------
(as far as the native libraries are concerned (steps 4 - 7).
Things to think about before building your system, you might think the following requirements and think of corresponding solution:
1) You want your testing system to be able to run automatically -- either by a website trigger, or a simple cron job that runs at certain times.
Proposed Solution: This means you want to Sikuli in command line mode, which is easier to run from script and capture all the logs
2) You want your testsuites to be built once and never have to change again, or have very low maintenance. This is because your test suties will keep growing, and if you have to update your testsuites all the time, it is hard to maintain.
Proposed Solution: Writing your test cases inside the testsuite by calling common library APIs, instead of using Sikuli directly.
3) When building your common library API, you want to call sikuli methods, however you do not want to update your images that often.
Proposed Solution: Build an image library first, then start writing your common library API.
4) More logging might be needed, especially when a test case fail and you need to know the exact screen flow sequence.
Proposed Solution: Build a screen recorder library that runs before each testsuites, and save after a testsuite is done. Along with your unit testing log, this screen log will help you figure out why a test failed.
So with the above requirements, here would be a feasible design (sorry I cannot draw graph here):
Top layer:
Folder with all the scripts that runs the testsuites, those scripts can be triggered from your designed website interface, or just cron job.
Middle Layer:
a) An image library that defines all the images that would be used, make sure you separate them in product lines and software versions etc, so that you can in the future support or switch among them
b) A bunch of libraries that includes the following:
* A set of common libraries that implements all the routines that can be repeatability called by your testsuites
* A recorder library that can triggers a screen record, this can be triggered at the beginning of each of your testsuites and save the log at the end of the testsuites.
* A logging library if needed to record all test results
* A notice library for you to notify developers (with links to test results, screen log etc. ) testresults after each test is done.
c) Test suites that calls common library routine
Bottom Layer:
Sikuli libraries, I'd like to keep different version of sikuli in different folders so that it is easier to upgrade and roll back if needed.
One simple example to implement this (without the recording, logging and notifier part), here I am using Mac OS, I can add windows after I figured out all the details for it.
Step 1) Download Sikuli 0.10 and put the .app file into /Application folder
Step 2) Setup your test folder, which from here on I will refer it to TEST_DIR
Step 3) Copy /Applications/
then run this following script under TEST_DIR/Frameworks
---
#!/bin/sh
for lib in *.*lib
do
for ref in `otool -L $lib | grep executable_path | awk '{print $1'}`
do
install_
done
done
---
After this, use otool -L *.*lib to make sure all @executable_path are replaced by @loader_path for all the libraries
Step 4) make a tmplib directory under TEST_DIR, then do the following
a) Extract this jar file from somewhere /Applications/
b) mv the libScreenMatchP
c) do "otool -L TEST_DIR/
If you have any problem doing the above step 3 and 4 (or if you get unsatisfied link error later), you can also download those two folders here (thanks to Tsung-Hsiang Chang):
http://
Step 5) Extract the Lib directory from /Applications/
Step 6) download this patch http://
cd TEST_DIR/Lib/sikuli
patch < sikuli-mod.patch
you will be prompt with which file to patch, and they should be
TEST_DIR/
TEST_DIR/
This is to make sure the import sikuli can be used by python modules that are not main program
Step 7) Make a folder called test_suites, and underneath which make a file named test.py and put it under folder TEST_DIR/
import sys
import unittest
import recorder
class Test(unittest.
def setUp(self):
def tearDown(self):
def test_recorderl(
suite = unittest.
TestLoader(
unittest.
Step 8) Make a class called recorder.py (make sure you put path to this class to your test.py) and put it under TEST_DIR/myLib
from sikuli.Sikuli import *
class Recorder:
def __init__(self):
pass
def find_and_
a = find("PATH_
if a:
By now your TEST_DIR should contain the following folders
Frameworks
Lib
sikuli-
tmplib
test_suites -- contains all the test suites
myLib -- your libraries like recorder.py
Step 9) Generate a run test script named run_test.sh with content like this
java -d32 -Dpython.path=Lib/ -jar sikuli-script.jar test.sikuli
If you have ton's of them, put them into a folder like Batch_Test
Now you can run your run_test.sh and see all your test pass or fail.
Hope the above helps, the platform is Mac, Sikuli version is 0.10
Reference:
You can find a lot of trouble shooting and solution finding here:
https:/
Look for the part with Libo and Chang's back and forth troubleshooting section.