SikuliX

Building a custom Sikuli/Eclipse IDE

Asked by joe maniaci

So for the last year I have gone down the path of researching various automated GUI test tools. One of the more promising ones was Testplant's Eggplant Functional tool. However, the price is just way too much for my company at the moment. So I started going down the path of Sikuli and I have come up with a satisfactory workflow for myself, but I just don't think it will be satisfactory for those I'll have to school up. So I work with giant printers and we don't really have spare engineers to build up tests but we do have a lot of testers that could. I even created my own drastically simplified Sikuli API(in Java) and all a user has to do is collect images with the sikuli IDE and piece it all together within Eclipse using Java. However for someone without a programming background I fear this might not be enough. So here is what I am thinking...

An all in one custom Eclipse IDE utilizing a custom DSL with the ability to capture imagery as well as tweak the target and similarity. This and the ability to model the interface, that is, laying out that if I perform Action A on View V, I should now expect to see View V2, where I could perform Action A2, so on and so forth. So imagine dragging and dropping a couple Thumbnail/ImageButton/Icon-like controls that represent the views, the actions that can be performed and how they all link together in a drawing area. Kind of like UML Designer in Eclipse.

I'm trying to write out requirements for all this but I'd like to prototype some more first and that's where I'm running into issues of what to use. For creating a scripting language that builds out Java it appears that Xtext will be the way to go so that's that. I'm thinking to create an Eclipse plugin to capture and tweak imagery, that way other Eclipse users could use just that for their projects(would be nice to know how to easily hook into Sikuli for this). In regards to modeling/IDE development I have come across the Eclipse Modeling Framework and the Graphical Editing Framework but I'm not entirely certain how to get started.

If anything I'd be curious if people could point me in the right direction, let me know if something is possible or/not, what Eclipse tools might be more suitably.. I think what's really throwing me off is all the various technologies there appears to be; Eclipse Modeling Project, Graphical Editing Framework, Plugin Development Environment, Standard Widget Toolkit, Rich Client Platform, Eclipse SDK, Eclipse Project, e4, etc, etc.

Thank you very much for any useful info.

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said : 		#1

Though I think, that to use Python language is not too complicated and challenging to write down basic to medium complex SikuliX based workflows, there is always room, for making it even easier for people, that should not be "tortured with programming".

With SikuliX workflows we have 3 challenges, to support these people:

1. a frontend, that easily allows to add the SikuliX features to a workflow, based on clickable templates, that "know" about the already existing workflow context and support the intuitive completion of specs and parameters. This should be available for automation workflows and testcases. Running such a workflow means to translate it to a runnable script and run it.

2. a tool, that allows to capture, optimize and manage the images needed in a workflow.

3. a recorder, that allows to setup a basic workflow while working with some application.

I am on the way to address all 3 aspects with SikuliX2, but have to admit, that I am some months away from a useable package.

my ideas about the 3 challenges (for all 3 there are already basic approaches available in SikuliX2):

-- at 1: the frontend should be table-based (like the workflows with RobotFramework).
Each line is an action like click, wait, type/write/paste.
An action can be selected from a context-menu or entered by typing in the first column (auto-complete) and a suitable template is filled into the line.
Parameters of an action are edited by just clicking on the respective column in the line (example: you can capture and/or manage a needed image supported by the tool of challenge 2)
With this basic sequential workflows can be created and run easily.
Medium complex workflows additionally need repeat (loop, for, ..) and decision (if/elif/else) features, some kind of supporting functions and import and some features to interact with the user, while the workflow is executed.
In SikuliX2 I already have a basic approach based on a tablebased JPanel, that translates into a runnable JavaScript script (JavaScript with Java has the advantage, that the interpreter is JVM-builtin (Nashorn))
After having implemented such a frontend, it might be easily complemented with a completely visual tool, that shows the workflow as a diagram with graphical elements, that can be edited by just clicking and dragging around (you mentioned UML).
It might be a good idea, to make this frontend browser-based.
And of course it could be implemented in Eclipse/Netbeans/IntelliJ IDEA using their plugin frameworks or even in JEdit.

-- at 2: the tool must allow to capture images and optimize them for the use of SikuliX features (concentrate on the key visual aspects, avoiding background towards the edges, allow transparency and images in parts and last but not least manage image sets).
The challenge here is a suitable GUI. This tool can then be used as a library in aspects 1 and 3.

-- at 3: this is the most challenging, since the normal approaches with Java are not useable (to catch mouse and keyboard actions you always need some overlay frames, which are not possible in all cases (e.g. full screen) and all over the place).
The most promising approach is using the JNativeHook library, that allows to run a recorder in the background that tracks all mouse and keyboard actions and decides wether it is a command for the recorder or going through to the running application. In the end you have some raw workflow, that can be used as is or optimised/augmented with the tool 1. The recorder also tracks screen content using intermediate screenshots in various aspects depending on the state of recorder and/or application. Images might be edited while recording or afterwards based on the screenshots using tool 2.

Conclusion: I think your ideas go into the right direction and I am very interested, to further discuss the requirements with you.

Maybe we can agree on some toolbox, that might be used in your Eclipse based approach and in SikuliX2 as well.

Can you help with this problem?

Provide an answer of your own, or ask joe maniaci for more information if necessary.

To post a message you must log in.