Allow for hooks before/during/after Sikuli actions

Asked by Will P

Hi,

We'd like to write an extension that, among other things, modifies common Sikuli actions (e.g., click, wait, any Image Search). As a few examples,
  - on type(image, 'txt'), we'd like to insert a delay between the click(image) and the type('txt') actions.
  - on any image search during a script, take a screenshot of the desktop and save it somewhere.
  - insert different logging statements (e.g., "Click foo.png" instead of "Click (100, 200)").

One possibility is to create an extension with the same interface as Sikuli and make people replace all their Sikuli calls with ours, e.g.,:
  Sikuli.click(image)
becomes
  our_extension.click(image).

What I'd like even more though, would be if people using our extension didn't have to modify their test code (much) in order to benefit from our modifications to these functions. Ideally all they'd have to do is something like "our_extension.initialize()", and this would either replace "Sikuli.click()" with our version, or maybe our extension registers handlers like "beforeSikuliClick(<our_extension.some_method>);" so that Sikuli knows to call some_method before it performs a click.

We found out that that's possible (replacing sikuli.Sikuli.click with our extension.click) to do through python, since you can just manually edit the modules you import, but haven't figured it out in Java. But before we try too hard on that route...

The question(s):
  1) Is there any support for doing these kinds of things already?
  2) Any support planned?
  3) If no to 1 and 2, but you think this might be useful, we might try to make these modifications ourselves and push them upstream. In which case, maybe we can chat offline.

Thanks!

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Tsung-Hsiang Chang (vgod) said :
#1

1. See SikuliActionManager.java. It's not completed, but does what you need. (This's not officially announced and supported yet.)
2. yes, but we need to come up with a more elegant way to do this first. I don't want to triple the number of methods for supporting this kind of hooks...
3. Patches are definitely welcome. :) If you are interested in this feature, feel free to take it over. We are not working on this currently.

Revision history for this message
RaiMan (raimund-hocke) said :
#2

I think, that more than one year after the big step to the new API (9.x -> 10.x/X) and the implementation of many new features, it is time now to revise/consolidate the API and overall design.

observations from the standpoint of a power user (based on one year experience dealing with user questions):

-- the Jython and the Java layer are not synchronized
   - some methods only exist on one layer
   - many method signatures are different on the two layers
   - some features are implemented on the Jython level (so not available on the Java level)

possible enhancements looking top-down:

1 -- implementation of features should only take place on the Java level and should isolate all system dependencies (Java implementation level, the engine), the find(text)/OCR feature should be a separate engine jar.
2 -- the current Jython level should be reduced to interface only (no feature implementation) (Jython API, maps to/uses the engine)
3 -- the Jython script run feature should be separated from sikuli-script.jar in its own jar and have a Jython and a Java API (Jython script runner), should be configurable to use external Jython
4 -- there should be an additional Java API layer, that has the "same" classes/methods as the Jython layer and no implementation of features (Java API, maps to/uses the engine)
5 -- every support for any script running should be concentrated in sikuli-ide.jar (which uses the "3 -- Jython script runner"), it should have a feature to generate jar packages (all target systems) that can be run from commandline only needing Java being available.
6 -- the image capture/store feature and the preview feature should be available as a separate jar, so it can be used standalone or implemented into other IDE's/editors

with this concept, we would have a true engine and a true API, so both could be exchanged/enhanced separately from each other (e.g. the vnc challenge or Selenium integration). And we have some tools (IDE, script runner, capture support, preview), that could be used or not.

The distribution package should be one jar, that contains everything and a self running installer, that does everything needed on the respective system (like the Jython distribution).

there should be separate packages available (or configurable from full download)
- basic Jython (1, 2, 3, 6)
- basic Java (1, 4, 6)
- basic complete (1, 2, 3, 4, 6)
- IDE basic (1, 2, 3, 5)
- complete (1 - 6)

For each of these different jars (1 - 6) there should be 3 levels of developement (stable, bug fix/feature add, experimental) and separate build workflows.

The distribution packages should have a configurable build workflow to integrate private versions of the jars 2, 4 and 6 together with needed other components.

Based on all this it would easily be possible to implement JRuby scripting (exchange jar 2 and 3), a Scala API (exchange jar 4) or any other API for environments, that can talk directly to Java. Even true integration into other not-directly-Java-aware environments (Python, Perl, Ruby, C, ...) would be possible rather easy (the API level has to be implemented for the respective language and "connected" to the engine's Java API).

One could implement an optional API version (even configurable), that fully supports before/after-callbacks (generic and specific) for the relevant features.

And it is easily possible to implement support for changing/complementing the API (statically like today with extensions or even dynamically on the fly during run).

Based on jar 6, there would be more motivation, to implement Sikuli support into other IDE's or editors.

Revision history for this message
Tsung-Hsiang Chang (vgod) said :
#3

@Raiman, I totally agree with you.
In fact, I'm doing a bit of refactoring toward this direction. The first step would be minimize the code in the Python layer and make the Java API more consistent with the current Python API. I'm also thinking of rewriting some code with Scala, but it could be a challenge to make the API compatible with pure Java at the same time.
Let's start a blueprint and we can discuss and refine the details there.

Can you help with this problem?

Provide an answer of your own, or ask Will P for more information if necessary.

To post a message you must log in.