Automatic script generation?

No feature in the current release and as far as I know even not in the upcoming version of Sikuli X.

What is possible in the moment:
- using getMouseLocation() you may record the mouse movement done by the user
- you may capture() the screen/a region on a regular base may be with regard to the mousemovement
- you may use some subprocessing/threading, to implement parallel analyzing and some basic interaction with the user based on the selectRegion() feature
- Sikuli X: there will be a new App class and OCR, that might give some more options

What is missing:
- you cannot record mouse actions like clicks
- you cannot record keyboard actions

It might be possible to combine these things with one of the available recorders out there.

And there might be a bunch of more options, if you are willing to use Sikuli on the Java level or even add your own features based on the source.

If you are really interested, it would be a good idea, to post a feature request bug with some detailed
 information about the needed features.

I am just looking for an idea of my school project.
Since this feature will not be available in the near future, it would be a viable option for me.

Thank you for your reply.

@Lono: We look forward to seeing it as your project (and hope you will open source it). :)

@vgod: Sure, I will open source it.

status on this project? :o

No progress so far. The project proposal was rejected by the professor.
I will try to propose it again for the next semester.

Hi there,
I find Sikuli very interesting, and am interested in working to build a recorder for it.
Particularly recording keyboard actions... to make it easy/helpful to write the script.
Also optionally to have mouse action capture.

Can you give me any leads, to get started.

Like what might be the available recorders out there... as hinted in
> It might be possible to combine these things with one of the available recorders out there.

I see pyhook there - but not probably that cannot be used in jython !

--- recorder
Sikuli is to some extent system independent (Windows, Mac, Linux), but recorders are system specific (I do not know a recorder, that works on all 3 systems). pyhook is not only C-based and so not useable with Jython it is even only available for Windows.

--- Java based
In my opinion, every solution has to be built on the Java level, to be globally useable with Sikuli. So a first research might be to find out, what Java supporting features are available for this.

--- starting point
What Tom Yeh is doing together with the extension Guide and some first researches to create a visual editor (you have to look into the sources) are very interesting.

--- possible solution: coexistence
You might take any tool, that allows to track GUI actions and produce some output. Then take this output and create Sikuli code from that. This might be implemented with sub processing to make it interactive to some extent.

Would be interesting to here more of your ideas.

Thanks for pointers.

I was thinking of making a separate application, which will track gui actions (mouse+keyboard initially), and will generate sikuli script on the fly.
Once recording is done, then generated script could be opened/pasted in sikuli ide.

Could you elaborate what you intend by sub-processing, to make it interactive?

As long as it is "only" mouse and keyboard replay, your app must only record clicks (together with the used button) and keyboard input. No images needed.

One challenge will be to assure that the GUI situation is the same as at time of recording(e.g. window size and position) or you did the recording relative to some screen objects, that must be restored on replay.

In case you want to use images additionally to use searches (which would make things somewhat independent from restoring the exact window situation), then you might need the support of the user to capture the image in a way, that makes sense. I do not think, that this is possible to automate without heavy image analyzing.

So you will need some hot-keys as tool for the user, to tell your app, what might be the next action (e.g. capture needed before click).

for some of these activities, it might make sense to delegate it to threads, to allow parallel processing. E.g. tracking mouse movement must be threaded, to make it effective and the capturing and listening to hotkeys and ....

But this might be going far beyond what you are thinking about in the moment.

Thanks this is very useful.

I will work on this as I get time, and report the progress here.

Best wishes

Vikramsjn - I'm interested in similar functionality and curious if you made any progress on the subject?

Thank You.

@RaiMain Every thread, someone proposes to make a recorder... So I leveraged AutoHotKeys recorder to generate AHK scripts, which are then converted to SikuliX. This is my "lazy mans" solution. Just instantiate the converter, and run the `convert` method on your AHK script and it will print out the SikuliX script in the SikuliX debugger.

https://gist.github.com/Wykleph/aad5b6b9b1eccd962caf3ca39399d525

I'll keep it updated to the best of my ability, and I'll add a small example to the comments. So far it only clicks and does keyboard events. It doesn't handle any logic or anything like that. Just a very simple AHK to SikuliX script.

Also, I don't have all the AHK keyboard bindings mapped to the SikuliX bindings yet, but you can add them on the fly.

Cheers!

@Wykleph
Thanks for your comments and your solution.

... but why don't you simply run the AHK script from within a Sikuli script?

Since AHK (as far as I know) runs scripts with nearly no startup delay, this would be a suitable solution.

I will add AHK to the inline support for running scripts as well
(http://sikulix-2014.readthedocs.org/en/latest/scripting.html#running-scripts-and-snippets-from-within-other-scripts-and-run-scripts-one-after-the-other)

@RaiMan, well I was converting some old AHK scripts I had to reduce dependencies, and the applications I'm automating haven't changed in years so I decided to write it so I wouldn't have to write it all out manually. It paid off due to the amount of AHK script I had to convert. I may just have sikuli run some of them for me though, as it's a pretty big job to convert the more complicated ones, even with that snippet I wrote.

@Wykleph
thanks, makes sense.