SikuliX

Using Sikuli to Assist with Visual Impairment

Asked by J. Mailshredder

I have used Sikuli in the past to automate a number of individual processes so I'm familiar with the IDE and how to make basic scripts.

Due to a traumatic brain injury I now have limited ability to sit in front of a computer screen and I'm interested in knowing if there is an example of using Sikuli as a "computer vision" assistant.

In essence, I'd like to define a number of images (things to look for) and have Sikuli perform an "infinite loop" looking for any of the images of interest. If an image match is found, I'd like Sikuli to launch an external process (a batch file, for example) passing parameters of which image was found and the (x,y) coordinates of where the image was found - and then immediately continue "looking" for the next appearance of one of the images of interest.

In essence, I'd like Sikuli to be my "eyes" and watch the screen for me - and then notify me (or do something, which I can code externally) if one of the items show up.

If anyone has done something similar, I'd appreciate any input you might have.

Thanks!

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said : 		#1

Yes we can: this is possible with SikuliX.

I am willing, to help you to get on the road with that.

Everything can be packed in scripts, which can be run triggered by any visual event, that is detected on the screen.

restrictions for parallel script run:
- only one script at a time can use mouse and keyboard.
- one script should not change the screen content in a way, that other running scripts are affected.

With version 1.1.0 the mouse usage among parallel running snippets can be managed to some extent.
As long as 2 scripts operate in different windows, that are fully visible at any time, parallel processing is possible.

What is your system environment?
Monitor setup? more than one monitor?
Features available, to speak text? How are they used from command line?

What is your scripting experience?
SikuliX directly supports Python and Ruby with the IDE, also scripts written in JavaScript can be run now (need to be edited currently with some external editor)

What are the major applications you want to use?

If you like, you can contact me privately from now on: https://launchpad.net/~raimund-hocke (top left)
Using Skype is possible as well.

Can you help with this problem?

Provide an answer of your own, or ask J. Mailshredder for more information if necessary.

To post a message you must log in.