Groovy API

*** your main issue:

--- I am having some really weird image location issues

What kind of host systems?
What kind of VM's?
What are the symptoms?

If you are talking about VMware, VirtualBox or similar solutions: there are some weird things happening in these environments. Some Sikuli features are not running in different setups. But there is no consistent image about that.

--- Is the lp:sikuli branch the best place
It is the only place to officially get the source. You need bazaar on your machine. Currently the source is already at X-1.0rc3.
There is an unofficial github mirror: https://github.com/arnaudgelas/sikuli (about 24 hours late)
They have implemented hudson, so you might download the current build: http://sikuli.org/hudson/

*** the rest of your statement:
very good summary of Sikuli's "current limitations" regarding GUI awareness.

--- Groovy
looks interesting. They are thinking about converting to Scala (https://blueprints.launchpad.net/sikuli/+spec/sikuli-moves-to-scala)

--- alternate API
the real challenge in the moment: there is no consistent API ;-) (but I think, they are working on that). The Python layer has features, that are not available on the Java level. Many function signatures are different in the 2 layers. The Python layer knows a default screen, which is not available on the Java level. And some other minor aspects ...
So principally you could implement an alternate API with any scripting language, that is Java aware (like Jython currently used, even Selenium might be possible).
My favorite in the moment is JRuby.
Another challenge: Currently there is no strict separation between the scripting level (now Python), the engine level (now Java) and the native system dependent layer (now C++).

--- I think a lot of these could be overcome using a VM or running the script remotely
One of the fascinating aspects of Sikuli is: it runs in front of your eyes and does what you would do, while you are drinking coffee. You are right from the standpoint of a system tester. But my experience with Sikuli Q&A tell me, that people, who are seriously testing software already have a testing framework and those who are based on Java or Python environments, have their dedicated machines, where the tests are run.

--- Cannot use the system whilst the scripts are running
Generally: When a script is running on a machine, "only" the GUI is blocked to some extent. But even this can be overcome by some effort in robust scripting. What still is left: If some other robot steels mouse and keyboard during a Sikuli script's run, you might be lost.

Host = Windows 7 64 bit
VM = VMWare Windows 7 32 bit
Symptom = Sikuli cannot find the image file. I have tried absolute paths, changing the SIKULI_IMAGE_PATH, explicitly settings location via Settings.BundlePath - none of these seem to work. Note I am using the Java API and not the Sikuli IDE.

What other weird behaviour have you seen when using VMs?

Do you think there's scope to enhance the image recognition engine to cope with things like windowing style changes? I'm guessing there's going to be some kind of text involved in the image capture to make it unique enough to match on.

Thanks

Ben

On 27/06/2011, at 5:55 PM, RaiMan wrote:

> Your question #162881 on Sikuli changed:
> https://answers.launchpad.net/sikuli/+question/162881
>
> Status: Open => Needs information
>
> RaiMan requested more information:
> *** your main issue:
>
> --- I am having some really weird image location issues
>
> What kind of host systems?
> What kind of VM's?
> What are the symptoms?
>
> If you are talking about VMware, VirtualBox or similar solutions: there
> are some weird things happening in these environments. Some Sikuli
> features are not running in different setups. But there is no consistent
> image about that.
>
> --- Is the lp:sikuli branch the best place
> It is the only place to officially get the source. You need bazaar on your machine. Currently the source is already at X-1.0rc3.
> There is an unofficial github mirror: https://github.com/arnaudgelas/sikuli (about 24 hours late)
> They have implemented hudson, so you might download the current build: http://sikuli.org/hudson/
>
>
> *** the rest of your statement:
> very good summary of Sikuli's "current limitations" regarding GUI awareness.
>
> --- Groovy
> looks interesting. They are thinking about converting to Scala (https://blueprints.launchpad.net/sikuli/+spec/sikuli-moves-to-scala)
>
> --- alternate API
> the real challenge in the moment: there is no consistent API ;-) (but I think, they are working on that). The Python layer has features, that are not available on the Java level. Many function signatures are different in the 2 layers. The Python layer knows a default screen, which is not available on the Java level. And some other minor aspects ...
> So principally you could implement an alternate API with any scripting language, that is Java aware (like Jython currently used, even Selenium might be possible).
> My favorite in the moment is JRuby.
> Another challenge: Currently there is no strict separation between the scripting level (now Python), the engine level (now Java) and the native system dependent layer (now C++).
>
> --- I think a lot of these could be overcome using a VM or running the script remotely
> One of the fascinating aspects of Sikuli is: it runs in front of your eyes and does what you would do, while you are drinking coffee. You are right from the standpoint of a system tester. But my experience with Sikuli Q&A tell me, that people, who are seriously testing software already have a testing framework and those who are based on Java or Python environments, have their dedicated machines, where the tests are run.
>
> --- Cannot use the system whilst the scripts are running
> Generally: When a script is running on a machine, "only" the GUI is blocked to some extent. But even this can be overcome by some effort in robust scripting. What still is left: If some other robot steels mouse and keyboard during a Sikuli script's run, you might be lost.
>
> --
> To answer this request for more information, you can either reply to
> this email or enter your reply at the following page:
> https://answers.launchpad.net/sikuli/+question/162881
>
> You received this question notification because you asked the question.

Ben Sullivan
f: +61 (0)7 33190991
w: bensullivan.net
LinkedIn

--- Symptom = Sikuli cannot find the image file
Sure, that the image file could not be found? Currently FindFailed exception does not distinguish between "cannot find the image file in the image search path" and "could not match the given image on the screen". So it might be a problem, that your image resolution does not match the VM's screen resolution.

--- What other weird behaviour have you seen when using VMs
There are some bug reports about strange behavior of Sikuli running in VM's.
I myself have some experience with Parallels (running Win7-32) and VirtualBox (running Ubuntu 10.10 64Bit) on Mac OS X 10.6. The main problems are with the overlay windows Sikuli is using in different situations (capture(), selectRegion(), highlight(), ...). And the timing is different, so you have to have really robust scripts.

--- enhance the image recognition engine to cope with things like windowing style changes
I know, that this feature is on the agenda. It might be an enhancement to the Pattern class, which would allow to define generic patterns like "application window" and it's elements, so you could say in your script:

this is not a specification;-)

aw = App("Firefox").window(x)
aw.windowType = "BROWSERWINDOW"
# aw at this point knows: it is a browser window of Firefox
aw.type(Pattern(aw.InputFieldURL()), "sikuli.org\n")
aw.click(Pattern(aw.CloseButton()))

and this works on Win/Mac/Linux.

Pattern(aw.InputFieldURL())
- gets application and window type from aw
- with this and InputFieldURL() gets a generic image id from a central repository (might be in the cloud), that is turned into a concrete image loaded from this central repository reflecting screen resolution, system environment and app specifics.

Thanks for your prompt replies.

As you mention, I had assumed that the error message was complaining about
the image on the file system rather than the image match on the screen.
Is there an issue open to address this as it's pretty misleading?

I have also found that I need to put waits in for the controls I want to
interact with. What exactly is Sikuli waiting for? As I watch the script
run I can see the controls on the web page and the page seems to have
loaded completely. Is the wait more about waiting for the image
recognition engine to find a match on the png I am trying to find? Are
there any best practices around helping the engine find matches quickly?

Also, can I grab a RC3 jar file for the sikuli-script component? I am on
a RC2 jar but my source from Bazaar is RC3 so when I attach it to debug I
am out of sync.

THanks

Ben

> Your question #162881 on Sikuli changed:
> https://answers.launchpad.net/sikuli/+question/162881
>
> Status: Open => Answered
>
> RaiMan proposed the following answer:
> --- Symptom = Sikuli cannot find the image file
> Sure, that the image file could not be found? Currently FindFailed
> exception does not distinguish between "cannot find the image file in the
> image search path" and "could not match the given image on the screen". So
> it might be a problem, that your image resolution does not match the VM's
> screen resolution.
>
> --- What other weird behaviour have you seen when using VMs
> There are some bug reports about strange behavior of Sikuli running in
> VM's.
> I myself have some experience with Parallels (running Win7-32) and
> VirtualBox (running Ubuntu 10.10 64Bit) on Mac OS X 10.6. The main
> problems are with the overlay windows Sikuli is using in different
> situations (capture(), selectRegion(), highlight(), ...). And the timing
> is different, so you have to have really robust scripts.
>
> --- enhance the image recognition engine to cope with things like
> windowing style changes
> I know, that this feature is on the agenda. It might be an enhancement to
> the Pattern class, which would allow to define generic patterns like
> "application window" and it's elements, so you could say in your script:
>
> this is not a specification;-)
>
> aw = App("Firefox").window(x)
> aw.windowType = "BROWSERWINDOW"
> # aw at this point knows: it is a browser window of Firefox
> aw.type(Pattern(aw.InputFieldURL()), "sikuli.org\n")
> aw.click(Pattern(aw.CloseButton()))
>
> and this works on Win/Mac/Linux.
>
> Pattern(aw.InputFieldURL())
> - gets application and window type from aw
> - with this and InputFieldURL() gets a generic image id from a central
> repository (might be in the cloud), that is turned into a concrete image
> loaded from this central repository reflecting screen resolution, system
> environment and app specifics.
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/sikuli/+question/162881/+confirm?answer_id=2
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/sikuli/+question/162881
>
> You received this question notification because you asked the question.
>

--- Is there an issue open to address this ...
Will be fixed in rc3

--- I need to put waits ...
It is simply to pause the script for a while, so that the next internally taken screenshot contains the visual object to search for. Since these are times below 0.5 seconds, we are talking about, you cannot realize it looking at the screen.

--- best practices around helping the engine find matches quickly
The only thing that really helps, is to restrict the region to the area, that contains the element you are looking for.
It is risky and time consuming, to search an element of 30x30 pixel (e.g. the buttons in an applications menu bar) on the whole screen, when you know, where it might be.

example:
- Firefox 5 HomeButton right side of menu bar
- MacBook Air (1280x800) Core2Duo 1,86
- whole screen: 0.4 seconds
- restricted to menu bar: 0.08 seconds

--- rc3 sikuli-script.jar
From Launchpad you only get the sources. If you want a sikuli-script.jar, you have to either build it from these sources or use the nightly build from http://sikuli.org/hudson/. Be aware, that it is not only the jar. The native modules written in C++ (Windows/Linux: libs directory, Mac: Contents/Frameworks directory) are also subject to change. So it is not recommended to mix elements of rc2 and rc3.