Does sikuli still need to focus on the window in order to interact with it?

The answer short answer is YES :-)

Sikuli does NOT interact with application at all. Sikuli sends mouse clicks to positions on screen and keyboard input to OS driver and whenever element is focused will receive them.

Does it require focus because of image detection because there is a way to
force it to redraw and capture even when the window is in the background.
It is common on Windows to use the desktop manager to take screen shots,
but this is just a layer on top of GDI+.
https://stackoverflow.com/questions/1163761/capture-screenshot-of-active-window

The source I posted goes one layer deeper.
https://github.com/Microsoft/WinAppDriver/issues/274

Can you refer me to some code where this is done in Sikuli and maybe I can
make it work in the background while you are working in the foreground?

On Sun, Oct 1, 2017 at 2:18 AM, TestMechanic <
<email address hidden>> wrote:

> Your question #658802 on Sikuli changed:
> https://answers.launchpad.net/sikuli/+question/658802
>
> Status: Open => Answered
>
> TestMechanic proposed the following answer:
> The answer short answer is YES :-)
>
> Sikuli does NOT interact with application at all. Sikuli sends mouse
> clicks to positions on screen and keyboard input to OS driver and
> whenever element is focused will receive them.
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/sikuli/+question/658802/+confirm?answer_id=0
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/sikuli/+question/658802
>
> You received this question notification because you asked the question.
>

As already mentioned in comment #1:
SikuliX only sees, what is currently visible on the screen. If something is in background, then you have to take care, that it comes to foreground, before SikuliX can see the window content for example.

To arrange the screen content (windows), the GUI and the apps running is completely your responsibility.

Is there a file I can look at the source where Sikuli "sees" maybe I can
improve it!

On Thu, Oct 5, 2017 at 9:48 AM, RaiMan <<email address hidden>
> wrote:

> Your question #658802 on Sikuli changed:
> https://answers.launchpad.net/sikuli/+question/658802
>
> Status: Open => Answered
>
> RaiMan proposed the following answer:
> As already mentioned in comment #1:
> SikuliX only sees, what is currently visible on the screen. If something
> is in background, then you have to take care, that it comes to foreground,
> before SikuliX can see the window content for example.
>
> To arrange the screen content (windows), the GUI and the apps running is
> completely your responsibility.
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/sikuli/+question/658802/+confirm?answer_id=2
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/sikuli/+question/658802
>
> You received this question notification because you asked the question.
>

This is a new code base for me, but I assume there is a connector from the
OCR to the native Windows/Gnome calls are made?

On Thu, Oct 5, 2017 at 10:04 AM, Brian Cullinan <email address hidden>
wrote:

> Is there a file I can look at the source where Sikuli "sees" maybe I can
> improve it!
>
>
>
> On Thu, Oct 5, 2017 at 9:48 AM, RaiMan <question658802@answers.
> launchpad.net> wrote:
>
>> Your question #658802 on Sikuli changed:
>> https://answers.launchpad.net/sikuli/+question/658802
>>
>> Status: Open => Answered
>>
>> RaiMan proposed the following answer:
>> As already mentioned in comment #1:
>> SikuliX only sees, what is currently visible on the screen. If something
>> is in background, then you have to take care, that it comes to foreground,
>> before SikuliX can see the window content for example.
>>
>> To arrange the screen content (windows), the GUI and the apps running is
>> completely your responsibility.
>>
>> --
>> If this answers your question, please go to the following page to let us
>> know that it is solved:
>> https://answers.launchpad.net/sikuli/+question/658802/+confi
>> rm?answer_id=2
>>
>> If you still need help, you can reply to this email or go to the
>> following page to enter your feedback:
>> https://answers.launchpad.net/sikuli/+question/658802
>>
>> You received this question notification because you asked the question.
>>
>
>

The source code is here: https://github.com/RaiMan/SikuliX-2014

You can look at the Screen Java class to see how it works.

Sikuli basically uses the existing Java Robot class to recognize the screen. How it works in each operational system is implemented inside Java Virtual Machine itself.

Oh. I guess I would have to replace Robot with BetterRobot and bind to the
native modules then. Thank you for the references I'll check the code out!

On Thu, Oct 5, 2017 at 6:04 PM, Edmundo V. Neto <
<email address hidden>> wrote:

> Your question #658802 on Sikuli changed:
> https://answers.launchpad.net/sikuli/+question/658802
>
> Edmundo V. Neto posted a new comment:
> The source code is here: https://github.com/RaiMan/SikuliX-2014
>
> You can look at the Screen Java class to see how it works.
>
> Sikuli basically uses the existing Java Robot class to recognize the
> screen. How it works in each operational system is implemented inside
> Java Virtual Machine itself.
>
> --
> You received this question notification because you asked the question.
>

@Brian
For what you want to do, the Java class Robot is not the right target for replacement/enrichment.

You have to design a superclass (e.g. named GUI) of class Region (and probably Location), that is the base class, to define a pixel area on the screen, that should be acted on.

Your superclass has to handle the highlevel window and GUI-element accesses (like Selenium for a web-GUI) and translate it to pixel areas and locations:

myElement = GUI.getGuiElement("some descriptive information about the needed GUI element")
myElement.click()

The first line, is what you have to develop. The second line internally uses the existing feature of GUI subclass Region.