Question #126270 “Is there an Online .skl processsing option to ...” : Questions : SikuliX

Revision history for this message

RaiMan (raimund-hocke) said on 2010-09-21:

#1

- user input is queued and processed sequentially
does this mean, that the user input is waiting as something like a csv-file to be processed (entered into forms of the db app)?
if True: why not have a Sikuli script, that handles the queue and processing is only starting a def()? No start up time in this case.
you can even use subprocessing, if somehow useful.

- the bounds of program windows
especially, in windows, to get the bounds of an app window is a little def() using the keyboard functions of the window menu and getMouseLocation (interested?). But you are right, this should be a feature of Sikuli.

- performance
restricting find operations to a region are a must if you want some performance. when working with forms, tabbing through the fields is a must too. on top you can only gain some more performance, when you avoid as much find operations as possible. In most form driven apps, it is a one time effort, to evaluate the relative positions of necessary click points and use coordinates with click(Location) instead of click(Pattern/Image).
Often overlooked: using < exists(img, 0) > instead of < try: find(img) except: #some code > in the standard saves about 2 seconds if img is not found(), when it should be there.

- java
I have no experiences, but it could be worth to test, wether things run faster, if the workflow is written in java and the usage of sikuli functions is reduced to the minimum.

- One more thing: Have you put some time.time() based evals in your script, to evaluate the timing structure of your script?
start = time.time()
# .... some code
print "check 1:", time.time()-start
# ... some code
print "check 2:", time.time()-start

and so on.

Revision history for this message

Lance Jensen (lance-ti) said on 2010-09-21:

#2

Thanks for your comments. I have done almost all the things you have suggested. But I want to walk through your comments to make sure that I understand your statments and clarify mine.

Finally I will also make a suggestion about a possible app object and its potential features that could make sikuli easier to use and run faster.

First statement / question:

More description: I have a dedicated pc that contains a client application to a database in which data must be entered. A persistent server exists on the system that accepts socket connections and allows remote systems to run sikuli programs to update the database. Since there is only one screen the clients are queued so that only one can execute a sikuli program at once. Currently I am using the running an exec command of the Sikuli-IDE.exe with the proper arguments for each client connection when it is that clients turn to run. When a new run is executed there is a start-up time of 5-9 seconds where the ide outputs the following statements:

             ScreenMatchProxy loaded
             VDictProxy loaded
             Win32util loaded
             ScreenMatchProxy loaded

I am trying to cut out this time. If I can get rid of this it will be a 20-30% speed improvement. If there is a mode where I can leave Sikuli running and just keep feeding it .skl files that would be ideal. If leaning to use the java library will give me this capability then I can do that as well but I am reluctant to do so at this time.

Bounds of windows:

Every find has a region associated with it. Due to random placement and robustness Issues I had to still use about a quarter of the screen for each match area. Run time gain in this area is not worth the reliability loss of being more aggressive in this area.

Performance:

I will try the change from find to exists. Once true then I perform a find? I have about 3 finds... Also is onAppear() offer equivalent savings or does it use find as well...

As for using tabs I started that way and achieved a best case performance of 9 seconds per transaction but found out the hard way that the form was dynamic and buttons would appear or disappear based on field values. This is what prompted me to ask you the key stroke questions yesterday to avoid using more finds...

Java:

My server is written in Java but I am getting pressure to release this and move on. It's only been 4 days. In any case there are not many examples out there that I saw for java usage so I am not certain I want to trek that path unless I can gain performance.

Timing:

I have only timed the round trip time. I learned just enough python to get the job rolling so I have not littered it with time statements. Watching it run and counting has lead to the 4-9 second estimate. Finds take about 5 seconds a piece.

Enhancement App Object:

In my mind making an app object would be very helpful and make Silkui easier to use and more reliable. An app should have an executiblePath, a name handle, a default size varibles. It should have .start()/run() to start the program, .switch() to toggle it between forground or background, .size() to set to default size with no arguments or to a dimension if arguments are passed, and getCenter(). With this type of structure then I think sikuli scripts could be more transportable. As it is, time optimizations cost reliability and transportability which affects the scaling of your software.

More questions on performance:

My application does not care about color.  If I set the video card to use a lower color quality with this help processing time.  I know that if I could get it to gray-scale it should reduce the work by 2/3s since there would be two less channels to process.  I could also reduce screen resolution.  Will these things break the .skl files I have already made?  I would guess that the answer is no but I really do not know a lot about the implementation details.

Revision history for this message

RaiMan (raimund-hocke) said on 2010-09-22:

#3

I think the performance of your script as such cannot be improved dramatically anymore:
- exists(img,0) only makes sense to use, if the find could fail (a failing find in the standard lasts about 3 seconds). The exists() variant will return after the first try (so normally after max 1 second) and return False in case of FindFailed. If successful, it behaves exactly like find() - so no need for an additional find()
- onAppear(): no performance advantage over find() - only handling comfort for "parallel" find()'s. BUT: does not run on Windows in current version 10.2 - so don't try now.
- screen resolution does not have any effect, since it is comparable to restricting the find's to a smaller region
- gray scale: from my few code visits and bug/patch descriptions, I think Sikuli is fixed to a color screen. But I'm not sure. So if it is easy to test, just try out

--- Start-up time:
As far as I understand, there is some trigger on your server, that fires a .skl with some client specific information.
So my idea was:
make your .skl a def() and make the "trigger-handling" in a Sikuli script that is running all the time and dispatches the run for each client just by calling the def() with the same parameters as you now use for your .skl - so you cut off startup time.

--- App object
good idea, but I think it is relatively complex especially with respect to the fact, that this feature has to be unique on the script level for the three environments (Windows, Linux, Mac). As already said, it is relatively easy, to define an App class on script level yourself if you really need it, that wraps all the functions you talk about. But your suggestion leaves out the really tricky cases, where you want to know all the open windows of one app and handle each of them.
Already now there are big differences: switchApp() in Windows selects a window by matching the window title (so you can switch to one specific window of an app that has open more than one, if they have different window titles), on Mac this is not possible now - Linux: don't know.
nevertheless I will make it a bug feature request (expecting that it will be triaged with medium priority ;-)

I think the performance of your script as such cannot be improved dramatically anymore:
- exists(img,0) only makes sense to use, if the find could fail (a failing find in the standard lasts about 3 seconds). The exists() variant will return after the first try (so normally after max 1 second) and return False in case of FindFailed. If successful, it behaves exactly like find() - so no need for an additional find()
- onAppear(): no performance advantage over find() - only handling comfort for "parallel" find()'s. BUT: does not run on Windows in current version 10.2 - so don't try now.
- screen resolution does not have any effect, since it is comparable to restricting the find's to a smaller region
- gray scale: from my few code visits and bug/patch descriptions, I think Sikuli is fixed to a color screen. But I'm not sure. So if it is easy to test, just try out

--- Start-up time:
As far as I understand, there is some trigger on your server, that fires a .skl with some client specific information.
So my idea was:
make your .skl a def() and make the "trigger-handling" in a Sikuli script that is running all the time and dispatches the run for each client just by calling the def() with the same parameters as you now use for your .skl - so you cut off startup time.

--- App object
good idea, but I think it is relatively complex especially with respect to the fact, that this feature has to be unique on the script level for the three environments (Windows, Linux, Mac). As already said, it is relatively easy, to define an App class on script level yourself if you really need it, that wraps all the functions you talk about. But your suggestion leaves out the really tricky cases, where you want to know all the open windows of one app and handle each of them.
Already now there are big differences: switchApp() in Windows selects a window by matching the window title (so you can switch to one specific window of an app that has open more than one, if they have different window titles), on Mac this is not possible now - Linux: don't know.
nevertheless I will make it a bug feature request (expecting that it will be triaged with medium priority ;-)

Revision history for this message

Lance Jensen (lance-ti) said on 2010-09-22:

#4

RaiMan,

Could you please point me to an example of what you are suggesting with the def(). I am pretty sure I could accomplish this if I used the Java library but then I would have to hunt through the methods and objects to figure out what I was doing and rewrite all the code. Since I am on a tight time line I can not accommodate that.

If I had an example of how to do that in Jthon then I am sure I could finish it before days end by making each .skl a method.

Best Regards,
Lance

Revision history for this message

RaiMan (raimund-hocke) said on 2010-09-22:

#5

This only makes sense, if you are able to process the queued requests within a Sikuli script, e.g. by appending a new line to a file, that is polled by a dispatcher .skl containing your workflow in a def().

since I don't know yet, how you trigger the processing of your .skl I make it generic:

your script:
def myWorkflow(clientinformation):
# here is the code, that you now have in your .skl
# the parameter handling has to be adjusted, since I
# guess you are currently using sys.argv
return some-information

# this is your dispatcher
# some setup code
while True:
    if there-is-anext-request:
        get-request
        if next-request-is-quit: break # to have an end ;-)
        clientinformation = evaluate-parms-from-request
        retVal = myWorkflow(clientinformation)
        # do some post-processing

so if you put this in a .skl and start it, it should process the incoming client cases without having the start up time.

There is a vary elegant method for a dispatcher, but I have not tested it on windows yet.
XML-RPC

the dispatcher script as a XML-RPC-Server in a .skl:

def myWorkflow(clientinformation):
# here is the code, that you now have in your .skl
# the parameter handling has to be adjusted, since I
# guess you are currently using sys.argv
return some-information

from SimpleXMLRPCServer import SimpleXMLRPCServer as Server
srv = Server(("127.0.0.1", 1337)) # as an example on the same machine
if not srv: exit(1)
srv.register_function(myWorkflow)
srv.serve_forever()

thats all. just start it and it waits for processing calls of myWorkflow.

To "call" myWorkflow, you have to implement a XML-RPC Client based on a XML-RPC-Library, that should be available in Java too, so you could just change your call to your .skl into an XML-RPC-Request.

In Python/Jython this is as easy as saying:
import xmlrpclib
cli = xmlrpclib.ServerProxy("http://127.0.0.1:1337")
retval = cli.workflow(clientinformation)

but without having it checked, I guess it will be similar in java.

to make it nice and easy, the parameters and return values used should be strings and/or numbers, since internally the http POST is used (though some more things are possible).

This only makes sense, if you are able to process the queued requests within a Sikuli script, e.g. by appending a new line to a file, that is polled by a dispatcher .skl containing your workflow in a def().

since I don't know yet, how you trigger the processing of your .skl I make it generic:

your script:
def myWorkflow(clientinformation):
# here is the code, that you now have in your .skl
# the parameter handling has to be adjusted, since I 
# guess you are currently using sys.argv
    return some-information

# this is your dispatcher
# some setup code
while True:
    if there-is-anext-request:
        get-request
        if next-request-is-quit: break # to have an end ;-)
        clientinformation = evaluate-parms-from-request
        retVal = myWorkflow(clientinformation)
        # do some post-processing

so if you put this in a .skl and start it, it should process the incoming client cases without having the start up time.

There is a vary elegant method for a dispatcher, but I have not tested it on windows yet.
XML-RPC

the dispatcher script as a XML-RPC-Server in a .skl:

def myWorkflow(clientinformation):
# here is the code, that you now have in your .skl
# the parameter handling has to be adjusted, since I 
# guess you are currently using sys.argv
    return some-information

from SimpleXMLRPCServer import SimpleXMLRPCServer as Server
srv = Server(("127.0.0.1", 1337)) # as an example on the same machine
if not srv: exit(1)
srv.register_function(myWorkflow)
srv.serve_forever()

thats all. just start it and it waits for processing calls of myWorkflow.

To "call" myWorkflow, you have to implement a XML-RPC Client based on a XML-RPC-Library, that should be available in Java too, so you could just change your call to your .skl into an XML-RPC-Request.

In Python/Jython this is as easy as saying:
import xmlrpclib
cli = xmlrpclib.ServerProxy("http://127.0.0.1:1337")
retval = cli.workflow(clientinformation)

but without having it checked, I guess it will be similar in java.

to make it nice and easy, the parameters and return values used should be strings and/or numbers, since internally the http POST is used (though some more things are possible).

Revision history for this message

Lance Jensen (lance-ti) said on 2010-09-22:

#6

Thanks for your response. As it is I think I will have to live with the speed as is and if they want a speed increase then they will have to buy new hardware or give me more time to implement it. My dispatcher is in Java and calls the executable each time. I was pretty sure you where suggesting this but I did not know for sure. This project just moved very fast from proof of concept to implementation where the proof of concept code was used for the production solution.

Best regards,
Lance

Revision history for this message

RaiMan (raimund-hocke) said on 2010-09-22:

#7

I understand, if you don't have any chance to go into the java code.
But if possible, you only have to change the call to the executable into a XML-RPC request, which should be a matter of minutes.

On the Sikuli side you just have to revise the parameter handling and may be the return value - and add/rearrange some statements.

... and I tell you: runs like a charm.

All the best to you and your efforts.

Revision history for this message

Lance Jensen (lance-ti) said on 2010-09-22:

#8

Perhaps I misunderstand. I am assuming you are saying write a new .slk file and have it read files placed in the same directory and execute the appropriate definition. To accomplish this I would have to learn more of Jython, convert the skl's to def()s which should not be too hard. Finally I would have to modify the server to write the arguments to an execution file which will be read and the appropriate def() would be executed.

It makes sense. I just don't have the time to rebuild everything and revalidate it. It will have to wait. I was really looking to see if there was an canned solution that was already in place. With a little more knowledge of Jython I could make it work but I do not have time for that. You have been really helpful. Thanks for your expertise and quick response to your questions.

Best Regards,
Lance Jensen
Circuit Design Repair & Diagnostics (F.I.B. Edit)
Office Phone: (972)-995-8910 Cell: (214)-882-3888
Priority Coordinators:
HPA: Chris Haga <email address hidden>
HVAL: Claire Jung <email address hidden>
PWR: Naweed Anjum <email address hidden>
WTBU: Kevin O'Halloran <email address hidden>
ASP: Debra Guillemaud <email address hidden>

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of RaiMan
Sent: Wednesday, September 22, 2010 10:03 AM
To: Jensen, Lance
Subject: Re: [Question #126270]: Is there an Online .skl processsing option to reduce run time

Your question #126270 on Sikuli changed:
https://answers.launchpad.net/sikuli/+question/126270

RaiMan posted a new comment:
I understand, if you don't have any chance to go into the java code.
But if possible, you only have to change the call to the executable into a XML-RPC request, which should be a matter of minutes.

On the Sikuli side you just have to revise the parameter handling and
may be the return value - and add/rearrange some statements.

... and I tell you: runs like a charm.

All the best to you and your efforts.

--
You received this question notification because you are a direct
subscriber of the question.

Perhaps I misunderstand.  I am assuming you are saying write a new .slk file and have it read files placed in the same directory and execute the appropriate definition.  To accomplish this I would have to learn more of Jython, convert the skl's to def()s which should not be too hard. Finally I would have to modify the server to write the arguments to an execution file which will be read and the appropriate def() would be executed.

It makes sense.  I just don't have the time to rebuild everything and revalidate it.  It will have to wait.  I was really looking to see if there was an canned solution that was already in place.  With a little more knowledge of Jython I could make it work but I do not have time for that.  You have been really helpful.  Thanks for your expertise and quick response to your questions.

Best Regards,
Lance Jensen 
Circuit Design Repair & Diagnostics (F.I.B. Edit) 
Office Phone:  (972)-995-8910 Cell:  (214)-882-3888
Priority Coordinators:
HPA:     Chris Haga                   c-haga@ti.com 
HVAL:   Claire Jung                        c-jung1@ti.com
PWR:    Naweed Anjum             naweed@ti.com
WTBU:  Kevin O'Halloran            k-ohalloran@ti.com
ASP:     Debra Guillemaud         d-guillemaud@ti.com

-----Original Message-----
From: bounces@canonical.com [mailto:bounces@canonical.com] On Behalf Of RaiMan
Sent: Wednesday, September 22, 2010 10:03 AM
To: Jensen, Lance
Subject: Re: [Question #126270]: Is there an Online .skl processsing option to reduce run time

Your question #126270 on Sikuli changed:
https://answers.launchpad.net/sikuli/+question/126270

RaiMan posted a new comment:
I understand, if you don't have any chance to go into the java code.
But if possible, you only have to change the call to the executable into a XML-RPC request, which should be a matter of minutes.

On the Sikuli side you just have to revise the parameter handling and
may be the return value - and add/rearrange some statements.

... and I tell you: runs like a charm.

All the best to you and your efforts.

-- 
You received this question notification because you are a direct
subscriber of the question.

Revision history for this message

RaiMan (raimund-hocke) said on 2010-09-22:

#9

Yes you misunderstood.

Forget about any files and this stuff, I'm only talking about XML-RPC, which is a standard to remotely call functions using the http protocol.

in your java dispatching you in the moment say something like:
call myWorkflow.skl using parm1, parm2, parm3

this turns into:
retval = make-XML-RPC-request(myWorkflow, parm1, parm2, parm3)

no need to change anything else, since your parameters are already strings.

on the Sikuli side it looks like this in the moment (your .skl):

parm1 = sys.argv[1]
parm2 = sys.argv[2]
parm3 = sys.argv[3]
find(img1)
click(img2)
type(parm1)
type(Key.TAB)
#... and so on
exit()

this simply has to be changed to
def myWorkflow(parm1, parm2, parm3):
   find(img1)
   click(img2)
   type(parm1)
   type(Key.TAB)
   # ... and so on
   return some-information-as-string

and after that you have to insert:
from SimpleXMLRPCServer import SimpleXMLRPCServer as Server
srv = Server(("127.0.0.1", 1337)) # as an example on the same machine
if not srv: exit(1)
srv.register_function(myWorkflow)
srv.serve_forever()

put it in a .skl, start it and look as your requests are processed.

Nice and easy, if
- you know how to activate and use XML-RPC-Client in Java
- and it works on windows as I mentioned

Revision history for this message

Lance Jensen (lance-ti) said on 2010-09-22:

#10

RaiMan,

Thanks for taking me to school. Currently I do not know anything about these protocols. But thanks to your willingness to instruct and your detailed information I will likely invest the time to learn this standard so that I can utilize it in the future.

I need to become more web savvy but I have never found the time. Assuming I get the time is it ok to post questions about this topic to the Sikuli question answer board on lauchpad? If I do start something like this it would not be for several months.

Thanks again for all your kindness.

Best Regards,
Lance Jensen

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of RaiMan
Sent: Wednesday, September 22, 2010 10:39 AM
To: Jensen, Lance
Subject: RE: [Question #126270]: Is there an Online .skl processsing option to reduce run time

Your question #126270 on Sikuli changed:
https://answers.launchpad.net/sikuli/+question/126270

RaiMan posted a new comment:
Yes you misunderstood.

Forget about any files and this stuff, I'm only talking about XML-RPC,
which is a standard to remotely call functions using the http protocol.

in your java dispatching you in the moment say something like:
call myWorkflow.skl using parm1, parm2, parm3

this turns into:
retval = make-XML-RPC-request(myWorkflow, parm1, parm2, parm3)

no need to change anything else, since your parameters are already
strings.

on the Sikuli side it looks like this in the moment (your .skl):

parm1 = sys.argv[1]
parm2 = sys.argv[2]
parm3 = sys.argv[3]
find(img1)
click(img2)
type(parm1)
type(Key.TAB)
#... and so on
exit()

this simply has to be changed to
def myWorkflow(parm1, parm2, parm3):
   find(img1)
   click(img2)
   type(parm1)
   type(Key.TAB)
   # ... and so on
   return some-information-as-string

and after that you have to insert:
from SimpleXMLRPCServer import SimpleXMLRPCServer as Server
srv = Server(("127.0.0.1", 1337)) # as an example on the same machine
if not srv: exit(1)
srv.register_function(myWorkflow)
srv.serve_forever()

put it in a .skl, start it and look as your requests are processed.

Nice and easy, if
- you know how to activate and use XML-RPC-Client in Java
- and it works on windows as I mentioned

--
You received this question notification because you are a direct
subscriber of the question.

RaiMan,

Thanks for taking me to school.  Currently I do not know anything about these protocols.  But thanks to your willingness to instruct and your detailed information I will likely invest the time to learn this standard so that I can utilize it in the future.

I need to become more web savvy but I have never found the time.  Assuming I get the time is it ok to post questions about this topic to the Sikuli question answer board on lauchpad?  If I do start something like this it would not be for several months.

Thanks again for all your kindness.

Best Regards,
Lance Jensen

-----Original Message-----
From: bounces@canonical.com [mailto:bounces@canonical.com] On Behalf Of RaiMan
Sent: Wednesday, September 22, 2010 10:39 AM
To: Jensen, Lance
Subject: RE: [Question #126270]: Is there an Online .skl processsing option to reduce run time