Optimizing resource usage

Asked by cemara

Hi, i'm working on home assistant (HA) display tablet based on windows10 LTSC 2019 for elderly,

the tablet serves various functions but mainly as viewing cctv station to know who's in front.

Now i'd also like to the tablet for auto answer videocall for emergency use. The only software capable for this so far is skype.

Problem is after a call session, skype contact window blocks the HA instance, and it wont minimize automatically.

So i've used sikuli like this to bring HA in front after each skype call. Where skype .png is a call / hangup button pic.

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
While True
  wait(skype.png, FOREVER)
  waitVanish(skype.png, FOREVER)
  HomeAssistant.focus()

The problem is this takes large toll on resources, at times 50-60% which is understandable since it's a NUC5cphy machine.
And it makes other apps impossible to run
But i'm restricted in terms of resoure since the whole system runs on POE (30 watts max)
So is there any other approach that is more resource friendly for my case?

Question information

Language:
English Edit question
Status:
Expired
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

Since I cannot see your screen content and do not really understand the workflow based on your info, I give you this general advise: When using SikuliX for workflow automation, first try to script, what you would do manually to reach the target situation. In most situations this is sufficient. Then add some logic, to evaluate the auto start of you workflow. All this usually is a sequence of wait's and click's.

on resource consumption: in the standard SikuliX tries 3 times per second when waiting for an image. On slow machines this usually leads to up to 100% peak machine use and high average over the time (especially with FOREVER).

So you either have to use exists(img, 0) (only one search) and a loop construction that wait's in between (e.g. wait(0.5)) or you use the parameter Settings.WaitScanRate to reduce the tries per second (e.g. 1 is 1 per second, 0.5 is one every 2 seconds, ...).

Revision history for this message
cemara (cemara) said :
#2

The workflow is nothing complex,
1. Skype call set for autoanswer.
2. Make call to tablet, Skype autoanswered, a contact window shows up,
3. wait until that call is finished > this part can only be done with GUI based automation since skype has no API handle.
4. So i'm using a call button pic that only shows up during a call session, and wait until that session is finished.
Thus wait and watvanish is needed.
5. Bring home assistant window on top again since that call window wont minimized by itself, also sometimes there is call quality survey window as well, this should be covered too.

I've looked into the waitscanrate but apparently it's used for region, how would i use it for single image like this?

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
While True
  wait(skype.png, FOREVER)
  waitVanish(skype.png, FOREVER)
  HomeAssistant.focus()

Revision history for this message
RaiMan (raimund-hocke) said :
#3

--- ...waitscanrate but apparently it's used for region,
every image search towards a screen in SikuliX is done on a Region (undotted it is the whole screen).
Hence setting WaitScanRate is respected by all following screen image searches until changed again.

waitVanish might not work as expected in earlier versions of SikuliX.
Same goes for someApp.focus() on Windows (try with App.focus("part of window-title"))

instead of waitVanish you might use:

while exists(img, 0):
    wait(0.5)

Revision history for this message
cemara (cemara) said :
#4

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
Settings.WaitScanRate = 3
While True
  wait(skype.png, FOREVER)
  while exists(skype.png, 0):
    wait(0.5)
  HomeAssistant.focus()

So i've tested this but when running the cpu usage is still high, am i coding it correctly?

Revision history for this message
Manfred Hampl (m-hampl) said :
#5

Are you sure that you need the "While True"?

I do not know the details of your system, but I would expect that the following may be sufficient, unless the skype window (identified by skype.png) re-appears again and again.

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
Settings.WaitScanRate = 3
wait(skype.png, FOREVER)
while exists(skype.png, 0):
  wait(0.5)
HomeAssistant.focus()

The WaitScanRate and wait(0.5) values could also be modified, if it is not time-critical.

Revision history for this message
cemara (cemara) said :
#6

@manfred hampl,
Wouldn't that end the script once 1 call session is made?
My intention is to make 24/7 skype monitoring system with sikuli.

Revision history for this message
Manfred Hampl (m-hampl) said :
#7

Sorry, it seems that I misunderstood the requirements.

If you want to keep the script running to answer skype calls multiple times, whenever they come in, then you need the "While True".

In that case the only possibility that I see is fine-tuning the frequency of checking for the skype pic by reducing the scan rate for the skype image or increasing wait time between the checks.

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
Settings.WaitScanRate = 3 # -reduce this number to 0.5 for checking only once every two seconds, etc.
While True
  wait(skype.png, FOREVER)
  while exists(skype.png, 0):
    wait(0.5) # increase this number to 2.0 for checking only every two seconds, etc.
  HomeAssistant.focus()

Can the skype.png button appear anywhere on the screen, or is it always in the same spot? In the latter case you should search only in that region instead of the whole screen. This should also reduce resource usage.

Revision history for this message
RaiMan (raimund-hocke) said :
#8

@Manfred: excellent answer as always.Thanks and all the best.

Revision history for this message
cemara (cemara) said :
#9

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
Settings.WaitScanRate = 0.5
while True:
  Region(1381,22,519,964).inside().wait("1659658059381.png", FOREVER)
  while exists("1659658059381.png", 0):
    wait(3)
  HomeAssistant.focus()

The skype button seems to always appear on the right side, I've narrowed it down to 1/4 upper right side.
is this the right way to code it? Any other possible optimization?
But it doesn't seem to help much reducing resource when i run it.
It seems now there are 2 openjdk bla3 instances on task manager instead of 1 when I run it..

Hmm other alternative would be running the whole thing in ubuntu light since skype sikuli and HA are also available in ubuntu, might take a while. But i'll see if I can get it running with spare ssd.
Thanks for all the help until now.

Revision history for this message
Manfred Hampl (m-hampl) said (last edit ):
#10

Also "exists" should be restricted to the smallest possible region.

Something like

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
Settings.WaitScanRate = 0.5
myRegion = Region(1381,22,519,964)
while True:
  myRegion.wait("1659658059381.png", FOREVER)
  while myRegion.exists("1659658059381.png", 0):
    wait(3)
  HomeAssistant.focus()

Further enhancement of the script could be to identify the exact position where the skype image is found in the "wait" statement, and then use that region for re-checking if it is still existing in the "exists" statement.

Perhaps something like

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
Settings.WaitScanRate = 0.5
baseRegion = Region(1381,22,519,964)
while True:
  foundRegion = baseRegion.wait("1659658059381.png", FOREVER)
  while foundRegion.exists("1659658059381.png", 0):
    wait(3)
  HomeAssistant.focus()

(For the openjdk processes I do not know enough about the technical details, hopeful RaiMan can answer that.)

Revision history for this message
RaiMan (raimund-hocke) said :
#11

What is your exact environment? versions of Windows, Java and SikuliX?
How do you run the script?

Revision history for this message
cemara (cemara) said :
#12

Windows10 ltsc 2019,
sikulixide-2.0.5-win.jar
OpenJDK16U-jdk_x64_windows_hotspot_16.0.2_7.msi

With above code, I export to runnable jar, and run it..

Revision history for this message
RaiMan (raimund-hocke) said :
#13

Ok, then you have to live with the situation, after having followed Manfred's suggestions.

You might also try with the pre-version (snapshot) of 2.0.6 from here:
https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=com.sikulix&a=sikulixidewin&v=2.0.6-SNAPSHOT&e=jar

Export to jar cannot be used here.
You have to run your script from command line like so:
java -jar path-to-skulixide....jar -r path-to-your-scriptfolder

Revision history for this message
cemara (cemara) said (last edit ):
#14

Sorry for the late reply, I was looking for AIO device strong enough for this and still can be powered by POE.
I ended up with Asus V161.

Now I think this code can run without freezing the system :

HomeAssistant = App(r"c:\HA\Home_Assistant.exe")
Settings.WaitScanRate = 0.5
while True:
  Region(1381,22,519,964).inside().wait("1659658059381.png", FOREVER)
  while exists("1659658059381.png", 0):
    wait(3)
  HomeAssistant.focus()

where 1659658059381.png = red hangup button

But there's an issue, the first time the code runs, it works properly with this flow :
1. Home assistant running, skype running in background
2. Make skype call from phone, skype call window appears, sikuli detects red hangup button.
3. Wait until red hangup button dissapear (call ends), bring Home assistant window to front.
The issue is,
4. Second time I make skype call, skype call window wont appear, it stays in the background,

Seems like Home assistant window is always is front after the first time code is running.
Any idea why?
If I put end at the end of the code above, it would terminate the monitoring process..

Revision history for this message
Manfred Hampl (m-hampl) said :
#15

If you would have to do this manually, how do you know that a new skype call is coming in?
Is there an additional icon in the task bar etc. etc.?
Then you may be able to use that as trigger in the script for the second round (or maybe also even for the first one).

Revision history for this message
cemara (cemara) said (last edit ):
#16

@m-hampl
By detecting the red hangup button from skype call window.
Problem is for the second round, skype call window stays in the background and won't go to front.
So sikuli can't detect the red button anymore..
It's like HomeAssistant.focus() effect wont reset after the code runs.

Well the taskbar shows blinking skype icon when there's incoming calls,
but If I'm going to use that then I will need to identify skype call window name in order to bring the window to front,
And the last time I tried that with autoit, there was no difference between the name of call window and skype main app window (hence I went for sikuli)

Besides I don't think there's any visual difference for call made and call ended from taskbar icon, so it cant be used I think

Revision history for this message
Manfred Hampl (m-hampl) said :
#17

"By detecting the red hangup button from skype call window.
Problem is for the second round, skype call window stays in the background and won't go to front.
So sikuli can't detect the red button anymore.."

Just talking about the second round:

How do you know that there is a red hangup button, if it is only in the background?

If you do it manually, how can you bring the skype call window to front?

Revision history for this message
cemara (cemara) said (last edit ):
#18

Sorry, after multiple testing's.
I'm not sure this is even related so sikuli or not.
So even without the java process that sikuli generates, skype won't go to front with incoming calls.
Really weird, it stays in the background and the taskbar icon is blinking.
This complicates things even more..
Maybe make a code to click skype taskbar icon when it's blinking to bring it to front?
But the problem is, skype sometimes bring it to front, and the taskbar doesn't blink..

A more surefire trigger would be when the mic or webcam is being used, but I've searched and haven't figure out this one yet.

Revision history for this message
Launchpad Janitor (janitor) said :
#19

This question was expired because it remained in the 'Open' state without activity for the last 15 days.