keyboard/mouse emulation and pipelight

Asked by Mitch Trachtenberg

Hi,

I'm seeking to be able to interact using prerecorded keystrokes and mouse activity with an in-browser Silverlight UI database but have found it to be unreliable in moving keyboard focus around. There is no direct API to this database, so all interaction has to be through a web browser. I understand (though I could easily be wrong) that this is some sort of security issue in Silverlight. Is the identical interaction problem likely to appear in Pipelight, or does anyone know any workaround that would allow someone to automate interactions like this? Basically, I want to be able to reliably navigate to particular buttons and text entry fields and simulate typing and button presses. Thanks.

Question information

Language:
English Edit question
Status:
Answered
For:
Pipelight Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Michael Müller (mqchael) said :
#1

Hi,

first of all I would recommend disable embedding (start the browser with the environment variable PIPELIGHT_EMBED=0 set), so that the input events directly go to silverlight and do not need to pass the browser. This also prevents the browser from accidentally getting the focus.

Moreover, I would suggest using a Windows software (in the same wineprefix) to simulate input events so that you only feed these events to the wine event system and do not need to simulate the events using the X server. It would most probably be similar to the setup you are currently using on Windows.

The only advantage you could have when using Pipelight, is that you can enforce the windowslses mode (PIPELIGHT_WINDOWLESSMODE=1). This forces all events to go through Pipelight, so that you drop existing ones or add new ones. This would however require modifications to the Pipelight source code. There is already something similar in the code: We hook the fullscreen window and drop the WM_KILLFOCUS event when the "Stay in fullscreen" option is set which prevents Silverlight from leaving fullscreen when using multiple monitors.

Pipelight can give you a better control over the existing events, but we do not have any API interfaces which gives you direct access to the data. Depending on how much data you need to enter into this database, it may be better to reverse engineer how the Silverlight application work so that you can directly send the data to the background server. You may take a look at the network traffic. Since all web requests go either through Pipelight or Wines wininet implementation, you can also take a look at SSL traffic when you add some debug output at the correct places.

Michael

Can you help with this problem?

Provide an answer of your own, or ask Mitch Trachtenberg for more information if necessary.

To post a message you must log in.