2 dimensional virtual slider

Asked by tyag

I am trying to automate calibration of the position of a rectangular printed circuit board. I have two markers on the board and a calibrating camera (with cross hairs) that automatically positions itself very close to each of these markers in turn.

A software projects the camera output to screen and provides buttons that allow the fine movement of the camera in 2 dimensions (up/down/left/right) with respective arrow icons.

The plan is to use sikuli to automate the calibration process ..i.e. Make sikuli
1...recognize that the 'marker' is not at the Centre of the crosshairs
2...recognize if it is to the left or right of the vertical crosshair and use the left/right arrows to bring it to vertical '0'
3...recognize if it is above or below the horizontal crosshair and use the up/down arrows to bring it to horizontal '0'
4...hit the save/set button

I have reviewed your example of how you can move the slider to adjust volume, but this is in 1 dimension. I was wondering if there is a way to achieve the above more complex activity in sikuli..

Would be awesome if we could do this...and it may just be the first of its kind of use of sikuli in a real brick-and-mortar manufacturing environment .

I can send pictures of the actual software, but don't see an option for attachments.

Question information

Language:
English Edit question
Status:
Answered
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Manfred Hampl (m-hampl) said :
#1

Just some ideas:

(Please clarify: are there two independent crosshairs for vertical and horizontal alignment or is it a single combined one?)

You will need an image of the marker alone, and an image of the crosshair alone.
search for both these images in the target region
compare the x values (of the image centers), and depending on sign of the difference press the left or right buttons

A problem will arise when the alignment is already very close, and the marker and the crosshair will overlap!

similar with y values and up/down buttons. You might need an image with the target marker and crosshair already overlapping in one dimension.

Finally you can check whether you find an image with target marker and crosshair fully matching.

Revision history for this message
tyag (tyagoo) said :
#2

Thanks ... I did consider the above steps but ran into the same issue you have mentioned...what if the initial marker position was touching one or both crosshairs...the marker would not be recognized as it wouldn't be distinct from the crosshair.

And if we did take into account the above, I am not sure how much time it would take to complete move, check, move again iterations.

I am new to sikuli....was wondering if this is a feasible approach:

0. Find out by actual testing beforehand how many pixels a single push of each of the arrows moves the crosshair by.
1. If there is a way to find the coordinates of the 'center-point' of each of the reference images (crosshair and the marker) find the delta x and delta y (pixels) vector needed to move the marker to the crosshair(only one crosshair exists)
2. Multiply the result of step 0 with the delta x and delta y vectors to determine how many button pushes (and of which button - arrow) are needed to move the crosshair to coincide with the Centre point of the marker.

If Feasible.... can someone help me with the code for step 1 above..

Revision history for this message
Manfred Hampl (m-hampl) said :
#3

How do the crosshair and the target marker really look like?
Is the crosshair always at the same screen position, or is that also variable?

You could use any free image hosting service on the web to upload sample pictures and paste the links into this question document that we can see the pics.

Revision history for this message
tyag (tyagoo) said :
#5

Link to crosshair pic..

https://ibb.co/nvUy9v

The marker is just a black circle (filled in black color)....will try to send a link to that too later.

The crosshair will be in the same position on the screen always (although technically it is the camera which has the crosshair that actually moves to the targeted marker when you click the arrow icons)

Revision history for this message
Manfred Hampl (m-hampl) said :
#6

Ok, the image helps to get a better understanding.
Just some thoughts:

The crosshair seems to be just a helping aid, you do not really need it, because the aim is bringing the target market to a certain x- and y-position (which is the center of the crosshair). And this position seems to be fixed such that it can be calculated once and need not be re-assessed based on an image of the crosshair.

Some additional factors to be taken into account:

To clarify: You have to click the buttons with the arrows to move the target, not the keyboard keys up / down / left / right ?
How much does the target marker move if you press the up / down / left / right buttons (in terms of fractions of the crosshair box)?
Is the amount of movement dependent on the duration of pressing these arrow buttons, or do long and short button presses move the same amount?
Once you have brought the marker to the center vertically and then focus on horizontal alignment, will it stay centered vertically, or will the movement in the second direction disturb (a bit) the other direction, such that you have to go back to alignment of the other direction again?
Is there a limit how far off the center the final position is allowed, or has it to be 0 in any case?

Revision history for this message
tyag (tyagoo) said :
#7

To clarify: You have to click the buttons with the arrows to move the target, not the keyboard keys up / down / left / right ?

-Yes

How much does the target marker move if you press the up / down / left / right buttons (in terms of fractions of the crosshair box)?

-There are two options (large steps and small steps that can be toggled with another button) and it always moves it by a fixed amount for each click of any of the arrow buttons. The movement distance is consistent and by the same value each time.

Is the amount of movement dependent on the duration of pressing these arrow buttons, or do long and short button presses move the same amount?

-yes.. if u keep the button pressed it moves continuously until u release.

Once you have brought the marker to the center vertically and then focus on horizontal alignment, will it stay centered vertically, or will the movement in the second direction disturb (a bit) the other direction, such that you have to go back to alignment of the other direction again?

-each movement is independent and does not disturb the other

Is there a limit how far off the center the final position is allowed, or has it to be 0 in any case?

- final position should be zero

Revision history for this message
Manfred Hampl (m-hampl) said :
#8

Developing such program without having access to the real system is almost impossible.
I am trying to outline the skeleton, but cannot provide a final solution.

You need:
a screenshot of the target marker ("marker,png")
the x-y-position of the center of the crosshair
the x-y-positions of the (center of the) up / down / right / left arrow buttons, eventually also the position of the button to switch from large to small movement.
the coordinates of the region with the crosshair.
the (average) amount of changing the x- respectively y-coordinates with one click on the arrow buttons in large and small movement

Now two proposals:

(eventually: switch to large step movement)
Search for the marker in the region of the crosshair
(if not found - abort processing)
calculate the difference in x coordinates between the center of the crosshair and the center of the marker position identified in the previous step
divide by the average distance of a large step
if positive:
 loop this many times to press the right button, (probably needing wait statements between)
else if negative:
 loop this many times to press the left button, (probably needing wait statements between)
(eventually: switch to small step and repeat from "search the marker" with small steps)

Do more or less the same in the y-direction

++++++++

Or another possibility with permanent observation of the marker movement:

Search for the marker in the region of the crosshair
(if not found - abort processing)
calculate the difference in x coordinates between the center of the crosshair and the center of the marker position identified in the previous step

switch to large step
loop until absolute value of difference between marker's x-position and center of crosshair is smaller than tolerance
 if difference >= average large step
  press left
 else if difference >= average small step
  switch to small step
  press left
 else if difference <= -(average small step)
  press right
 else if difference <= -(average small step)
  switch to small step
  press right
 search for marker again
 calculate the difference in x coordinates between the center of the crosshair and the center of the new marker position

calculate the difference in y coordinates between the center of the crosshair and the center of the latest marker position
switch to large step
do the loop as above for the alignment action in the other direction

Remark: the tolerance for the loop criterion should not be smaller than about two thirds of a small movement. Otherwise the result might be an oscillation between two positions slightly off the center.

Also here you might need to add wait statements after the button presses to ensure that the alignment action has already been completed.

Revision history for this message
tyag (tyagoo) said :
#9

Thanks for the detailed response.

However once the marker touches or overlaps the crosshair/s it will not match the image of the marker stored initially... so how will the iterations continue at this point.

Revision history for this message
tyag (tyagoo) said :
#10

I guess the first of the two proposals may work provided we don't look for the marker image except initially to determine its coordinates.

Add a step to Store the current calculated position of marker after every press of arrow. So at any time system has the current position of the marker.

End loop when current position of marker coincides (within limits) with crosshairs coordinates.

Revision history for this message
Manfred Hampl (m-hampl) said :
#11

Re: "However once the marker touches or overlaps the crosshair/s it will not match the image of the marker stored initially... "

Searching for images in Sikuli has a similarity parameter.
The image on screen need not be 100% identical to the stored one.
It should be possible to set the similarity degree to a value that the marker is found even if it partly overlaps the crosshair.
This of course will need testing on the real system.

Can you help with this problem?

Provide an answer of your own, or ask tyag for more information if necessary.

To post a message you must log in.