dragDrop not working correctly

Asked by Test

Hey, I have a script which has to execute a dragDrop function.

The GUI I'm testing against has a list of items in a sort of list (each in a rectangular box):

item1
item2
item3
item4

And I need to drag 1 item into an area to the right of the list.

However, when I used the dragDrop function on let's say item4, instead, item 2 gets dragged. My code is as follows:

    ref = find(image.png)
    dragDrop(ref.offset(Location(150, 390)), ref.offset(Location(335, 125)))
    wait(1)

(150, 390) are the coordinates of item4 and (335, 125) are the coordinates of the box to drop in. The straight line path cuts through the other items.

I have tried a number of variations of this, including going to a lower level by using the mouseDown(), mouseMove() and mouseUp() functions but none of them seem to work, all of them give me the same result of dragging the wrong image.

Oh, and I also tried drag() and dropAt(). I've also tried substituting the coordinates with images, all producing the same result of another item being dragged, but not the same item all the time.

The area I drop the item into is a box located next to item1. If I need to drag item 4, the path the cursor takes 'cuts' through all the other items, and results in a different item being dragged.

It's very odd, even after I changed Settings.DelayAfterDrag, Settings.DelayBeforeDrop, Settings.MoveMouseDelay, all to no avail. Restarted the Sikuli IDE a couple of times too in case it was an odd JRE bug, but nope.

Any help is appreciated, thanks.

Running on: Sikuli 1.0.1 and on a 64-bit Windows 8 machine

Extra info (unsure of relevance, just putting it here anyway):

* The code above is within a function.
* I do in fact have a dragDrop() command in a previous functionA BUT I do not execute functionA, only this one. The code appears but is never executed.
* Definitely not a GUI error. I can manually drag and drop all items just fine.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
RaiMan Edit question
Solved by:
Test
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

To make things like that robust, you first should evaluate the geometrics of your list and then check using highlight() and/or hover() to make sure the final locations are correct. Do this in a separate script tab in the IDE and finally use the values you found.

In your case you would have seen, that ref.offset(Location(150, 390) has the same dimension as ref, since you are working with regions in this case and hence evaluates to an unwanted click point (centre of ref.offset(Location(150, 390))

Since I suppose you want to hit the centers and have evaluated the offset values accordingly:

ref = find(image.png).getCenter()

might do it, supposing the offset values are correct

Furthermore i recommend to use the relative dimensions of your list and address a list entry with its number.
listFirst = ref.offset(Location(x, y))
listH = 60 # to be evaluated

... and to hover over the 4th item:
item = 4
hover(listFirst.below((item -1) * listH)

this could even be moved into a def like
def dragItem(n):
   # implement the drag here

Revision history for this message
Test (c4456517) said :
#2

I will take some time to try it out, but I'm not sure if that's the problem because I use that same variable as a reference point for other click() functions and they all work fine.

I mean I actually see the cursor going to item 4 and then going into the drop area, but for some reason it drags the wrong item (maybe a delayed mouseDown event but I've tried that already to no avail).

Revision history for this message
RaiMan (raimund-hocke) said :
#3

definitely: if item 4 is dragged, where you expect item 2 to be dragged, then the offset values do not fit (are wrong).

This happens, if you evaluate the offset with reference to the top left corner and not to the centre of the 2 regions (ref and target item). Because in your case, you are working with regions and not with points.
A regions default click point is it's centre, which depends on width and height.

Revision history for this message
Test (c4456517) said :
#4

Ok, that makes sense, but why would I see the cursor moving to the intended location in that case?

Oh and item 2 is dragged instead of item 4 (the item 2 spaces above).

Revision history for this message
Test (c4456517) said :
#5

When I tried this out:

ref = find(image.png).getCenter()

I get this error:

[error] TypeError ( offset(): expected 2 args; got 1 )

When in actuality my code was:

dragDrop(ref.offset(Location(150, 390)), ref.offset(Location(335, 125)))

I finally was able to find a workaround doing the following:

    hover(ref.offset(Location(150, 390)))
    mouseDown(Button.LEFT)
    Settings.MoveMouseDelay = 5
    mouseMove(ref.offset(Location(335, 125)))
    mouseUp()

So what I realized I hadn't tried out was using the lower level functions of the mouse AND changing the speed of the cursor movement. Works like a charm.

Revision history for this message
RaiMan (raimund-hocke) said :
#6

--- error] TypeError ( offset(): expected 2 args; got 1 )

ref = find(image.png).getCenter()

now ref is a Location object and it's offset() function is defined as getting x and y directly:

dragDrop(ref.offset(150, 390), ref.offset(335, 125))

--- your workaround
dragdrop is a convenience feature, that internally is implemented nearly exactly as your workaround.

So it would help me a lot to understand what is happening in your case: could you send me a screenshot silently to my mail at https://launchpad.net/~raimund-hocke?

BTW:
Settings.MoveMouseDelay = 5
I doubt that this has any effect on the dragging. The only possibility (I have to check in the implementation code) is that it inserts a longer first wait after mouseDown at the beginning.
So you might try instead:
    hover(ref.offset(Location(150, 390)))
    mouseDown(Button.LEFT)
    wait(0.5) # Settings.MoveMouseDelay = 5
    mouseMove(ref.offset(Location(335, 125)))
    mouseUp()

Many GUIs have a short timeout so that a mouseDown is only accepted as a drag start after this time has elapsed.

Revision history for this message
Test (c4456517) said :
#7

The wait didn't work for me.

The GUI is quite responsive and detects a click and drag quite readily.

I shall send an email to you within the next 3-4 hours detailing the different cases I tried out.

Revision history for this message
RaiMan (raimund-hocke) said :
#8

great, thanks.

Might help in the end to improve the API.