SikuliX

Combo box scrolling

Asked by David Brown

I have a widget called a combo box. If I click it, I get a scrollable list of options. I want to potentialy scroll through the entire list to find my desired option, since only some of the available options are visable at any one time and I don't know where I will start at in the list. An answer to Question #205813 suggest checking to see when the image stops changing as a means of determining having reached either end of the scroll list. But I can't get that onChange to work right.

A selected item in the scroll list is highlighted blue. Some image comparisons mistakenly key off of that blue bar, which no longer works when the blue bar does not change relative position. The selected item also shows up unhighlighted at the top of the list, which is another area I've tried to do image comparisons on.

I can't figure out how to manualy compare a saved image of the list from prior to the last scroll to what the list currently looks like. At times my comparisons are saying nothing has changed too soon, and at other times my comparisons are not detecting the fact that nothing has changed. If I do an exists, I have to do so within a region, instead of a previosly saved image, and I can't throw a similar qualifier on it. If I use getScore, I either can't attach that to the previous exists, or the score does not reflect my desired match.

My first combo box is just a list of months. I should be able to force some kind of nearly exact match on the text of the month name, but I can't.

import shutil
import os

Settings.ActionLogs=True
Settings.InfoLogs=True
Settings.DebugLogs=True
#setFindFailedResponse(PROMPT)

click(Pattern("Month-1.png").similar(0.66).targetOffset(11,0))

direction = WHEEL_DOWN # direction of wheeling/scrolling
reachedBottom = False # flag to see if bottom of the list was reached
reachedTop = False # flag to see if top of the list was reached
entryFound = False # is set True if the image was found
maxScrolls = 20 # maximum number of scroll attempts

# the image that is searched in the list
#goalImage = "October.png"
goalImage = "January.png"

click(Pattern("Latitude.png").targetOffset(-54,0))
wait(3)
click(Pattern("Month-1.png").similar(0.66).targetOffset(11,0))

#smallRegion = getLastMatch().below(150)
reg = getLastMatch()
#smallRegion = Region(reg.x + 40, reg.y, reg.w-40, reg.h+140)
smallRegion = Region(reg.x + 42, reg.y, reg.w-42, reg.h)
lastImage = capture(smallRegion)
shutil.copy(lastImage, os.path.join("c:\DKB\Sikuli", "image_a.png"))
for i in range(maxScrolls): # avoid endless loop
    thisImage = capture(smallRegion)
    shutil.copy(thisImage, os.path.join("c:\DKB\Sikuli", "image%02d.png"%(i)))
    if (Region(smallRegion).exists(goalImage) == None): #no entry found #find fails & aborts
        print("did not find %d"%(i))
        if (direction == WHEEL_DOWN): # scroll down
            type(Key.DOWN)
        else:
            type(Key.UP)
        wait(4)

        #Region(getLastMatch().nearby(200)).highlight(2)
        #largeRegion = smallRegion.nearby(200).highlight(2)
        #largeRegion = smallRegion.nearby()
        #if (lastImage.exists(thisImage).similar(0.95)):
        #if (Region(getLastMatch().nearby()).exists(lastImage)):
        if (smallRegion.nearby(1).exists(lastImage)):
        #if (smallRegion.nearby().exists(lastImage) and getLastMatch().getScore() > 0.8):
            print("same as last time %f"%(getLastMatch().getScore()))
            #getLastMatch().highlight(2)
            #smallRegion.nearby().highlight(2)
            if (direction == WHEEL_DOWN): # bottom reached, change wheel direction
                reachedBottom = True
                direction = WHEEL_UP
                print("bottom")
            else: # top reached, change wheel direction
                reachedTop = True
                direction = WHEEL_DOWN
                print("top")
            if (reachedTop == True and reachedBottom == True): # both ends of the dropdown-menu were reached, no more scrolling needed.
                break
        else:
            print("not same as last time %f"%(getLastMatch().getScore()))
            #getLastMatch().highlight(2)
            #smallRegion.nearby().highlight(2)
            lastImage = thisImage
    else: # entry was found
        entryFound = True
        break

if (entryFound): # do stuff if successfull
    click(goalImage)
    print("found & clicked")

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
RaiMan
Solved:
Last query:
Last reply:
Revision history for this message
Brian Redmond (bredmond) said : 		#1

It's worth testing to see if the current selection can be copied to the clipboard. Some controls that you wouldn't expect would work do and it makes thing much easier because you can just have Sikuli read from the clipboard. This would get you to the last item in the list (assuming that the same value never appears twice in the list):

#After you've made the initial selection
lastSelection = ""
while True:
  type("c", KeyModifier.CTRL)
  currentSelection = Env.getClipboard()
  if currentSelection == lastSelection:
    break
  lastSelection = currentSelection
  type(Key.DOWN)
#Now you're at the last item

Hope this helps at some point if it doesn't happen to work with the box you're trying now :)

Revision history for this message
David Brown (cppege430dtvg7d94rok5h0us2f-david-9ei9nyjpwdexk1if796soz1u44j) said : 		#2

That is indeed clever, but it is not working in my situation. Neither using your code, nor interacting with my GUI manualy and doing ctrl c captures the text on the combo box. I.e. the clipboard contains something completely different.

No item ever does appear twice in the list. Since I may not start at the top of the list, I have to go down to the bottom, and then work my way back up to the top of the list. If I make it to both bottom and top, then I know I've searched the entire list. But it is tricky to get it to both turn the corner at the bottom, and also push past the top of the bottom half of the list.

Revision history for this message
Best RaiMan (raimund-hocke) said : 		#3

Oh yes, this is one of the most challenging things with Sikuli.

If you want to have a chance, to get it working visually (hence without "reading" the menu entries textually - seems to be the case her), the combo box (or any other menu like box, that can be went through using the direction keys DOWN/UP).

In your case I understand, that you have the following type of box:
- text field contains a current value from a list
- on click this list is displayed below the field
- the currently selected entry is highlighted and positioned
- the box does not show all entries at once

The goal:
select a specific entry based on the knowledge of the list content (number of entries and their sequence)

The approach:
position on the first entry and key down to the specific entry

The challenge:
make sure we position on the first entry in the list, no matter, where the current list position is (hence first entry might be hidden on list opening).

Slolution 1:
If the list supports a "jump to first entry" (e.g. PgUp), then this is the easiest way (some wait()'s between type()'s might be needed)

Solution 2:
If only UP and DOWN is supported, we first have to key UP until the highlight has reached the topmost position and then we have to key UP until the list content does not change anymore.

the steps:
1. evaluate somehow the area of the topmost list entry (first)

2. key DOWN 2 times to make sure the topmost entry is not selected

3. loop until the first box entry is selected

while True:
    img = capture(first)
    type(Key.Up)
    if not first.nearby(2).exists(img,0): break

now we are on the topmost entry, but not necessarily at the top of the list.

4. loop until positioned on the first entry of the list
we look wether the first entry has changed after a key UP, if not, we are at the top

while True:
    img = capture(first)
    type(Key.Up)
    if first.nearby(2).exists(img,0): break

the region first should have as little background as possible, to concentrate on the area having pixels that change (high similarity, should be above 0.9 or even better 0.95)

This can be evaluated rather easy:

r = selectRegion() # the target area
print r.nearby(10).find(capture(r))

together with the base position of the list, this helps to calculate the needed offsets/positions/dimensions

Revision history for this message
RaiMan (raimund-hocke) said : 		#4

Sorry, too early in the morning - need more coffee ;-)

The principal idea of solution 2 is ok, but the above implementation is nonsense.

Having a "good" region of the first box entry see comment on that above), the observation of the first entry should be sufficient, no matter if selected already or not.

so again:

1. evaluate somehow the area of the topmost list entry (first)

2. loop until positioned on the first entry of the list
we look wether the first entry has changed after a key UP, if not, we are at the top

while True:
    img = capture(first)
    type(Key.Up)
    if first.nearby(2).exists(img,0): break

Revision history for this message
David Brown (cppege430dtvg7d94rok5h0us2f-david-9ei9nyjpwdexk1if796soz1u44j) said : 		#5

It turns out that Key.HOME will take me to th top of the list, and Key.END will take me to the bottom of the list. Also, I could loop on Key.UP some large number and then go down the list, as suggested. It also would help if I know how many items are in the list so that I go down through the list without having to image compare test for the bottom of the list. In addition, I can type(image, "text") and have that work, which I was lead astray from when I was typing manualy, since if you don't type fast enough it starts keying off of the extra letters as new entries. (The A of Janunary would take me to April.)

So, I have 3 methods that solve the problem, but the attempt to cyle down then up through the list, though it should work, does not and has been abandoned. Thank you for your help.

Revision history for this message
David Brown (cppege430dtvg7d94rok5h0us2f-david-9ei9nyjpwdexk1if796soz1u44j) said : 		#6

Thanks RaiMan, that solved my question.