SikuliX

Navigating to text using Sikuli in Eclipse

Asked by Boipelo Mawasha

I'm automating a program that runs in a terminal window. I would like to navigate to a specific account number on the screen and perform an action on it. I've managed to determine that the account number exists in the screen, however, I'm struggling to navigate to that specific account after I find it.

Finding the account:

textCapture = ms.getScreen().text();
query = "SELECT maccno FROM agreement";

con = Login.getConnection();
st = con.createStatement();
rs = st.executeQuery(query);
rs.next();

if (textCapture.contains(rs.getString(1))){
 System.out.println("Text found :)");
}else{
 System.out.println("Damn!");
}

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
Boipelo Mawasha
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said : 		#1

The only Sikuli statement I can see:
textCapture = ms.getScreen().text();

So i guess, that ms is a region object, that has been evaluated before somehow.

So already just
String textCapture = ms.text();

should do the job of extracting the text from ms using Sikuli's OCR feature.

But before going on with it, it should be checked by e.g.

System.out.println(textCapture)

to be sure, the OCR feature works as expected (see bug 710586).

*** you say: I'm struggling to navigate to that specific account after I find it.
It is not clear, what you mean by that.

Revision history for this message
Boipelo Mawasha (bmawasha) said : 		#2

The text that is read from the database is an account number that has to appear on the screen. The variable textCapture holds a string representation of the Screen object ms. The list is not ordered in a specific way, so the account number can appear anywhere in the list. I want to be able to navigate(using keyboard actions) to the specific account number and do something with it.

e.g.
rs.getString(1) = ACC007

Account number ------ Name
ACC004 User4
ACC002 User2
ACC007 User7
ACC003 User3

while (the cursor is not on the account number){
         type Key.Down
}

Once it's on the account: type Key.F2

Something along those lines

Revision history for this message
RaiMan (raimund-hocke) said : 		#3

send me a screenshot of the situation to my mail at https://launchpad.net/~raimund-hocke

Revision history for this message
Boipelo Mawasha (bmawasha) said : 		#4

Thanks RaiMan. Your solution worked perfectly. You were right about Sikuli's OCR not being reliable enough. It kept on returning "?" in place of some characters. I opted to just use the clipboard and compared the values straight from there.