How to get on the road as a newcomer to Java programming

Asked by Alexander Schone on 2018-03-20


I am new to coding in general and currently learning Java. I have a little project where which I'm using to learn and came across this piece of python code in one of the question here. I want to transfer this to java, I have tried finding code examples online but all I can find is very complex example far beyond my learning level or python examples. Furthermore the sikuli documentation I could find seems to be written for people who already know how to code.

r = regionOfTheScreen
images = [p0.png, p1.png, p2.png, etc]
while True:
    matches = a.findAnyList(images)
    for match in matches:

I want to do the same in Java but I am stuck. So far I have:

Region totalRollArea = new Region(20, 20, 100, 50);

  Match number1 = new Match("number1.png");
  Match number2 = new Match("number2.png");
  Match[] numbers = {number1, number2};
  totalRollArea.findAnyList("number1.png", "numbers.png"); // This part im quite certain of

  for (int i = 1; :i++) {
   if (totalRollArea.exist(numbers [1])); // ???

  return indexValue;

I am unsure about how to use the array (do I need it? the documentation for sikuli 1.1x+ doesnt mention it) and how to iterate over it using the List/Array.

Any help would be greatly appreciated as it helps me a lot in learning the language.

Question information

English Edit question
Sikuli Edit question
No assignee Edit question
Solved by:
Last query:
Last reply:
Best RaiMan (raimund-hocke) said : #1

ok, I admit, that it is not that easy to start with SikuliX as a newcomer to coding AND Java ;-)

The docs are primarily written for users of Python scripting and to understand the basics of the SikuliX features.

Just some basics looked from Java:
- a Region object is where you search something in (in your case: new Region(20, 20, 100, 50);)
- the search methods are basically find, wait and exists and more complex findAll, findAny ...
- the result (return value) of these features is a Match object or a list of Match objects

So if you want any search result it always is
Match match = someRegion.find(someImage); // or wait or exists
List<Match> matches = someRegion.findAnyList(imageList);
List<Object> imageList = ... an ArrayList of Strings (images) and/or Pattern objects

... and then for findAny
for (Match match : matches) {
    // do with match what is needed

To get on the road generally:
- use the latest version 1.1.2 (
- read across the docs, to get an idea of the features (
- to "translate" a scripting feature to Java there are some hints in the docs, but finally you have to look at the method definition either in the javadocs ( or even in the code ( This is usually available anyways, when you program with a mature IDE like Eclipse or IntelliJ IDEA.

Ah that explains my struggles so far.. I thought I was just a very slow learner but apparently I chose a tough entry :)

Thank you for the clarifications! with the help of a friend I was able to get to this stage:

import java.util.ArrayList;
import java.util.List;

import org.sikuli.basics.Debug;
import org.sikuli.script.*;

public class Script {

 public static void main(String[] args) throws FindFailed {
// Debug.setDebugLevel(3);
  Screen s = new Screen();

  // Declare region
  Region totalRollArea = new Region(20, 20, 100, 50);

  List<Object> imageList = new ArrayList();
  List<Match> numbers = totalRollArea.findAnyList(imageList);
  int indexValue = 0;

  for (Match match : numbers) {
   indexValue = match.getIndex();
    if (indexValue >= 1) {

This code works to find images "number1.png" and "number2.png" and print "true". For test I took windows icon and folder icon as number1 and number2 and used screen instead of region which of course led to long execution time.

The purpose of my script is to automate the abilities section of DnD based games like Icewind Dale (in my case) or Baldurs Gate. The game will roll a value which I want to compare to an internally set value (roll value should be >= 86). This roll value is composed of two numbers (ten digits and one digits) which I want to get from screen and then if value isnt sufficient press the "reroll" button and start the for loop again. Now the question: Do you think this part of the script can be executed quick enough to use lets say not more than 0.2 seconds to determine if the roll value goal is met (the area to check will be defined by a very small region so the actual find area will be only about 200 pixels)?
And do you see any room for improvement or more simplicity or better readability?

I hope I phrased this in a way that it makes sense.

Thanks RaiMan, that solved my question.

RaiMan (raimund-hocke) said : #4

- findAnyList()
... yep, goes into the right direction.

... but (if I understand my implementation well) a match in the returned matches list might be null.
 for (Match match : numbers) {
   if (match == null) continue;
   indexValue = match.getIndex();
   System.out.println(String.format("%s was found", imageList.get(indexValue));

Interesting situation - possible to have a live look at the game in the net?

Is it a 2-digit-number?
If yes, then you need a separate region for each digit I guess.

RaiMan (raimund-hocke) said : #5

--- quick enough to use lets say not more than 0.2 seconds to determine
should be possible on a capable machine.

Why could it return null? It will always find a number there.
a live look? i can post a screenshot of the screen i will be running the script on if that helps.
yes it is a w digit number and yes i will have two regions.

would it be possible to use .exists() ? like this in python:

def get_digits_in_region(region):
    foundNumbers = []
    for match in numbers:
        if region.exists(number[1], 0.001):
    return found_numbers

not sure what the def does here i suppose it would not be needed in java due to syntax differences. but if exists could be used then it would run faster i guess?

sorry foe dp but i have another question regarding findAny. What exactly is the difference between findAny and findAnyList?

RaiMan (raimund-hocke) said : #8

--- Why could it return null? It will always find a number there.
yes ... findAnyList() will always return a List object,
... but in your case all match entries in the list for the numbers not found will be null, only for the one found there will be a valid match entry.

--- difference between findAny and findAnyList
findAny accepts a parameter series of images, whereas findAnyList accepts only one parameter, which is a List of images (where images are either image filename strings or Pattern objects) (the javadocs mentioned in comment #1 might have told you ;-)

--- would it be possible to use .exists()
yes, of course: (Python script ;-)

numbers = (n0, n1, n2, ..., n9) # nX is a string having the number image filename
reg = <some region> # the search region
found = -1
for number in numbers:
    if not reg.exists(number, 0): continue
    else: found = found + 1
if found > -1:
    print "found number:", found
    print "no success"

this is a sequential search, which will in the average need 5 searches to identify the number (max 10 searches).

findAny() internally delegates each search into a separate thread, that run in parallel. My checks say, that beginning with 4 images to be searched in parallel, findAny() is faster than sequential exists()

I'll think about visiting the javadoc next time! I see so in my case I dont really need to use findAnyList correct? since I have the images as files already I dont need to create patterns?
How about findBest, would that not suit my needs better here since it would only return one number, or is it slower?

Hmm if findAny is faster at 5+ pictures and I have 10 pictures then I suppose I wont use it, especially if it cant be done in java :)
thank you for the example though!

RaiMan (raimund-hocke) said : #10

the exists example should break when number is found:

found = False
foundNumber = -1
for number in numbers:
    foundNumber += 1
    if not reg.exists(number, 0):
        found = True
if found:
    print "found number:", foundNumber
    print "no success"

RaiMan (raimund-hocke) said : #11

--- findBestList() vs. findAnyList()
yep, in your case findBestList() is the preferable feature

--- just made a test with 10 images in a 500 pixel region:
- exists loop: 90 msec
- findBest: 24 msec

so findBest is faster and the code is leaner

why do I need findBestList instead of findBest? Im not fully getting the difference even though I looked up both functions in the docs.

RaiMan (raimund-hocke) said : #13

--- why do I need findBestList instead of findBest?

same answer as this:
--- difference between findAny and findAnyList
findAny accepts a parameter series of images, whereas findAnyList accepts only one parameter, which is a List of images (where images are either image filename strings or Pattern objects)

which version you use depends on how you setup your image list:

listOfImages is a List<Object> object, that is filled elsewhere.

findbest(image1, image2, image3, image4, ...)
here the list of images is represented as a "static" parameter list

Yes, that is what I meant, I only need the "List" function when I expect the images(objects) to be created in the process of execution, but since I have all pictures already ill go without that feature. Thanks a lot! This thread has given me a good start with your help and clarifications :)

RaiMan (raimund-hocke) said : #15

always welcome - have fun with SikuliX

I have fiddled around with my little bot a bit more and Im almost done with the main function (rerolling). However I came across a problem: the number 3 is often identified as 8. I have tried using MinSimilarity but I dont think my script recognises this..

import org.sikuli.script.*;
import org.sikuli.basics.Debug;
import org.sikuli.basics.Settings;

public class TestAuto {

 public static double MinSimilarity = 0.99;
 public static float MoveMouseDelay = 0f;
 static Screen s = new Screen();

 public static void main(String[] args) throws FindFailed, InterruptedException {


  Match reRoll = s.find(new Pattern("Reroll.png"));
  Match totalRollLocator = s.find(new Pattern("TotalRoll.png"));
  Match strLocator = s.find(new Pattern("Strength.png").targetOffset(170, 0));
  Match intLocator = s.find(new Pattern("Intelligence.png").targetOffset(220, 0));
  Match wisLocator = s.find(new Pattern("Wisdom.png").targetOffset(220, 0));

  Region totalRollArea = new Region(totalRollLocator.getTopRight().getX(), totalRollLocator.getTopRight().getY(), 70, 50);
  Region tensDigitsArea = new Region(totalRollLocator.getTopRight().getX() + 30, totalRollLocator.getTopRight().getY(), 15, 12);
  Region onesDigitsArea = new Region(totalRollLocator.getTopRight().getX() + 42, totalRollLocator.getTopRight().getY(), 17, 12);

  int tensDigitsValue = 0;
  int onesDigitsValue = 0;
  int totalRollValue = 0;

  for (int i = 0; i < 50; i++) {
   // while (true) {
   Match tensDigitsRoll = tensDigitsArea.findBest("number0.png", "number1.png", "number2.png", "number3.png",
     "number4.png", "number5.png", "number6.png", "number7.png", "number8.png", "number9.png");
   tensDigitsValue = tensDigitsRoll.getIndex() * 10;

   Match onesDigitsRoll = onesDigitsArea.findBest("number0.png", "number1.png", "number2.png", "number3.png",
     "number4.png", "number5.png", "number6.png", "number7.png", "number8.png", "number9.png");
   onesDigitsValue = onesDigitsRoll.getIndex();
   totalRollValue = tensDigitsValue + onesDigitsValue;
   if (totalRollValue >= 85) {
   } else {;


how can I increase the similarity needed from the standard 0.7 to lets say 0.95?
the way it is now will the script "recognize" this line ?

public static double MinSimilarity = 0.99;

Many thanks in advance again!

RaiMan (raimund-hocke) said : #17

I made some changes - see //raiman ;-)

The Settings.MinSimilarity is "recognized" in the Finder.
a good start is 0.9 (as suggested).
With my code you get feedback about the scores, so you might adjust the setting as needed.

Debug.user instead of System.out.println has the advantage, that you get a timestamp for free and can use the String.format() systematic.

import org.sikuli.script.*;
import org.sikuli.basics.Debug;
import org.sikuli.basics.Settings;

public class TestAuto {

  public static double Settings.MinSimilarity = 0.9; //raiman
  public static float Settings.MoveMouseDelay = 0f; //raiman
  static Screen s = new Screen();

  static final String n0 = "number0.png"; //raiman
  static final String n1 = "number1.png"; //raiman
  static final String n2 = "number2.png"; //raiman
  static final String n3 = "number3.png"; //raiman
  static final String n4 = "number4.png"; //raiman
  static final String n5 = "number5.png"; //raiman
  static final String n6 = "number6.png"; //raiman
  static final String n7 = "number7.png"; //raiman
  static final String n8 = "number8.png"; //raiman
  static final String n9 = "number9.png"; //raiman

  public static void main(String[] args) throws FindFailed,
          InterruptedException {


    Match reRoll = s.find(new Pattern("Reroll.png"));
    Match totalRollLocator = s.find(new Pattern("TotalRoll.png"));
    Match strLocator = s.find(new Pattern("Strength.png").targetOffset(170, 0));
    Match intLocator = s.find(new Pattern("Intelligence.png").targetOffset(220, 0));
    Match wisLocator = s.find(new Pattern("Wisdom.png").targetOffset(220, 0));

    // raiman: use .x and .y instead of .getX() and .getY() - shorter ;-)
    Region totalRollArea = new Region(totalRollLocator.getTopRight().x, totalRollLocator.getTopRight().y, 70, 50);
    Region tensDigitsArea = new Region(totalRollLocator.getTopRight().x + 30, totalRollLocator.getTopRight().y, 15, 12);
    Region onesDigitsArea = new Region(totalRollLocator.getTopRight().x + 42, totalRollLocator.getTopRight().y, 17, 12);

    int tensDigitsValue = 0;
    int onesDigitsValue = 0;
    int totalRollValue = 0;

    for (int i = 0; i < 50; i++) {
      // while (true) {
      Match tensDigitsRoll = tensDigitsArea.findBest(n0, n1, n2, n3, n4, n5, n6, n7, n8, n9); //raiman
      tensDigitsValue = tensDigitsRoll.getIndex() * 10;
      Debug.user("tensDigit: %d score: %f", tensDigitsValue / 10, tensDigitsRoll.getScore()); //raiman
      Match onesDigitsRoll = onesDigitsArea.findBest(n0, n1, n2, n3, n4, n5, n6, n7, n8, n9); //raiman
      onesDigitsValue = onesDigitsRoll.getIndex(); //raiman: moved here
      Debug.user("onesDigit: %d score: %f", onesDigitsValue, onesDigitsRoll.getScore()); //raiman
        wait(0.2f); //raiman
      totalRollValue = tensDigitsValue + onesDigitsValue;
      if (totalRollValue >= 85) {
        Debug.user("rollValue: %d", totalRollValue); //raiman
      } else {;

Thank you for these suggestions!

I have tried applying this to my code but i get these markers on

  public static double Settings.MinSimilarity = 0.9; //raiman
  public static float Settings.MoveMouseDelay = 0f; //raiman

Multiple markers at this line
 - Duplicate field
 - Syntax error on token ".", ,

and for wait(0.2f); //raiman i get:

The method wait(long) in the type Object is not applicable for the arguments (float)

Am I doing something wrong here? these are the only things i changed.

RaiMan (raimund-hocke) said : #19

sorry, both my fault ;-)

these 2 lines
 public static double Settings.MinSimilarity = 0.9; //raiman
  public static float Settings.MoveMouseDelay = 0f; //raiman

have to be moved to the beginning of main() as
Settings.MinSimilarity = 0.9; //raiman
Settings.MoveMouseDelay = 0f; //raiman

and the wait has to be written as

Use the Debug.user() should help to solve your 3/8 problem.

Ah that works thank you :)
I have solved the 3/8 issue, it turned out that i just had insert some wait() after every reroll to give .findBest() time to detect the new value.
why would I use wait instead of thread.sleep though?

RaiMan (raimund-hocke) said : #21

--- why would I use wait instead of thread.sleep though?
shorter and clearer ;-)

wait(1.0d) instead of Thread.sleep(1000);

but the functionality is the same - just pauses script?
can I use it like this:

wait(0.025f) ?

RaiMan (raimund-hocke) said : #23

yes, just pauses the script.

It internally just uses Thread.sleep() as well, but catches a possible Interrupted exception with no action.
So it is just a shortcut.

I see.. another question:

as I understand it, sikuli creates objects for matches and patterns, in other words these objects are pictures right?

at what point will these go in and out of memory? and how big is the impact on the system performance?

Concrete example:

In my script the most used part is creating two matches (using the same method findBest on the same object) of 16x12 pixels. Lets say the loop runs 50.000 times. Will these taken pictures be stored in the memory until the script exits or will the be dumped each time the same match is created?

Im worried that if I run this script for a very long time, that it will clutter the memory but this concern might be totally unjustified as I dont really understand how it actually works.

RaiMan (raimund-hocke) said : #25

Java has an automatic garbage collection (GC) at the top controlled by object's reference counts: in the moment an object is no longer referenced (ref count 0), it will become a candidate for GC (but it is not predictable when in the future it will really purged from memory (see the Java limits)).

so if you have something like that:

while(true) {
    Match match = someRegion.find(new Pattern(someImg).similar(0.9))

end loop 1:
match references new object match1
anonymous object Pattern only lives until find() returns - hence GC'd (no outside reference)

end loop 2:
match 1 GC'd
match references new object match2
Pattern same as before

.... and so on

The role of images:
- a captured image (file name) is loaded to memory when first used and then cached for reuse
- a used region (someRegion here) holds the screenshot last used for search (size depends on region size)

So in your "simple case" you might run it for years ;-) and your machine should show about 300MB memory consumption for the Java process run from commandline. Running it from within the IDE it might be a bit more of course.

... and a screenshot of the whole screen costs about pixel width * height * 4 bytes ;-)

Feel free to use the respective tools in your IDE or the task-manager to watch.

Interesting, that means i wont have to deal with the script getting slower then,

I made some tests measuring the ms it takes to find images and i have made the observsation the time varies between 3 and up to 40 ms for the same looped findBest(10 images), small region 17x12 pixel.
I had it running for 50000+ iterations and there is no continuous increase or decrease, it jumps from 4 to 20 to 28 to 5 to 3 to 35 to 4 and so on, there is no pattern (that i could see at least). So my assumption is that these fluctuations are due to cpu? is there a way to allocate more resources to the program so that it performs the find operations even faster or more stable?
This is not really necessary but it would greatly improve the reroll speed for my bot :)

RaiMan (raimund-hocke) said : #27

- there is no continuous increase or decrease,
This is how the GC works: completely asynchronous and unpredictable within the given ranges. As I understand: it tries to work, when cpu usage allows it.

- time varies between 3 and up to 40 ms
no idea - I think you have to live with that (might be caused by my internal basic threading implementation)

You might dive into and implement your own parallel find, that tries to reuse resources as much as possible.

Generally: since your loop mainly is searching images, it is mostly number crunching leading to a very high cpu usage. If other processes on the machine consume CPU, this will influence your loop timing of course.

"You might dive into and implement your own parallel find, that tries to reuse resources as much as possible."

I suppose that is higher level java?
I would like to have these processes run in parallel if possible . Not really needed the script is very fast as of now already but why not perfect it. (the German in me speaks strong here ;))

RaiMan (raimund-hocke) said : #29

LOL, wir können gerne auch Deutsch reden ;-)

Also in der implementierten Lösung laufen die internen Suchen (10 in deinem Fall) schon parallel.

Das mit dem Initiativ-Vorschlag geht halt in die Richtung, dass es in Java 8+ schon Möglichkeiten gibt, die ganze Performance etwas stabiler zu machen gegenüber den Einflüssen der anderen Tasks auf dem Rechner. Für mich ist die gegenwärtige Lösung aber ok für den durchschnittlichen Anwendungsfall. In Version 2 werde ich da aber mit Sicherheit nochmal reinschauen.

Zusammenfassende Bewertung: ich schätze für Dich würde der zu treibende Aufwand in keinem Verhältnis zum Nutzen stehen - behaupte ich mal - rein so gefühlsmäßig ;-)

Haha, ja das koennten wir tun :)

Ich vermute mal, du hast Recht und der Aufwand das zu lernen und dann zu implementieren, lohnt sich fuer mich wohl nicht..

Version 2 ist schon geplant/in Bearbeitung? Wird die neue Version weitere features enthalten oder mehr ein Update oder beides?

da kommt mir noch eine weitere Frage:
In Bezug auf script Steuerung, wuerdest du empfehlen dies ueber Java zu implementieren, oder die Sikuli integrierte hotkey function? Konkret will ich das script pausieren, stoppen und wieder starten koennen.

Ich habe mir beides nur kurz angeguckt. Nachdem ich nun Blut geleckt habe, will ich ein komplettes Programm daraus schreiben ueber die naechsten Monate. Mit GUI, logging, einer database oder integrierte tables (je nach wie umfangreich es wird).

RaiMan (raimund-hocke) said : #31

-- Version 2: dauert noch > 6 Monate - also nicht drauf warten. wird aber abwärts kompatibel sein.

-- pausieren, stoppen, wieder starten von Abläufen (scripts): da bist du schon wieder mitten drin im Arbeiten mit Tasks: nur was du programmintern angestoßen hast, kannst du theoretisch später auch wieder programmatisch beeinflussen. Die Trigger dafür könnten zum Beispiel über HotKeys implementiert werden, haben aber mit den Wirkungen (pausieren, stoppen, ...) nichts zu tun. Diese Funktionalität musst du so implementieren, dass sie von außen steuerbar wird (im Groben sowas wie ein Debugger leistet)

--- Dein Projekt: Stichwort GUI: wenn du das Ganze hinter ein GUI bringen willst, von dem aus dann irgendwelche SikuliX Abläufe gesteuert werden, dann bist du natürlich in der Königsdisziplin angelangt.
Wenn die Benutzung von SikuliX dein Hauptanliegen ist, dann kommt eigentlich nur ein Kombi aus Java Swing oder gar JavaFX und einer Programmiersprache/Scriptsprache die "java-aware" ist in Frage (Java, Scala, Kotlin, Clojure, Jython, JRuby, ....) und Eclipse oder IntelliJ IDEA CE (benutze ich).
Logging und Datenbank/Tabelle sind da Kleinkram gegen ;-)

Hier probiert grade jemand:

Wünsch Dir auf jeden Fall alles Gute dabei und viel Spaß.
Always welcome

Ja, es soll eine GUI werden die dann die Sikuli funktionen steuert. Das wird ein Haufen Arbeit und viel Lernen, aber das ist ja der Sinn der ganzen Sache :) Es ist ein Lernprojekt.

Wo ist der vorteil der "zweiten" (Java aware) Methode?
Ich denke man lernt mehr wenn man die Koenigsdisziplin waehlt richtig?

Eclipse benutze ich jetzt schon, ueberlege auf IntelliJ umzusteigen weil wir das auch auf der Arbeit nutzen habe aber bisher noch keine wesentlichen Vorteile feststellen koennen (meine Einsicht ist natuerlich stark begrenzt).

RaiMan (raimund-hocke) said : #33

Wenn es ein Lernprojekt ist, dann auf jeden Fall erst mal komplett in Java starten.

Dann auch am besten direkt bzgl. des GUI mit JavaFX statt Java Swing anfangen.

Der Hinweis auf die "java-aware" Sprachen bezieht sich auf das Programmieren/Scripten von allem unterhalb des GUI Layers.
Mit all den genannten kann man viel sogenannten "BoilerPlateCode" vermeiden und viele Sachen einfach eleganter lösen.
Da kann man später aber immer noch ran gehen. Basic Java ist zum Lernen und Begreifen aber eh unumgänglich.

Eclipse vs. IntelliJ IDEA:
ist wirklich Geschmacksache. Wenn es auf der Arbeit genutzt wird ist das natürlich vorteilhaft bei der privaten Nutzung.
Für mich war das Entscheidende, dass die Unterstützung für Python/Jython und Ruby/JRuby bei IntelliJ einfach durchgängiger ist (Java IDEA CE inkl. Python Support, PyCharm für reine Python/Jython Projekte und RubyMine für Ruby/JRuby). Und die haben auch für die meisten kostenpflichtigen Sachen für OpenSourceProjekte freie Lizenzen auf Antrag.

Na dann: viel Erfolg und Spaß.

Ah, interessant, ja dann werde ich wohl auch umsteigen..

Vielen Dank fuer deine Hilfe und deine Arbeit an dem Projekt! Wenn ich wieder Fragen habe melde ich mich ;)
(wahrscheinlich eher frueher als spaeter, werde aber versuchen es auf sikuli fragen zu beschraenken)

RaiMan (raimund-hocke) said : #35

Always welcome ;-)

Fragen kannst du was du willst - wie ich antworte ist dann meine Entscheidung.

Die kürzeste Antwort wäre z.B.: Keine Ahnung ;-)

... aber ernsthaft:
Wenn Du Fragen zum wie und was rund um die Java Programmierung allgemein hast und das Netz Dich im Unklaren lässt:
benutze meine SikuliX Mail für direkten Kontakt: sikulix---at---outlook---dot---com

mein Slogan:
Ich habe keine Ahnung, aber immer eine Meinung.

Für Fragen mit SikuliX Bezug ist natürlich dieses Forum hier am besten, weil es dann viel mehr Menschen sehen können.
Ich sehe hier immer alles was vor sich geht.

... und für die Zukunft: neues Thema -> neue Frage oder neue Mail.