want to work with BufferedImage instead of the Screen's internal capture with find/findAll

Asked by David

Hi RaiMan (and others) !

First of all, let me say you that Sikuli is a beautiful work and a very usable library. I played with some years ago and enjoyed it. Now I decided to base a bot project on it and my choice was obvious... ;)

I have one question :

I have to extract all matches founds in a specified regions and then work again to extract larger images from the specified matches coordinates on the screen.

The problem is that my screen is very dynamic and that's why for performance I'm looking for the minimum pattern size.
But when I call Region.findAll(), I get match regions relatives to the full screen.
I didn't found a way to get the source bufferedImage. I have to be sure to work on the same image used to search because after that I extract my datas from the matches. Here is my piece of code :

  try {
   image = Screen.getPrimaryScreen().capture().getImage();
   matchs = searchArea.findAll(lookingFor);
  } catch (FindFailed e) {
   return;
  }

  while (matchs.hasNext() && (cards.size() < 52)) {
   Match m = matchs.next();
   BufferedImage card = image.getSubimage(m.getX(), m.getY(),
     extractW, extractH);

   CardsElector elector = new CardsElector(card);
   elector.start();
  }

I extract subImages from the screen BufferedImage, relative to the "matching" Regions. But in my case, if there is a screen change between the screenCapture and the findAll, I'll get bad datas.
So I would like to get the same BufferedImage the Finder worked on when I called the findAll() method.

Could you tell me please how to do that ?
Maybe it could be usefull for peoples like me to add the possibility to work directly on BufferedImages and apply finAll(Pattern), find(Pattern) contains(Pattern). Like that, we could get the LastScreenImage and work directly on it. Because when we actually call the findAll method, we don't know on which image reference it's done.

Thanks for your help and your work !

Best regards,
David

PS : I'm waiting for the SikuliX 2014 because I got some troubles with observers. So I did my own RegionObserver that uses the find() method to detect changes and invoke onFind(), onNotFind(), onAppear(), onVanish() home made FindObserverEvent implementations.

Question information

Language:
English Edit question
Status:
Solved
For:
SikuliX Edit question
Assignee:
No assignee Edit question
Solved by:
David
Solved:
Last query:
Last reply:
Revision history for this message
RaiMan (raimund-hocke) said :
#1

since I do not know your exact scenario and overall approach, I made an example that works with version 1.0.1

It works on this question site on Mac in Safari and was setup and run as normal Java app in Netbeans.
It shows an implementation allowing to sort the matches returned by findAll topdown.
The images are in the project folder in images.sikuli, which in parallel is opened in the Sikuli IDE to capture needed images.

This example shows, how to use findAll() on a region and on the ScreenImage of the same region, both yielding the same result.

With version 1.1.0 there will be even more convenience features to support such a scenario.

the stuff zipped can be downloaded: https://dl.dropboxusercontent.com/u/42895525/Test.zip

package test;

import java.io.File;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Date;
import java.util.Iterator;
import java.util.List;
//a ref to sikuli-java.jar is in the projects class path
import org.sikuli.script.*;
import org.sikuli.basics.*;

public class Test {

  private static List<Found> matches = new ArrayList<Found>();

  public static void main(String[] args) throws FindFailed {
    // set the imagepath
    ImageLocator.setBundlePath(new File(System.getProperty("user.dir"), "images.sikuli").getAbsolutePath());
    p("ImagePath %s", ImageLocator.getBundlePath());

    Screen s = new Screen();

    // the site as in images.sikuli/ScreenShot.png must be visible
    App.focus("Safari");
    Region win = App.focusedWindow().grow(-20);

    // to eat up primary Sikuli initialization
    s.find("person.png");

    int debugLight = 0; // set to 0 to switch off highlighting

    // restrict to a region that makes sense
    Region reg = win.find("head.png").below().intersection(win).highlight(debugLight);

    long start = (new Date()).getTime();
    // we look on the screen
    Iterator<Match> persons = reg.findAll("person.png");

    // copy the matches to a more flexible list
    while (persons.hasNext()) {
      matches.add(new Found(persons.next()));
    }
    // sort the matches top down
    Collections.sort(matches);

    p("ScreenMatches %d", matches.size());
    for (Found m : matches) {
      // show the matches
      m.getMatch().highlight(debugLight);
    }
    p("elapsed: %d", (new Date()).getTime() - start);
    matches.clear();

    start = (new Date()).getTime();
    // we look in the image taken from reg
    Finder fndr = new Finder(s.capture(reg));
    fndr.findAll("person.png");
    while (fndr.hasNext()) {
      matches.add(new Found(fndr.next(), reg));
    }
    Collections.sort(matches);

    p("ImageMatches %d", matches.size());
    for (Found m : matches) {
      m.getMatch().highlight(debugLight);
    }
    p("elapsed: %d", (new Date()).getTime() - start);

    // to get back to the dev IDE at normal and
    App.focus("NetBeans 7.3.1.app");
  }

  private static void p(String m, Object... args) {
    System.out.println(String.format("[user] " + m, args));
  }
}

class Found implements Comparable<Found> {
  Match match;

  public Found() {
    match = null;
  }

  public Found(Match m) {
    match = m;
  }

  public Found(Match m, Region r) {
    match = m;
    m.x += r.x;
    m.y += r.y;
  }

  public Match getMatch() {
    return match;
  }

  @Override
  public int compareTo(Found f) {
    Match m1 = getMatch();
    Match m2 = f.getMatch();
    if (m1.y < m2.y) {
      return -1;
    }
    if (m1.y > m2.y) {
      return 1;
    }
    return 0;
  }
}

Revision history for this message
RaiMan (raimund-hocke) said :
#2

A Finder() can be setup as well with a BufferedImage.

Revision history for this message
David (borratdavid) said :
#3

Thanks for your answer.

A Finder() can be setup as well with a BufferedImage >> Of course, I'm dummy !

It could be the perfect way for me to search a pattern, but the Finder.finAll(Pattern p) method return String !?
I would like to get an Iterable of Match or Region objects... Could you help ?

Revision history for this message
RaiMan (raimund-hocke) said :
#4

exactly this is in the example:

    // we look in the image taken from reg
    Finder fndr = new Finder(s.capture(reg));
    fndr.findAll("person.png");
    while (fndr.hasNext()) {
      matches.add(new Found(fndr.next(), reg));
    }
    Collections.sort(matches);

    p("ImageMatches %d", matches.size());
    for (Found m : matches) {
      m.getMatch().highlight(debugLight);
    }

instead of an Iterator<Match> you ask the Finder object after findAll using hasNext and next.

Revision history for this message
David (borratdavid) said :
#5

Thanks, it's perfect for my usage :))