How to detect moving images and images in 3D games

The challenges are:

1. moving images in 2D

2. images in 3D (in your case not real 3D, but some kind of perspective view mapped into 2D)

3. moving images in 3D

-- at 1: The implementation depends on wether you know the direction of movement or not.

If you know it, then you can wait in appropriate places, where the image should appear somewhen in the future.

If you do not know it, you have to watch for changes in direct surrounding of the actual place of the object. In most cases this will allow to calculate the direction of movement. Then you have to follow the movement again depending on, wether the object moves constantly in one direction until something happens or wether it might change the direction at any time without an obvious reason.

All this can be done with the standard features of SikuliX, but might end up in time consuming actions, that have to be optimised if possible, so the bot is always faster than the game.

at 2: The problem here is the perspective: size and shape of an image depend on where it is in the scene. The perspective view has some internal rules about size and distortion of an image at a specific point.

Image detection in all places of such a scenario is currently not possible, since you would have to adjust size and shape of the given image according to the rules of the view, even if you only wanted to look for the image in one exact place. Such features are currently not available in SikuliX (apart from a resize feature).

at 3: does not make sense to talk about that in the moment regarding SikuliX

... But SikuliX is an example of making features of OpenCV available at a higher level (image search and change detection). SikuliX uses the Java API of OpenCV, but this is also available for Python based environments.

... and OpenCV has everything, to address the mentioned scenarios.

So if you really want to move to scenarios 2 or even 3, you have to leave SikuliX behind and dive into Python+OpenCV (which is a bit easier than Java+OpenCV).
There are tons of examples in the net .

But feel free to use the SikuliX sources as a base and add your features based on OpenCV features not yet used in SikuliX.

The easiest way I have gone about navigating games with Sikuli is to:

1) Base my image searches on static images/words or ones that change slowly (giving time for sikuli to scan the screen)

2) If I can, setup and offset of those static images if I have to click an area with a mouse

3) Use hotkeys as much as possible (for selecting units or movement)

$) Use launch arguments if I can