Intermittent failure of Sikuli - unpaired surrogate

Ok, I just learned something about site.py with another request bug, where the error seems to happen (a standard Python/Jython module, that is called on startup and analyses the sys.path and other setup situations).

I will work on this problem the next days and surely get an idea what might be the problem in your situation.

Until then you have to live with it ;-)

Is one of your Java versions Java 6?
If yes, then you should run the setup with this Java version (the fitness does not matter and is surely not the cause for the problem), but with Java 6 a different Jython version (2. is packed instead of 2.7 with Java 7+.

If this is not the case, then I have no idea what can help currently.

correct:
If yes, then you should run the setup with this Java version (the bitness does not matter and is surely not the cause for the problem), but with Java 6 a different Jython version (2.5) is packed instead of 2.7 with Java 7+.

I just hade a look into the modules site.py and os.py of Jython 2.7.

The problem comes up, when internally during Jython startup the Windows specific module nt.py is imported.
The mentioned unpaired surrogate error has something to do with the processing of some unicode items.

So I think the problem arises from your mixed Python/Jython 32-Bit/64-Bit environment. So it might well be, that in some situations through some whatever path settings the module loading gets mixed up.

I guess if you assure, that you always have a fitting setup of any path specs (system path, java class path, python path, python path, ...) the problem should vanish.

Thanks for the answers. I'm not aware of any strangeness with the paths, and it works 80% of the time...

I'm fairly sure I have Java 7 with both 64-bit and 32-bit.

If it's related to unicode, my setup is Swedish, using the UTF-8 encoding, if that's relevant.

But I'll check this again carefully when I'm next at work.

The problem seems to have gone away now. I think the most likely cause was that I had previously installed Sikuli 1.0.0 and hence there was some kind of clash between Jython 2.5 and Jython 2.7 files. I discovered a Sikuli 1.0.0 reference in my PATH and since removing that the problem hasn't occurred again.

ok, thanks for feedback.

Unfortunately, I still get this problem occasionally, so it seems my previous guess isn't correct. It doesn't seem to be as frequent but it still happens sometimes.

Since this problem can only happen on startup, you should instantly check your environment, when it happens.

You should run with option -d 3 set and at the beginning of your script say Debug.off().
This will give you some startup debug info about the environment as seen by SikuliX.

Setting -d 3 doesn't help: it's a Sikuli option and in this case Jython crashes before it gets as far as doing anything with Sikuli.

But I've printed the environment before starting it and I can't see anything strange. It's the same as the environment when it works...

It should also be said that I'm running a pure Python script from the command line, using the setup described at

http://doc.sikuli.org/faq/010-command-line.html

using the section marked "Run a script from Command line using the Sikuli contained Jython".

I notice now that your new version at

http://sikulix-2014.readthedocs.org/en/latest/faq/010-command-line.html

does not mention this possibility, is this no longer supported? Would be a shame if so, as it's very useful to be able to just develop .py scripts in an ordinary IDE and store the images elsewhere.

In any case, this might be affecting things.

Another weird thing is that I tried to work around this by examining stderr after Sikuli exited and restarting it if it said "unpaired surrogate", but then it always failed on subsequent attempts also. If I restart the whole thing it usually works again straight afterwards...

does not mention this possibility, is this no longer supported?

It is still possible, but not really recommended, since it does not work as one would expect things to work, when using a Jython prepared to be used from a command line.

With an approach like yours, I recommend to use some full featured IDE like Eclipse with PyDev or some others.
The sikulix.jar cannot be used in such IDE's as the running Jython, so you have to run the scripts from a command line and cannot use the debugging feature of these IDE's.

In parallel you can use the IDE to manage your images in different .sikuli containers, that can be imported into your scripts making the contained images available on the ImagePath.
With a suitable image naming convention, this is rather comfortable.

If you use the sikulix.jar in such contexts it surely comes to some clashes on class path and sys.path.
With your own Jython, the usage of the sikulixapi.jar should be used. If you use the sikulix.jar in such contexts it surely comes to some clashes on class path and sys.path.
see: http://www.sikulix.com/quickstart.html#qs8

OK, thanks. I changed to the suggestion setup (with sikulixapi.jar on my CLASSPATH) and am now running my own Jython. Unfortunately, I still get the same problem, i.e. this

Exception in thread "main" Traceback (most recent call last):
  File "C:\jython2.7b4\Lib\site.py", line 62, in <module>
    import os
  File "C:\jython2.7b4\Lib\os.py", line 59, in <module>
    from nt import *
ValueError: unpaired surrogate 0xde23 at code unit 890

about 1 time in 8.

I guess this may be a bug in Jython? Is it possible to run Sikulix under Jython 2.5, which is meant to be more stable?

Of course you can use Jython 2.5.

... but there is a new Jython 2.7-rc3 that says to have fixed some Windows problems (I bundle this now with version 1.1.0)

Jython 2.7-rc3 also has the problem.

But Jython 2.5.3 seems to work without issues. Thanks for the answers!

Thanks for feedback.

After checking again about this behaviour:
There MUST be some unicode characters be involved in your environment and startup sequence.

The problem comes up at the edge between UTF-8 (Jython level) and UTF-16 (Java level) and according to the comments in the net is a common problem with implementations, that have to handle both worlds.
According to the spec, the unicode points U+D800 to U+DFFF are invalid as single character (you always need 2 of them).
So it surely is some Jython bug, that cannot arise with Jython 2.5, since this does not handle unicode internally at all.

So this might be solved for you now, but I will turn it to a bug, to watch it further.