2.1.2 python 2.7 UnicodeDecodeError when loading docstrings about pyvips HOT 12 CLOSED

Huh strange. I'll have a look.

from pyvips.

It seems to be working for me:

john@katamata:~/try$ ./test.py 
DEBUG:pyvips:Loaded binary module _libvips
DEBUG:pyvips:Inited libvips
DEBUG:pyvips:
john@katamata:~/try$ pip list
Package    Version
---------- -------
cffi       1.11.5 
pip        10.0.1 
pkgconfig  1.3.1  
pycparser  2.18   
pyvips     2.1.2  
setuptools 39.1.0 
wheel      0.31.0 
john@katamata:~/try$ which python
/usr/local/bin/python
john@katamata:~/try$ which pip
/usr/local/bin/pip
john@katamata:~/try$ vips --version
vips-8.6.3-Thu Mar  8 15:18:35 UTC 2018

from pyvips.

If I disable the binary module, it seems to fall back correctly too:

john@katamata:~/try$ mv /usr/local/lib/python2.7/site-packages/_libvips.so x
john@katamata:~/try$ ./test.py 
DEBUG:pyvips:Binary module load failed: No module named _libvips
DEBUG:pyvips:Falling back to ABI mode
DEBUG:pyvips:Loaded lib <cffi.api.FFILibrary_libvips.42.dylib object at 0x10fbc8110>
DEBUG:pyvips:Loaded lib <cffi.api.FFILibrary_libgobject-2.0.dylib object at 0x10fbc8090>
DEBUG:pyvips:Inited libvips
DEBUG:pyvips:

from pyvips.

My test program was just:

#!/usr/bin/env python

import logging
logging.basicConfig(level = logging.DEBUG)
import pyvips

if it makes any difference.

from pyvips.

It appears to depend on the locale:

• This fails: LANG=de_DE.UTF-8 python2 -c 'from pyvips import Image'
• This works: LANG=C python2 -c 'from pyvips import Image'

from pyvips.

I was just about to ask about your locale! Yes, I think you're getting a German translation in utf-8 and pyvips is not converting to ascii. I'll have a look at that.

from pyvips.

I think git master pyviups may have a fix, could you test it?

It was not decoding strings in py2, and .format in py2 seems to need a unicode literal.

jcupitt@ac33aa7

from pyvips.

It works on ac33aa7 but is broken again on b33ef40.

from pyvips.

Looks like there's a filename encoding issue on macOS as well.

from pyvips.

Looks like there's a filename encoding issue on macOS as well.

AFAIK HFS+ uses unicode decomposition using NFD, so characters like ü are stored as u ̈, I don't know if the same applies for APFS. It can certainly cause trouble when comparing filenames to utf8 strings that have not been decomposed. This can usually be solved by converting them to the same normalization form.

from pyvips.

This is strange, the following for me still breaks:

LANG=de_DE.UTF-8 python2 -c 'from pyvips import Image'

But this works fine on master:

LANG=de_DE.UTF-8 python2
>>> from pyvips import Image

The first example also works if I modify /usr/local/lib/python2.7/site-packages/sitecustomize.py and add sys.setdefaultencoding('UTF8') to change the default encoding for python2.7 from 'ascii' to 'utf8'.

from pyvips.

@jcupitt The following patch on top of master works for me and the tests are green on 2.7 and 3.6:

diff --git a/pyvips/vobject.py b/pyvips/vobject.py
index 5fa2109..e407267 100644
--- a/pyvips/vobject.py
+++ b/pyvips/vobject.py
@@ -6,7 +6,7 @@ import gc
 import logging
 
 import pyvips
-from pyvips import ffi, vips_lib, gobject_lib, Error, _to_bytes, _to_string
+from pyvips import ffi, vips_lib, gobject_lib, Error, _to_bytes
 
 logger = logging.getLogger(__name__)
 
@@ -71,7 +71,7 @@ class VipsObject(pyvips.GObject):
         """Get the blurb for a GObject property."""
 
         c_str = gobject_lib.g_param_spec_get_blurb(self._get_pspec(name))
-        return _to_string(ffi.string(c_str))
+        return _to_bytes(ffi.string(c_str)).decode('utf8')
 
     def get(self, name):
         """Get a GObject property.
@@ -131,7 +131,7 @@ class VipsObject(pyvips.GObject):
         """Get the description of a GObject."""
 
         vo = ffi.cast('VipsObject *', self.pointer)
-        return _to_string(ffi.string(vips_lib.vips_object_get_description(vo)))
+        return _to_bytes(ffi.string(vips_lib.vips_object_get_description(vo))).decode('utf8')
 
 
 __all__ = ['VipsObject']

This ensures that get_blurb and get_description return unicode strings.

from pyvips.