Unittest failure with py3.2: prettytable_test.PrintJapanestTest about prettytable HOT 6 CLOSED

Hi Sandro,

Thanks for reporting this, this is strange.  All I can think of to check off 
the top of my head is: what is the environment variable LANG set to inside of 
your chroot environment?

# echo $LANG
C

I think that is the problem.  I just tried running the test suite with LANG=C 
and I get the same error you do.  Setting LANG back to en_US.UTF-8 gets it to 
work like normal.  I guess the reason you can run it fine outside the chroot is 
that in your normal environment you too have LANG set to something with UTF-8 
in it.

Line 566, print(self.x), calls the PrettyTable object's __str__ method.  When 
running under Python 3.x, this returns a unicode string (i.e. the string is not 
encoded).  I don't really understand in detail how the Python print statement 
interacts with your console or terminal emulator to get that unicode string, 
but I suppose at some point it needs to be encoded to get passed from one place 
to another, and the value of LANG influences how that coding is done.  When 
LANG=C, something somewhere attempts to encode the string in ASCII, which fails 
when there is, e.g. Japanese text in the string, as in this case.  If LANG is 
set to something like en_US.UTF-8 (or I presume any locale ending with .UTF-8) 
then the attempted encoding is into UTF-8, which of course works just fine.

Note that in Python 2, __str__ *can* return a unicode string, but the more 
normal behaviour is for it to return an encoded byte string (while 
__unicode__() must return a unicode object).  In PrettyTable, __str__ under 
Python 2 returns an encoded string, and by default it is encoded using UTF-8 
(unless the user overrides this).  So this very same test works just fine under 
Python 2 even if LANG=C.

__str__ is certainly expected to return unicode in Python 3, so it seems to me 
that Python 3 applications are basically fundamentally incompatible with 
LANG=C, unless they are written specifically to overcome this.  In principle I 
suppose it would be possible to check the value of LANG at the time of calling 
__str__ and return an encoded string if LANG=C, but I am not sure if Python 3 
will object to __str__ returning a byte object, and I'm also not sure if that 
would be considered bad practice or not.

At any rate, I don't *think* this is a bug in PrettyTable per se.  The problem 
is that your chroot environment has what is probably by now considered an 
obsolete setting of LANG.  I don't remember exactly how the Debian installer 
goes with locales, but I suspect it probably has a UTF-8 option selected by 
default?  I don't think that this issue is likely to impact very many people 
using the package, because most of them should have a more unicode-friendly 
LANG setting in their day-to-day work environments.

Sandro, were you able to either
(i) get your chroot packaging environment using a non-C locale, or
(ii) verify that the package works correctly anyway when installed in a 
"proper" Debian system as set up by the installer?

Hi Luke,
it was indeed the lack of a UTF-8 charset in the building chroot; once I 
installed & instructed tests to use it, they pass correctly - hence I was just 
able to upload 0.7.2 to Debian!

Hi Sandro,

Wonderful, thanks for letting me know that you got this sorted.  I'll close the 
issue now.