Unicode characters break script about gcalcli HOT 9 CLOSED

Fixed. Pull request sent.

The difference between print and sys.stdout.write(msg) is that, print does some extra work, such as appending an '\n' at the end, properly parsing unicode characters, transforming arguments to string, among others, I suggest to do some quick google search about this.

from gcalcli.

As @xmad mentioned the print function auto adds a newline and will auto add spaces as well. I HATE the python print system with a passion. sys.stdout.write gives you access to the raw stream and more control. Unfortunately some unicode stuff only works via print. :-( There is a problem with this change though. If the PrintMsg() function in gcalcli was called twice and the first call's string didn't have a '\n' in it then the adjacent printed string would be separated by a space. So lame. Fortunately gcalcli always builds whole lines (i.e. trailing '\n') so this issue isn't exposed.

Any python wizards have insight with unicode on the raw stdout stream?

I'll wait a bit before pulling this request in.

from gcalcli.

@msoida Does your terminal support UTF-8 characters? If you'r under unix type locale. You should have a bunch of names that end with with .UTF-8. Or paste this: echo "По оживлённым берегам" and see if it comes out right. I put a whole range of strange unicode characters in my google calendars and they come out just fine with gcalcli.

@insanum The usual way is to decode all input into unicode, work only with unicode inside the program, and encode it going back out. The most common encoding to use is utf8. http://www.darkcoding.net/software/unicode-in-python-2-decode-in-encode-out/

from gcalcli.

@grahamking Yes, unicode characters are displayed correctly. I guess without UTF-8 support gcalcli would break also when using print.

I found this today: http://stackoverflow.com/questions/1473577/writing-unicode-strings-via-sys-stdout-in-python

Adding sys.stdout = codecs.getwriter(sys.stdout.encoding)(sys.stdout) after imports solved problem for me :)

from gcalcli.

These unicode issues are a real pain. I thought all was working well based on input from others but maybe not. Can someone please post an example that doesn't work with the latest gcalcli code? Either expose a test calendar publicly or send me an ics/vcal file containing a problem event.

from gcalcli.

Hi, I'm not sure if this helps, but I've been getting this error when piping the output of gcalcli. It works fine if I simply run gcalcli in a terminal (uxrvt with utf-8 encoding) and have stdout display there, but when I pipe the output (to "more" for example), I have to set the PYTHONIOENCODING=utf-8 environment variable first, then it works fine (I guess python assumes ascii when piping).

from gcalcli.

FYI, there was a new "--locale" option added to gcalcli. I encourage you all to give it a try and let us know how it affects the ongoing unicode issue.

from gcalcli.

I think I may have fixed this. Was able to reproduce the behavior by using DTerm and also by piping output of gcalcli into a text file, more, or anything else really and having the Forecast calendar enabled (and later adding an event with Japanese characters):

DTerm gives:

Traceback (most recent call last):
  File "/opt/local/bin/gcalcli", line 1709, in <module>
    BowChickaWowWow()
  File "/opt/local/bin/gcalcli", line 1647, in BowChickaWowWow
    gcal.CalQuery(args[0])
  File "/opt/local/bin/gcalcli", line 1057, in CalQuery
    self._GraphEvents(cmd, start, count, eventList)
  File "/opt/local/bin/gcalcli", line 666, in _GraphEvents
    locale.getpreferredencoding())
LookupError: unknown encoding: 

+----------+----------+----------+----------+----------+----------+----------+
|Sunday    |Monday    |Tuesday   |Wednesday |Thursday  |Friday    |Saturday  |
+----------+----------+----------+----------+----------+----------+----------+
|27 Jan ** |28 Jan    |29 Jan    |30 Jan    |31 Jan    |01 Feb    |02 Feb    |

Piping gives:

gcalcli --nc calw | more
Traceback (most recent call last):
  File "/opt/local/bin/gcalcli", line 1709, in <module>

+----------+----------+----------+----------+----------+----------+----------+
|Sunday    |Monday    |Tuesday   |Wednesday |Thursday  |Friday    |Saturday  |
+----------+----------+----------+----------+----------+----------+----------+
|27 Jan ** |28 Jan    |29 Jan    |30 Jan    |31 Jan    |01 Feb    |02 Feb    |
|          |          |          |          |          |          |          |
|Forecast  |Forecast  |Forecast  |Forecast  |Somethi   |          |Groundhog |
|for 83709 |for 83709 |for 83709 |for 83709 |ng        |          |Day       |
    BowChickaWowWow()
  File "/opt/local/bin/gcalcli", line 1647, in BowChickaWowWow
    gcal.CalQuery(args[0])
  File "/opt/local/bin/gcalcli", line 1057, in CalQuery
    self._GraphEvents(cmd, start, count, eventList)
  File "/opt/local/bin/gcalcli", line 719, in _GraphEvents
    PrintMsg(CLR_NRM(), line + "\n")
  File "/opt/local/bin/gcalcli", line 246, in PrintMsg
    sys.stdout.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 4: ordinal not in range(128)

By changing from using unicode() to using str.decode() and str.encode() I've gotten everything plaything nice and so far haven't had any side effects. I did have to force a default of "UTF-8" due to Python returning an empty encoding when being used in a pipe, but after doing a little bit of research this seems to be the "normal" course of action.

from gcalcli.

I had the same issue with u'\xb0'

from gcalcli.