Python throws error while parsing on UTF 0xc3 and 0xc4 chars about planetjogger HOT 13 CLOSED

ERROR:planet.runner: File "/home/mt3o/src/venus/planet/spider.py", line 229, in writeCache
output = xdoc.toxml().encode('utf-8')

encode('utf-8', 'ignore') could probably help here.

from planetjogger.

:-(

ERROR:planet.runner:Error processing https://lottamowi.wordpress.com/feed/
ERROR:planet.runner:UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
ERROR:planet.runner: File "/home/mt3o/src/venus/planet/spider.py", line 480, in spiderPlanet
writeCache("uri", feed_info, data)
ERROR:planet.runner: File "/home/mt3o/src/venus/planet/spider.py", line 229, in writeCache
output = xdoc.toxml().encode('utf-8','ignore')
ERROR:planet.runner: File "/usr/lib/python2.7/xml/dom/minidom.py", line 46, in toxml
return self.toprettyxml("", "", encoding)
ERROR:planet.runner: File "/usr/lib/python2.7/xml/dom/minidom.py", line 61, in toprettyxml
return writer.getvalue()
ERROR:planet.runner: File "/usr/lib/python2.7/StringIO.py", line 271, in getvalue
self.buf += ''.join(self.buflist)

Is this related somehow?
http://stackoverflow.com/questions/3011939/reading-utf-8-xml-and-writing-it-to-a-file-with-python

from planetjogger.

Funny thing. I split problem line into:

partial = xdoc.toxml()
output = partial.encode('utf-8','xmlcharrefreplace')

and problematic line is xdoc.toxml() not the encode one...

ignore parameter seems to be unnecessary. I'll try to find out if we can convert the data that goes into xdoc.

ERROR:planet.runner:Error processing https://lottamowi.wordpress.com/feed/
ERROR:planet.runner:UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
ERROR:planet.runner: File "/home/mt3o/src/venus/planet/spider.py", line 484, in spiderPlanet
writeCache("uri", feed_info, data)
ERROR:planet.runner: File "/home/mt3o/src/venus/planet/spider.py", line 231, in writeCache
partial = xdoc.toxml()
ERROR:planet.runner: File "/usr/lib/python2.7/xml/dom/minidom.py", line 46, in toxml
return self.toprettyxml("", "", encoding)

from planetjogger.

Cannot reproduce.
INFO:planet.runner:Updating feed https://lottamowi.wordpress.com/feed/
What python and planet version do you use?

@mt3o As we talk about errors (tried to comment on your blog, but somehow comment didn't appear):
ERROR:planet.runner:Error 502 while updating feed http://blog.mt3o.it/feed/
Tried both HTTP and HTTPS version, without luck, and other WordPress blogs work well.

from planetjogger.

Cloned venus couple hours ago, installed ubuntu 16.04 on friday with its
default python version.
My blog gives no errors for me 😀

Comment was hidden because until I set everything up, they need to be
approved by me manually. I already pushed away four spam comments that went
thru akismet... But this is irrelevant to this problem :-)

Tomorrow i'll try to run venus on mydevil and see if anything changes.
On Apr 24, 2016 8:51 PM, "rozie" [email protected] wrote:

Cannot reproduce.
INFO:planet.runner:Updating feed https://lottamowi.wordpress.com/feed/
What python and planet version do you use?

@mt3o https://github.com/mt3o As we talk about errors (tried to comment
on your blog, but somehow comment didn't appear):
ERROR:planet.runner:Error 502 while updating feed
http://blog.mt3o.it/feed/
Tried both HTTP and HTTPS version, without luck, and other WordPress blogs
work well.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#10 (comment)

from planetjogger.

Well, version from Debian repository works fine on Jessie. Blame Ubuntu. ;-)

Regarding problems with feed - I managed to figure out the problem. Your host does have AAAA record, but feed isn't available there and returns 502. It happens from IPv6 enabled hosts. Fix your DNS (remove AAAA records) or your server.

from planetjogger.

God bless sending patches to the upstream...
On Apr 25, 2016 6:22 AM, "rozie" [email protected] wrote:

Well, version from Debian repository works fine on Jessie. Blame Ubuntu.
;-)

Regarding problems with feed - I managed to figure out the problem. Your
host does have AAAA record, but feed isn't available there and returns 502.
It happens from IPv6 enabled hosts. Fix your DNS (remove AAAA records) or
your server.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#10 (comment)

from planetjogger.

Upstream version seems to work fine too. At least on Debian unstable, at least grabbing feeds (but config wasn't tuned, so it's normal). Blame Ubuntu.

from planetjogger.

@rozie
i found out that cloudflare was enabling ipv6, disabled it. can you try again?

from planetjogger.

Works fine now. :-)

Regarding UTF - cannot reproduce. Works on Debian stable and planet venus from repo and Debian unstable planet venus from GH.

from planetjogger.

on mydevil it occurs :D

but i guess, it's irrelevant since you can run it without problems.

✓ [14:00:20] [email protected] :/dev/pts/92 +1 [3] 0:16658
~/venux $ python planet.py jogger.ini
INFO:planet.runner:Socket timeout set to 20 seconds
INFO:planet.runner:Building work queue
INFO:planet.runner:Updating feed uri @ http://koval.com.pl/feed/
INFO:planet.runner:Updating feed uri @ https://kronikaparanoika.wordpress.com/feed/
INFO:planet.runner:Updating feed uri @ https://lottamowi.wordpress.com/feed/
ERROR:planet.runner:Error processing https://lottamowi.wordpress.com/feed/
ERROR:planet.runner:UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
ERROR:planet.runner: File "/usr/home/mt3o/venux/planet/spider.py", line 484, in spiderPlanet
writeCache("uri", feed_info, data)
ERROR:planet.runner: File "/usr/home/mt3o/venux/planet/spider.py", line 231, in writeCache
partial = xdoc.toxml()
ERROR:planet.runner: File "/usr/local/lib/python2.7/xml/dom/minidom.py", line 46, in toxml
return self.toprettyxml("", "", encoding)
ERROR:planet.runner: File "/usr/local/lib/python2.7/xml/dom/minidom.py", line 61, in toprettyxml
return writer.getvalue()
ERROR:planet.runner: File "/usr/local/lib/python2.7/StringIO.py", line 271, in getvalue
self.buf += ''.join(self.buflist)
INFO:planet.runner:Updating feed uri @ http://www.michalszafranski.pl/feed
INFO:planet.runner:Updating feed uri @ http://blog.mt3o.it/feed/
INFO:planet.runner:Updating feed uri @ https://siwa.wordpress.com/feed/
ERROR:planet.runner:Error processing https://siwa.wordpress.com/feed/
ERROR:planet.runner:UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 3: ordinal not in range(128)
ERROR:planet.runner: File "/usr/home/mt3o/venux/planet/spider.py", line 484, in spiderPlanet
writeCache("uri", feed_info, data)
ERROR:planet.runner: File "/usr/home/mt3o/venux/planet/spider.py", line 231, in writeCache
partial = xdoc.toxml()
ERROR:planet.runner: File "/usr/local/lib/python2.7/xml/dom/minidom.py", line 46, in toxml
return self.toprettyxml("", "", encoding)
ERROR:planet.runner: File "/usr/local/lib/python2.7/xml/dom/minidom.py", line 61, in toprettyxml
return writer.getvalue()
ERROR:planet.runner: File "/usr/local/lib/python2.7/StringIO.py", line 271, in getvalue
self.buf += ''.join(self.buflist)
INFO:planet.runner:Loading cached data
Traceback (most recent call last):
File "planet.py", line 89, in
splice.apply(doc.toxml('utf-8'))
File "/usr/home/mt3o/venux/planet/splice.py", line 135, in apply
if not os.path.exists(output_dir): os.makedirs(output_dir)
File "/usr/local/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/www'

✗ [14:00:40] [email protected] :/dev/pts/92 +1 [4] 0:20825
~/venux $ python --version
Python 2.7.11

from planetjogger.

unstable:
python --version
Python 2.7.11+

stable:
python --version
Python 2.7.9

from planetjogger.

@mt3o Any progress on UTF thing? Anyway, cannot reproduce and it's not related to planet's configuration, so I'm gonna close this issue soon. But I'm still curious what is causing this problem.

from planetjogger.