lakostis / mailman2sympa Goto Github PK

My mailman is not in UTF8

To get the ./scripts/mm2s_unpickle.py script to work and not get the error:

Traceback (most recent call last):
  File "./scripts/mm2s_unpickle.py", line 31, in <module>
    print(json.dumps(config_dict))
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 127: invalid continuation byte

import os
+import unicodedata

mm_path=os.getenv('MAILMAN_HOME', '/usr/share/mailman')

[...]

config_dict['bounce_info'] = str(config_dict.get('bounce_info'))
+for key in config_dict:
+    if type(config_dict[key]) == str:
+        config_dict[key] = unicodedata.normalize('NFKD',unicode(config_dict[key],"ISO-8859-1")).encode("ascii","ignore")
print(json.dumps(config_dict))
infile.close()

When trying to migrate lists with utf-8 characters anywhere in their configs, I get the a decode error from mm2s_unpickle:

Traceback (most recent call last):
File "./mm2s_unpickle.py", line 29, in
print(json.dumps(config_dict))
File "/usr/lib64/python2.7/json/init.py", line 244, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xdc in position 0: invalid continuation byte

I modified mm2s_unpickle to print the raw unpickled config before the json.dumps() call, and digging around in that output I found a user as follows:

'[email protected]': '\xdcz\xfcc\xfchemzem'

Digging around further I found hundreds of examples in this one list's config and user community of non-ascii characters that are turning up in that output. In their passwords and in some users' names. I have other lists where we have such characters in the descriptions and info. All of these fail to migrate in explosive ways (tens of thousands of lines of console output VERY VERY QUICKLY). While some of these lists we could work around the problem by changing one subscriber's name to ascii, migrating, and then changing back, that one list has 14,000 subscribers and literally hundreds of examples of this breaking the migration.

I tried adding a ensure_ascii=False to the json.dumps() call and it made no difference in mm2s_unpickle.py. It appears that json.dumps() in Python3 would just do the right thing, but that won't load the mailman.bouncer module.

And of course, mailman2 itself has no problems with these characters, it's just the migration tool.

I get an ImportError with scripts/mm2s_unpickle.py:

Traceback (most recent call last):
  File "./scripts/mm2s_unpickle.py", line 14, in <module>
    config_dict = pickle.load(infile)
  File "/usr/lib/python2.7/pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1096, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.7/pickle.py", line 1130, in find_class
    __import__(module)
ImportError: No module named Mailman.Bouncer

sys.path.append('/usr/lib/mailman')
import Mailman.Bouncer