UnicodeEncodeError: 'charmap' codec can't encode characters in position 31758-31761: character maps to <undefined> - when fetching the DBPedia_IT dataset about octis HOT 2 CLOSED

Hi,
thanks @ERijck for reporting and @Ravi2712 for your suggestion. It is indeed a problem of encoding. I'll fix this in the next release.

Silvia

from octis.

@ERijck I faced the similar error while using a function preprocessor.preprocess_dataset() that comes with OCTIS. I found out that my dataset has some unicode characters and emojis.

The error you are facing comes from line-number-(77) in downloader.py where OCTIS is trying to create relevant files of dataset. Link here:

A quickfix would be to read/write corpus that allows unicode characters and looks like following:

with open(corpus_path, 'w', encoding='utf8') as f:
            f.write(corpus.text)

Two quick modifications:

• You can try to fork the repository and change these lines until OCTIS provides Unicode support for dataset.
• You can edit the the files present in the environment (Not recommended).

NOTE:
If you endup doing this modification by yourself before OCTIS, you also need to change some appropriate functions which are reading the "Dataset" files before you start training model.

from octis.