UTF-8 encoded CSV does not work with import about pim-community-dev HOT 6 CLOSED

Hi @dirkjonker,

Thanks for reporting this.

Is there a way to test your file ? On our side, we work only with files in UTF-8 and we have already did some import in Japanese.
It was on Beta1, so it could be a regression on beta2.

Can you provide as well the content of the app/logs/batch_execute.log as well as prod.log from the same directory ?

Thanks in advance !

from pim-community-dev.

Hello @BitOne how do I send you these files?

from pim-community-dev.

@dirkjonker I cannot see any file here. Did you upload them ?

from pim-community-dev.

@BitOne I cannot seem to upload anything here. Where can I send the files?

from pim-community-dev.

@dirkjonker , can you send the files to [email protected] ?

Thanks in advance !

from pim-community-dev.

Ok, @dirkjonker, I got your files.

There's several problems with the one encoded in UTF-8:

• your text editor has added a BOM at the beginning of the file (see http://en.wikipedia.org/wiki/Byte_order_mark). It's used to get the right byte order for Unicode encoding different than UTF-8. It's not recommended for UTF-8 and it breaks the import.
• the string "Zwart/cr...me" is encoded with a 3 bytes sequence EF BF BD. This is the sequence for the U+FFFD character in UTF-8 which is the replacement character (http://en.wikipedia.org/wiki/Specials_%28Unicode_block%29). It means that your editor was not able to find a the correct UTF-8 character for the character you provided.

The one encoded in "ANSI" does not have the BOM character at the beginning, so I think that's reason why you managed to import it. But the content of the file has the same problem with the broken character.

So clearly, your file was broken before you converted them to UTF-8. In fact it seem it has been converted several times to UTF-8, as your "ANSI" file used a UTF-8 encoded character.

You should try to get a good source file and convert it to UTF-8 only if you're sure it's not already UTF-8.

from pim-community-dev.