PanicException: Python API call failed when using map_elements on a column about polars HOT 6 OPEN

Ok, I've find the final reason. This file is not a UTF-8 file, it is a "MS932", "CP932" or "Windows-31J", often called "Shift-JIS". You can see that standard utf8 decoding in python fails:

>>> b = b'\xe3\x82\xa8\xe3\x83\xad\xe4\xba\x8b\xe5\xb8\xab\xe3\x81\x9f\xe3\x81\xa1\xe3\x82\x88\xe3\x82\x8a \xe4\xba\xba\xe9\xa1\x9e\xe5\xad\xa6\xe5\x85\xa5\xe9\x96'
>>> b.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 37-38: unexpected end of data

But if we specify Shift-JIS it works:

>>> b.decode("Shift-JIS")
'繧ｨ繝ｭ莠句ｸｫ縺溘■繧医ｊ 莠ｺ鬘槫ｭｦ蜈･髢'

I'm not sure if it's intended to be unicode and simply has some bad encoding, or if it was mis-encoded, but the TL;DR is that polars doesn't support this encoding if it is indeed Shift-JIS, sorry!

from polars.

The relevant data can be found here. I am using the title.akas.tsv.gz data.

from polars.

As an FYI, this has nothing to do with unidecode. Performing just a title_akas_df.get_column("title").to_list() can reproduce the issue. Could you please create that as a minimal example instead?

from polars.

The issue occurs because of line 467,864. If we load into df then we can see:

>>> df = pl.read_csv(
...    "title.akas.tsv",
...     separator="\t",
...     dtypes={"title": str},
...     ignore_errors=True,
... )
>>> df["title"][467862]
'人類學入門'
>>> df["title"][467863]
pyo3_runtime.PanicException: Python API call failed

The issue is that the string cannot be rendered, and has nothing to do with map_elements or lists.

If we cast to binary, we get:

>>> df["title"].cast(pl.Binary)[467863]
b'\xe3\x82\xa8\xe3\x83\xad\xe4\xba\x8b\xe5\xb8\xab\xe3\x81\x9f\xe3\x81\xa1\xe3\x82\x88\xe3\x82\x8a \xe4\xba\xba\xe9\xa1\x9e\xe5\xad\xa6\xe5\x85\xa5\xe9\x96'

Perhaps someone with unicode experience can decipher this. If we try to grab just that one row with a selection, we panic and python is killed:

>>> s = df.select(pl.col("title").filter(pl.int_range(pl.len()) == 467863))
thread '<unnamed>' panicked at library/core/src/panicking.rs:155:5:
unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached
thread caused non-unwinding panic. aborting.
Aborted

from polars.

FYI the space in the byte array seen above is \x20. There is some badly formed unicode in here.

from polars.

Also, it's not your fault, the IMDB Page specifically cites Utf-8 as the encoding, but they are just wrong. I'd maybe contact them if you really need this dataset, or you can create a workaround by re-exporting the file with that one record deleted.

from polars.