pl.read_csv can cause invalid UTF-8 strings to be generated about polars HOT 7 CLOSED

@nameexhaustion Sorry I just read ritchie's comment and closed based on that. We do need to ensure we only generate valid UTF-8.

from polars.

@pj-ml Perhaps, but that is not a P-high Priority: high bug. If the fix for this doesn't cover that I'd separate it into a separate bug/enhancement.

from polars.

I believe that's not a proper encoded csv file. Quotes must be escaped with quotes. Polars follows https://www.ietf.org/rfc/rfc4180.txt. Anything that doesn't follow that spec, we don't read. Though the error must be better.

from polars.

f.write('col1\n""• ') with a space appended to the end parses fine though

from polars.

I read https://www.ietf.org/rfc/rfc4180.txt and I see that if double quotes are used inside the entry, the entry must be encapsulated by double quotes too.

I think we can close this out then. I will just have to use use_pyarrow=True since I receive files that do not follow the standard used by polars.

from polars.

@orlp There could actually be an issue here - using read_csv polars will create a dataframe containing invalid Utf8 instead of immediately failing, see an example here:

import polars as pl

df = pl.read_csv('col1\n""•'.encode())  # this loads a dataframe without complaining

print(df.select(pl.all().cast(pl.Binary)))
# ┌─────────────┐
# │ col1        │
# │ ---         │
# │ binary      │
# ╞═════════════╡
# │ b"\xe2\x80" │
# └─────────────┘
print(df.select(pl.all().cast(pl.Binary).cast(pl.String)))  # casting it to binary and then back to run utf-8 validation to show that validation fails
# polars.exceptions.ComputeError: invalid utf8
print(df)  # printing will cause a panic due to invalid utf-8
# thread '<unnamed>' panicked at crates/polars-core/src/fmt.rs:393:21:
# byte index 34 is out of bounds of `�`

from polars.

And how about cases where it generates valid UTF-8, but the input file does not follow the standard? Shouldn't there at least be a warning in such a case?

See example (no errors and no warnings):

import polars as pl

df = pl.read_csv('col1\n""• '.encode())

print(df.select(pl.all().cast(pl.Binary).cast(pl.String)))
# shape: (1, 1)
# ┌──────┐
# │ col1 │
# │ ---  │
# │ str  │
# ╞══════╡
# │ •    │
# └──────┘

from polars.