[QST] Do CSV from users have different data types in the same column .. ? about cudf HOT 4 CLOSED

Thanks for the issue.

I've seen some mention

Could you link where this was mentioned? Generally read_csv will infer the data types of the columns of a CSV: https://docs.rapids.ai/api/cudf/stable/user_guide/api_docs/api/cudf.read_csv/#cudf-read-csv

from cudf.

Thanks for the issue.

I've seen some mention

Could you link where this was mentioned? Generally read_csv will infer the data types of the columns of a CSV: https://docs.rapids.ai/api/cudf/stable/user_guide/api_docs/api/cudf.read_csv/#cudf-read-csv

Seems like it was a misunderstanding/miscommunication from the person who mentioned this to me, I did not actually see this in the docs. It was cleared up a bit after.

With that being said though, I was mostly interested what libcudf would do if it receives a file like this

ID, Mixed_Data, Description
1, "John Doe", "Name"
2, 42, "Age"
3, "2024-01-11", "Date"
4, true, "Boolean Value"
5, 55000.75, "Salary"

Mixed_data column has different data types. Obviously i would hope users aren't using these kinds of CSV files. I'm curious to know what libcudf's priorities would be here, because there may exists potential optimizations in the CUDA kernels for the csv reader by transposing the rows to a columnar format, but this would only be doable if you don't delegate the burden on libcudf to deal with poorly formatted csv files like this.

Hopefully that makes sense, lmk if it doesn't.

from cudf.

A column holding a mixed typed values isn't supported in cuDF like in pandas, but for an IO situation like CSV reading, the data is coerced to a common type which would be string (object in cudf)

In [5]: data = """ID, Mixed_Data, Description
   ...: 1, "John Doe", "Name"
   ...: 2, 42, "Age"
   ...: 3, "2024-01-11", "Date"
   ...: 4, true, "Boolean Value"
   ...: 5, 55000.75, "Salary"
   ...: """

In [6]: import io

In [7]: cudf.read_csv(io.StringIO(data))
Out[7]: 
   ID     Mixed_Data       Description
0   1     "John Doe"            "Name"
1   2             42             "Age"
2   3   "2024-01-11"            "Date"
3   4           true   "Boolean Value"
4   5       55000.75          "Salary"

In [8]: cudf.read_csv(io.StringIO(data)).dtypes
Out[8]: 
ID               int64
 Mixed_Data     object
 Description    object
dtype: object

In [13]: cudf.read_csv(io.StringIO(data)).loc[1, " Mixed_Data"]
Out[13]: ' 42'

from cudf.

Thank you, that makes sense. I suspected as much but wanted to confirm just incase. Will close this question

from cudf.