Automatically guess separators about polars HOT 4 OPEN

I don't know. Knowing your file type is a pretty reasonable ask I would say.

You also don't pass csv files to our parquet scanners.

from polars.

@orlp, if the reference is to data.table, then yes, it's typically to do with CSV, or specifically for "regular delimited files; i.e., where every row has the same number of columns" as the package calls it. Two excerpts from the documentation for fread that might help:

sep: The separator between columns. Defaults to the character in the set [,\t |;:] that separates the sample of rows into the most number of lines with the same number of fields. Use NULL or "" to specify no separator; i.e. each line a single character column like base::readLines does.

and how it does this guessing:

A sample of 10,000 rows is used for a very good estimate of column types. 100 contiguous rows are read from 100 equally spaced points throughout the file including the beginning, middle and the very end. This results in a better guess when a column changes type later in the file (e.g. blank at the beginning/only populated near the end, or 001 at the start but 0A0 later on). This very good type guess enables a single allocation of the correct type up front once for speed, memory efficiency and convenience of avoiding the need to set colClasses after an error.

from polars.

Can you elaborate a bit more, with motivating examples? Are you talking about CSV?

from polars.

and how it does this guessing:

A sample of 10,000 rows is used for a very good estimate of column types. 100 contiguous rows are read from 100 equally spaced points throughout the file including the beginning, middle and the very end. This results in a better guess when a column changes type later in the file (e.g. blank at the beginning/only populated near the end, or 001 at the start but 0A0 later on). This very good type guess enables a single allocation of the correct type up front once for speed, memory efficiency and convenience of avoiding the need to set colClasses after an error.

I think this quote only refers to its dtype guesser not it's separator guesser.

from polars.