Ability to control the assumed century cutoff with `Expr.str.to_date` when parsing 2-digit years about polars HOT 5 OPEN

If such a thing were added it should definitely be explicit rather than something stashed invisibly in the Config object 🤔 Got any examples of other APIs that allow this? (It makes sense, but I can't recall actually having seen it done elsewhere).

from polars.

First assumption was that it is not on polars but on chrono (rust date/datetime parser) but chrono seems to be "fine"

fn main() {
    use chrono::NaiveDate; // 0.4.38

    let Y_M_D = "%y-%m-%d";
    let D_M_Y = "%d-%m-%y";

    let Y_M_D_Text = ["49-01-01", "49-12-31", "50-01-01"];
    let D_M_Y_Text = ["01-01-49", "31-12-49", "01-01-50"];

    for date_str in Y_M_D_Text {
        println!("{:?}", NaiveDate::parse_from_str(date_str, Y_M_D));
    }

    for date_str in D_M_Y_Text {
        println!("{:?}", NaiveDate::parse_from_str(date_str, D_M_Y));
    }
}

// Ok(2049-01-01)
// Ok(2049-12-31)
// Ok(2050-01-01)
// Ok(2049-01-01)
// Ok(2049-12-31)
// Ok(2050-01-01)

Must be some polars optimized performance implementation that is not using chrono?! 😆

Anyway, as a side note, would strongly advise you to use a clear and unambiguous format whenever possible! 😉

from polars.

Must be some polars optimized performance implementation that is not using chrono?! 😆

there's a fast-path for some fixed-length-formats, it might be that

from polars.

python's docs for datetime suggest a different cutoff, but no way to override it:

When 2-digit years are parsed, they are converted according to the POSIX and ISO C standards: values 69–99 are mapped to 1969–1999, and values 0–68 are mapped to 2000–2068.

pandas seems to have the same cutoff, but again no way to override it. Perhaps with it being an extra 20 years out it's not an issue as frequently.

I suggest that even if we don't agree on implementing a way to control this in polars then we should at least make it explicit in the docs that we follow a different convention to python's datetime.

would strongly advise you to use a clear and unambiguous format whenever possible! 😉

Unfornately, I encountered this with data received from a third party - certainly not a fan of YY myself if I can avoid it.

from polars.

One possible solution, given that this is already being handled by polars, rather than delegated to chrono, is to add a new format specifier, perhaps %20y could indicate "2 digit year assumed to start with a 20".

This would be in line with the convention in chrono for allowing a number between the % and the letter to indicate a parameter, e.g. %3f to indicate a three-digit decimal fraction of a second.

from polars.