Summarizing column (describe-sheet) is much slower (+300%) with typed date column about visidata HOT 6 OPEN

Great repro case for this report.

I narrowed down where to look. This line creates the slowness for dates. If I comment it out, describe-sheet finishes quickly.

from visidata.

Thank you for such a detailed report, and providing sample data!

This regression is not seen with other typed column (integer)

Do you mean that this behaviour has regressed since a previous version, or that you noticed a difference in behaviour between int and date?

from visidata.

Parsing dates is expensive, especially with python-dateutil. If you know the format, try using z@.

from visidata.

Thank you for such a detailed report, and providing sample data!

This regression is not seen with other typed column (integer)

Do you mean that this behaviour has regressed since a previous version, or that you noticed a difference in behaviour between int and date?

No, it's not a regression in 3.x per se: the problem was already there in 2.11

from visidata.

Parsing dates is expensive, especially with python-dateutil. If you know the format, try using z@.

Even when specifying a custom date format on that particular big file, the summarizing still takes 49 seconds
which is +122% compared to the 22 seconds (no typing)
but still better that without specifying (122 seconds)

$ time vd -b -p describe_cd.vdj 
saul.pw/VisiData v3.0.2
Support VisiData: https://github.com/sponsors/saulpw
opening describe_cd.vdj as vdj
opening people-2000000.csv as csv
set type of current column to custom date format
open Describe Sheet with descriptive statistics for all visible columns
replay complete

real    0m49,026s
user    0m48,227s
sys     0m0,809s

from visidata.

Yes, this makes sense. Again, parsing dates is expensive, even with strptime (which is what z@ uses). If you write a Python script that converts all elements in that column to date objects and then summarizes them, you should find that it takes about the same amount of time. If you want the values summarized as dates, the work has to be done somewhere!

from visidata.