Surprised by type inference: csvjoin removing underscores in columns containg values like: 1_100 about csvkit HOT 2 OPEN

We use Python's Decimal for parsing: https://github.com/wireservice/agate/blob/e6fc5beb65f444a84b02884e9c5e7b3e344599dd/agate/data_types/number.py#L95

For example:

>>> from decimal import Decimal
>>> Decimal('1_1')
Decimal('11')

Decimal works this way because, in Python, the underscore can be used as a grouping character:

>>> 1_000_000
1000000
>>> 1_1_1_1
1111

I'd be curious to know how tidyverse or pandas do this.

In terms of feedback to the user, we can hide it behind the -v (--verbose) flag, or we can figure out a way to keep the feedback minimal (e.g. only warn once, not every time).

from csvkit.

I'll check to confirm my memory that it doesn't occur in pandas/tidyverse. I'm not super familiar with their underlying code, but I can take a quick look to figure out how they're handling it.

I generally lean towards cli tools being as quiet as possible, so in that respect I'm leaning towards the verbose option.

However, I also value no surprises - the root cause of surprise was that csvkit was doing any datatype inference at all.
My mental model had it just treating everything as string, probably because while I use it a lot (thanks!), I don't do any real data analysis with it, just a bunch of joins, stacks, and greps. In that respect, a one-off warning might be better.

from csvkit.