Pandera timezone-agnostic datetime type about pandera HOT 7 OPEN

Hi @max-raphael this is somewhat of a challenging use case to fulfill with datetimes because if we have a timezone-agnostic datetime, how do we deal with coercion?

Imagine we support something like:

class MySchema(DataFrameModel):
    local_datetime: DateTime(has_tz=True)  # just checks that the datetimes have any timezone

    class Config:
        coerce = True

If we do coerce=True, what timezone should we coerce to? Solutions here would be:

• Default to UTC
• Raise an exception

from pandera.

This is similar to the problem of having a generic Number type: this can check if the data type is any of the int or float types, but when we coerce, what data type should it default to?

from pandera.

I hear you, that does pose a tricky problem. Thinking about it from my perspective as a user, I think I would prefer to have this as an option but be disallowed from coercing this field (via some Exception) due to the ambiguous nature of the data type rather than not have it accessible to me at all.

Perhaps even an Exception is too much. Pandera could still allow users to specify coerce=True and coerce other fields, and add a warning level log statement that informs the user that this field cannot be coerced due to its data type.

from pandera.

Perhaps even an Exception is too much. We could still allow users to specify coerce=True and coerce other fields, and add a warning level log statement that informs the user that this field cannot be coerced due to its data type.

How would you feel about defaulting to UTC on coercion (if the incoming raw data is not TZ-aware) and raising a warning that the dtypes are coerced to UTC? I generally like to do something rather than nothing on coercion to prevent propagation of surprise (i.e. a non-TZ aware dataframe after validation with coerce=True).

from pandera.

That seems acceptable to me. I think if incoming data is not tz-aware, then that's a reasonable approach so long as Pandera logs the warning and includes it in the documentation!

from pandera.

@cosmicBboy Hi, just following up here. Are we aligned on the feature? If so, what are the next steps? Thanks again for engaging with this, I think it would be helpful to many Pandera users.

from pandera.