`Expr.meta` function that computes the dtype of an expression about polars HOT 3 OPEN

Came across this after looking for info on type hints in Polars (via the issue you referenced). Off the top of my head, I don't know that there wouldn't be cases where the type resulting from certain operations would be unknown if the argument types are known (apart from maybe some uses of map_elements or operations on Object types, but haven't thought very far on this).

I think your enhancement idea is a worthwhile endeavor (though I admit I am not very familiar with the Rust code base myself). It did however get me thinking more about typing in Polars and has made me wonder: why are all expressions the same "type" anyway (i.e., Expr)?

It seems it would actually be useful to have multiple subclasses of Expr that are specific to each of the native types (e.g., something like StringExpr, Float64Expr, and so forth). This way, a chain of methods like:

pl.col('some_floats').cast(pl.String) would have a type that explicitly describes the types of its elements (e.g., StringExpr) which would allow direct introspection without the need for a meta accessor. What's more, you wouldn't be permitted to do something like pl.col('some_floats').cast(pl.String).round(2) because no such method would exist, thereby minimizing run-time errors maybe.

And of course, this would be a major improvement for type hinting with functions such as this:

def foo(a: pl.Expr, b: pl.Expr) -> pl.Expr:
    return pl.when(a.round(0) == 4, b.str.strip_chars()).then(200).otherwise(900)

Where it would be helpful to have a signature more like:

def foo(a: pl.Float64Expr, b: pl.StringExpr) -> pl.Int32Expr:

It may be that I am missing some critical piece that would show why this cannot be done, but from what I can tell, it seems like something that could help to clarify types (which seems to fit well with Polars' already excellent treatment of types and type consistency) and I would be interested in seeing something that at least takes things in this direction. Curious if you have any thoughts.

from polars.

Came across this after looking for info on type hints in Polars (via the issue you referenced). Off the top of my head, I don't know that there wouldn't be cases where the type resulting from certain operations would be unknown if the argument types are known (apart from maybe some uses of map_elements or operations on Object types, but haven't thought very far on this).

I think your enhancement idea is a worthwhile endeavor (though I admit I am not very familiar with the Rust code base myself). It did however get me thinking more about typing in Polars and has made me wonder: why are all expressions the same "type" anyway (i.e., Expr)?

It seems it would actually be useful to have multiple subclasses of Expr that are specific to each of the native types (e.g., something like StringExpr, Float64Expr, and so forth). This way, a chain of methods like:

pl.col('some_floats').cast(pl.String) would have a type that explicitly describes the types of its elements (e.g., StringExpr) which would allow direct introspection without the need for a meta accessor. What's more, you wouldn't be permitted to do something like pl.col('some_floats').cast(pl.String).round(2) because no such method would exist, thereby minimizing run-time errors maybe.

And of course, this would be a major improvement for type hinting with functions such as this:

def foo(a: pl.Expr, b: pl.Expr) -> pl.Expr:
    return pl.when(a.round(0) == 4, b.str.strip_chars()).then(200).otherwise(900)

Where it would be helpful to have a signature more like:

def foo(a: pl.Float64Expr, b: pl.StringExpr) -> pl.StringExpr:

It may be that I am missing some critical piece that would show why this cannot be done, but from what I can tell, it seems like something that could help to clarify types (which seems to fit well with Polars' already excellent treatment of types and type consistency) and I would be interested in seeing something that at least takes things in this direction. Curious if you have any thoughts.

Perhaps this is best raised in a separate issue? Though type hinting is a very important tool and this kind of feature would be great, having runtime access to column dtypes unlocks new capabilities that are otherwise impossible/impractical with the current Polars API.

from polars.

Thanks @butterlyn. I decided to write this out in more detail in #18082 and looking to get some feedback.

from polars.