Comments (10)
In Polars (conceptually speaking) pl.lit(0)
is a scalar expression, just like pl.col.x.sum()
is. Scalar expressions are returned as a new row by themselves if all columns are scalar, and are broadcasted to the other columns' lengths otherwise.
So in these two examples pl.lit(0)
and pl.col.x.sum()
are returned as a new row:
>>> df = pl.DataFrame({"x": []}, schema={"x": pl.Int32})
>>> df.select(pl.lit(0))
shape: (1, 1)
┌─────────┐
│ literal │
│ --- │
│ i32 │
╞═════════╡
│ 0 │
└─────────┘
>>> df.select(s=pl.col.x.sum())
shape: (1, 1)
┌─────┐
│ s │
│ --- │
│ i32 │
╞═════╡
│ 0 │
└─────┘
In these four examples the scalar values are broadcasted along with the other column, x
, which in this case means the result has zero rows when x
is empty:
>>> df = pl.DataFrame({"x": [1, 2]}, schema={"x": pl.Int32})
>>> df.select(pl.col.x, pl.lit(0))
shape: (2, 2)
┌─────┬─────────┐
│ x ┆ literal │
│ --- ┆ --- │
│ i32 ┆ i32 │
╞═════╪═════════╡
│ 1 ┆ 0 │
│ 2 ┆ 0 │
└─────┴─────────┘
>>> df.select(pl.col.x, s=pl.col.x.sum())
shape: (2, 2)
┌─────┬─────┐
│ x ┆ s │
│ --- ┆ --- │
│ i32 ┆ i32 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 2 ┆ 3 │
└─────┴─────┘
>>> df = pl.DataFrame({"x": []}, schema={"x": pl.Int32})
>>> df.select(pl.col.x, pl.lit(0))
shape: (0, 2)
┌─────┬─────────┐
│ x ┆ literal │
│ --- ┆ --- │
│ i32 ┆ i32 │
╞═════╪═════════╡
└─────┴─────────┘
>>> df.select(pl.col.x, s=pl.col.x.sum())
shape: (0, 2)
┌─────┬─────┐
│ x ┆ s │
│ --- ┆ --- │
│ i32 ┆ i32 │
╞═════╪═════╡
└─────┴─────┘
This is why Polars has different behavior when you use with_columns
†.
As for the original issue, I don't know exactly how to match SQL's behavior here without a lot of work on our end...
† I'm still not 100% sure what we should do in the with_columns
case if we end up overwriting every original column in the with_columns
expression. You could imagine that there's always a hidden null column for with_columns
to broadcast along, which is what the current engine does:
>>> df = pl.DataFrame({"x": []}, schema={"x": pl.Int32})
>>> df.with_columns(x=0)
shape: (0, 1)
┌─────┐
│ x │
│ --- │
│ i32 │
╞═════╡
└─────┘
This might however be problematic for the new streaming engine, so the jury isn't out on this one yet.
from polars.
Also, is it possible to create expression that means "0 distributed for each row even if nothing else is selected"?
Kind of. You'll have to select a column to broadcast with, then drop that column after the broadcast:
>>> df = pl.DataFrame({"x": [1, 2, 3]})
>>> df.select(pl.first(), pl.lit(0)).drop(pl.first())
shape: (3, 1)
┌─────────┐
│ literal │
│ --- │
│ i32 │
╞═════════╡
│ 0 │
│ 0 │
│ 0 │
└─────────┘
But I don't believe you can literally do it as a stand-alone expression.
EDIT: you can actually make such an expression by broadcasting by abusing pl.when
:
>>> df.select(pl.when(True).then(0).otherwise(pl.first()))
shape: (3, 1)
┌─────────┐
│ literal │
│ --- │
│ i64 │
╞═════════╡
│ 0 │
│ 0 │
│ 0 │
└─────────┘
Still relies on broadcasting with an existing column but that column never has to show up in your result.
from polars.
The same applies to python API
I expected L.select(pl.lit(0))
to return 2 lines, not 1 as it's returned.
But that's harder to compare with other DBMSes
from polars.
@ritchie46 @orlp can a mod delete/ban the malware link above? Edit: I've reported it as malware abuse, not sure how quickly they act.
@AlexeyDmitriev for the python API, 2 lines is intended I believe. See the discussion in #17107.
from polars.
Actually I saw that issue, but didn't quite get what is expected and what is not
from polars.
So--the below is in reference to with_columns
, which apparently has different behavior than select
.
pl.lit
does not have super obvious intended behavior, but in essence it is supposed to mean "a literal value that fills the height of the columns in the DataFrame." This has the confusion that if no columns are present, a height of 1 is returned, but it is not inconsistent.
In your example, the frame has height 2, so selecting a literal value be also length 2. In the example in the linked issue, an empty frame has height 0, so selecting a literal from there will be length zero. This is consistent with the definition:
import polar as pl
# In an empty frame, literal returns length 1
pl.DataFrame().with_columns(pl.lit(1))
shape: (1, 1)
# ┌─────────┐
# │ literal │
# │ --- │
# │ i32 │
# ╞═════════╡
# │ 1 │
# └─────────┘
# As long as we have any columns, literal returns that length
# In this case, that length is 0.
pl.DataFrame({"a": []}).with_columns(pl.lit(1))
shape: (0, 2)
# ┌──────┬─────────┐
# │ a ┆ literal │
# │ --- ┆ --- │
# │ null ┆ i32 │
# ╞══════╪═════════╡
# └──────┴─────────┘
Now, with select
, we do indeed get a literal of length 1, even if our columns are length 0:
pl.DataFrame({"a": []}).select(pl.lit(1))
shape: (1, 1)
┌─────────┐
│ literal │
│ --- │
│ i32 │
╞═════════╡
│ 1 │
└─────────┘
I am not sure what to think about that.
from polars.
Thanks for the clarification @orlp, makes perfect sense.
from polars.
@orlp thanks for explanation, I can see now that this behaviour make sense (in particular you allow selecting aggregates and non-aggregates with simpler syntax then in classic sql as in your df.select(pl.col.x, s=pl.col.x.sum())
example)
Personally, I'd find more intuitive if there were to separate functions to select something for each line of the df (accept both, always broadcasted) and select an expression (accepts only scalars, doesn't)
- This way it's easier to understand the shape of what is returned when you read the code (even if you don't know the specific things user passed)
def f(expr):
...
some_df.select(expr) # you know this is always the same shape as some_df, even if user passed literal
...
- You can always remove unnecessary column safely, e.g.
df.select(x=x, y=y).select('y')
is the same asdf.select(y=y)
But I guess, that wouldn't work with the semantics of functions like Expr.top_k
(because it's neither of my options)
And also, that would be completely changing the API people got used to
Anyway, it would be good if this was explained in some obvious place in the docs, because when I tried to see explanation I failed.
Also, is it possible to create expression that means "0 distributed for each row even if nothing else is selected"?
As for the original issue, I don't know exactly how to match SQL's behavior here without a lot of work on our end...
For me personally I only care for python API and showed sql behaviour only because with sql there's a system to compare to
BTW the fact that pl.lit is scalar also does something strange with joins
from polars.
BTW the fact that pl.lit is scalar also does something strange with joins
@AlexeyDmitriev you mean this? #9603
from polars.
@AlexeyDmitriev you mean this? #9603
I added somewhat simpler example there.
from polars.
Related Issues (20)
- Column-autocomplete for eager DataFrames within interactive environments HOT 1
- Fix import of Worksheet from xlsxwriter HOT 1
- polars.read_csv_batched encoding Parameters not worked HOT 1
- Error when chaining a LazyFrame `.with_row_index()` with `.sink_csv()`
- Column name not found when joining by expressions (starts_with) HOT 1
- Unexpected behaviour when applying `struct.with_fields` to multiple columns HOT 1
- `scan_parquet` + `with_row_index` + selecting only the index returns empty dataframe
- scan_parquet doesn't work when reading files that have "|" in their names from blob storage HOT 6
- Allow numbers with thousand seperators to be parsed as Integer in `read_csv` HOT 1
- Unexpected behavior when apply group_by.agg
- `.str` methods silenty cast input HOT 1
- FreezeGun / Temporal subclasses not working in filter HOT 1
- The skip_nulls parameter of the map_elements method is unreasonable and confusing
- Attribute Error message not printing the attr that's missing
- pl.Concat of empty LazyFrames fails
- Unable to access the partitioning_flavor option when using write_parquet
- DataFrame.write_database() not passing along engine_options when using ADBC HOT 1
- Documentation suggestion: cover options for DataFrame creation from different Python data structures in User Guide and add example of list of dicts to API docs
- Calculate Entropy for non-numerical columns HOT 1
- Feature (rust, python) make the polars-python module public HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.