Comments (7)
The reason is that, in general, operations:
:y = :x .* 2
and
:z = :y .+ 2
could be executed in parallel, so :y
is not yet present when you want to compute :z
.
However, DataFramesMeta.jl could split this single @transform
call into multiple transform
calls to get what you want (but it requires a decision if we want it).
from dataframesmeta.jl.
Yes, this would be nice. However DataFramesMeta.jl is restricted by how DataFrames.transform
works.
There is a work-around using the @astable
macro flags. Think of @astable
as a nice way to make multiple inter-dependent columns, as well as define multiple intermediate variables. In the latter sense, it's over kill for what you want, which is just to make columns. S you can do this:
julia> @transform DataFrame(x=[1,2,3]) @astable begin
:y = :x .* 2
:z = :y .+ 2
end
3×3 DataFrame
Row │ x y z
│ Int64 Int64 Int64
─────┼─────────────────────
1 │ 1 2 4
2 │ 2 4 6
3 │ 3 6 8
as well as
julia> @transform DataFrame(x=[1,2,3]) @astable begin
:y = :x .* 2
a = 5
:z = (:y .+ 2) .* a
end
3×3 DataFrame
Row │ x y z
│ Int64 Int64 Int64
─────┼─────────────────────
1 │ 1 2 20
2 │ 2 4 30
3 │ 3 6 40
We don't have @astable
as default because it allows for too many things, like intermediate variables, but also makes it so you have to keep track of how you type column names more carefully. The following will error
julia> @transform DataFrame(x=[1,2,3]) @astable begin
:y = :x .* 2
:z = $"y" .+ 2
end
So when you need interdependent variables, I recommend wither @astable
or just two @transform
calls.
Final note: You are aware of @rtransform
, right? Which makes row-wise operations easier.
from dataframesmeta.jl.
Ah that's great; I didn't realize @astable
had this effect. Perhaps worth adding/modifying an example in the docs to illustrate this; the current example only shows the temporary variable, not using previously defined columns?
I do wonder about @bkamins' suggestion of implementing @Transform with multiple transform calls. But perhaps this a substantial performance cost---as far as I know DataFrames doesn't try to parallelize within transform calls, so that doesn't seem like an issue to me.
And yes, I have m-tab bound to @rtransform
(m is a habit from dplyr).
from dataframesmeta.jl.
yeah a doc fix might be good. Do you want to start a PR (It's not a big deal, though, I can do it!)
I don't think I'm too concerned about the performance cost. Interested parties can always write a transform
call. I'm not concerned about code complexity and maintenance burden.
Not to put the burden on @bkamins , but a keyword argument in transform
and select
might be what I want. But if he doesn't want to add it, I'm happy with the status quo.
I do think @astable
is an underrated "killer feature" relative to dplyr
, so I should probably evangelize it more.
from dataframesmeta.jl.
But if he doesn't want to add it, I'm happy with the status quo.
It is not the issue of the burden, as we already have threads
keyword argument that controls that. The issue is just that I was afraid that changing the parsing rules depending on the value of this keyword argument would be a bit risky.
from dataframesmeta.jl.
Ah, I wasn't aware that transform was threaded by default. That's neat! In that case I retract the request; it definitely doesn't make sense to have different semantics based on the threads keyword.
I do think adding an example of using a previously defined column using @astable would be great though.
from dataframesmeta.jl.
I don't think this really warrants setting up a fork. Here's a suggestion to add to the docs:
You can also refer to previously defined columns within an `@astable` block.
julia> @transform df @astable begin
:c = :b .+ :a
:d = :c .^ 2
:d .-= maximum(:d)
end
Note that this is not possible in a bare `@transform` statement because
the separate columns might be created in parallel by separate threads (see [DataFrames Multithreading-support](https://dataframes.juliadata.org/stable/lib/functions/#Multithreading-support)).
You can use arbitrary julia syntax within @astable including other macros and control flow.
Note that we are using `@rtransform` here to treat each row independently.
julia > @rtransform df @astable begin
slow(x) = sleep(0.1 * x)
:runtime = @elapsed slow(:a)
bad_function(x) = iseven(x) ? error("odd!") : "even"
:c = try
bad_function(:a)
catch
missing
end
end
However, one important gotcha is that all new columns must be defined at the top level.
So, the following code would not work
@rtransform df @astable begin
if iseven(:a)
:parity = "even"
else
:parity = "odd"
end
end
This can be fixed by moving the `:parity =` before the `if` (like we did with the try block example)
or by initializing the variable, e.g. `:parity = missing` before the if statement.
from dataframesmeta.jl.
Related Issues (20)
- operators do not work inside function call inside macros HOT 3
- typos HOT 3
- Macro @rolling for scrolling through a column or columns of values? HOT 3
- Add a `@bycol` macro-flag HOT 5
- Add metadata for working with DataFrames HOT 1
- Access subdf in @by and @combine HOT 7
- Request - grouped by columns available as single values rather than vectors HOT 5
- Request: `@order` to mimic `DataFrames.order` in `@orderby` HOT 2
- Very slow `@astable` macro outside a function HOT 4
- `@with` macro clashes with `Base.@with` in Julia 1.11+ HOT 8
- `ByRow` not defined when importing DataFramesMeta HOT 1
- docs question HOT 7
- Request @rsubset_rtransform HOT 7
- Special-case `==` as with other one-argument functions HOT 2
- Add an alternative syntax escaping than `$` HOT 1
- MethodError occurred when broadcasting a string inside @astable HOT 3
- Speculative future of `@groupby` macro
- `groupby` derived columns
- Add convenience function to look up a single value in a `DataFrame` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataframesmeta.jl.