Comments (4)
I've seen instrumentation, but they are asynchronous
Instrumentation is not asynchronous, see for example https://github.com/zorbash/opus/blob/master/lib/opus/pipeline.ex#L118
Is there a way for an if/unless step to modify the pipeline data?
Do you mean something like:
defmodule ArithmeticPipeline do
use Opus.Pipeline
step :add_one, with: &(&1 + 1)
step :enhance, if: &(&1 > 42)
def enhance(n), do: %{number: n, metadata: "the number was greater than 42"}
end
The library supports conditional stages, so you can modify the pipeline data.
from opus.
Ah. Actually you are right about instrumentation. I think that explains why when I tried to access data in my pipeline instrumentation that didn't happen to exist, and instrumentation raised an error stopped the pipeline. It might be better if it was treated more like a tee: or wrapped in a way that it cannot stop the pipeline if the steps inside raise an error.
Back to metadata Let's take the example repo guide, and modify it a little. I tend to use a Map in the pipeline for complex flows.
defmodule ArithmeticPipeline do
use Opus.Pipeline
step :add_one
step :randomize, if: :lucky_number?
# Entry point takes initial value and creates a map for the duration of the pipeline
def add_one(value) do
%{number: value + 1}
end
# steps mutate the pipeline
def randomize(pipeline) do
new_value = pipeline.n*:rand.uniform(10)
put_in(pipeline, [:number], %{number: new_value})
end
# Your conditionals normally return a simple true/false, but I would like to be able to return data into the pipeline data stream, so I'm meta-programming an example of how this might work.
If a conditional did not want to mutate the pipeline, it could just return a boolean for compatibility.
def lucky_number?(pipeline) do
if pipeline.number == 41 do
meta = %{
lucky: true,
metadata: "Lucky number."
}
{true, Map.merge(pipeline, meta)}
else
meta = %{
lucky: false,
metadata: "Sorry - No luck here."
}
{false, Map.merge(pipeline, meta}
end
ArithmeticPipeline.call(41)
# {:ok, %{number: 42, lucky: true, metadata: "Lucky number."}}
ArithmeticPipeline.call(4)
# {:ok, %{number: 32, lucky: false, metadata: "Sorry - No luck here."}}
Hope this example make sense - it would be awesome to be able to capture data from the conditional checks.
from opus.
Why do you want the condition function to transform the pipeline context?
In your example:
def lucky_number?(pipeline) do
if pipeline.number == 41 do
meta = %{
lucky: true,
metadata: "Lucky number."
}
{true, Map.merge(pipeline, meta)}
else
meta = %{
lucky: false,
metadata: "Sorry - No luck here."
}
{false, Map.merge(pipeline, meta}
end
ArithmeticPipeline.call(41)
# {:ok, %{number: 42, lucky: true, metadata: "Lucky number."}}
ArithmeticPipeline.call(4)
# {:ok, %{number: 32, lucky: false, metadata: "Sorry - No luck here."}}
You can achieve the same result with:
defmodule ArithmeticPipeline do
use Opus.Pipeline
step :add_one
step :classify
step :add_metadata
step :randomize, if: :lucky_number?
# Entry point takes initial value and creates a map for the duration of the pipeline
def add_one(value) do
%{number: value + 1}
end
# steps mutate the pipeline
def randomize(pipeline) do
new_value = pipeline.n*:rand.uniform(10)
put_in(pipeline, [:number], %{number: new_value})
end
def classify(%{number: 41} = context), do: put_in(context, [:lucky?], true)
def classify(%{number: _} = context), do: put_in(context, [:lucky?], false)
def lucky_number?(%{lucky?: true}), do: true
def lucky_number(_), do: false
def add_metadata(%{lucky?: true}) do
put_in(context, [:meta], {
lucky: true,
metadata: "Lucky number."
})
end
def add_metadata(%{lucky?: false}) do
put_in(context, [:meta], {
lucky: false,
metadata: "Sorry - No luck here"
})
end
In many cases you don't even need to make a stage conditional and you can pattern-match instead.
from opus.
I would like conditional logic features - which today are if/unless/skip - to capture or mutate data within that context/pipeline. I might inject context specific data about why something was skipped, or why it didn't pass a check. These are metadata or checks or warnings found from checks during the pipeline, in synchronous long pipelines that must complete, and having that would explain in the end what was skipped, and why in their payload.
You could do the above. I've done it, and it works. I love pattern matching, but when we are orchestrating from the pipeline as the the controller, I see the power of the Opus library in keeping the control logic patterns at the top to keep it readable and manageable for long flows. I feel that the method using deeper pattern matching reduces overall readability of the orchestration of the pipeline to have to explore multiple pattern matches.
Instead of talking straight conditionals like if/unless, can I track the value of a "skip:" step using an alternative code like the above (inline to the data, not instrumentation)? That's also of value to me, but I don't see how to record that event in the pipeline data.
Maybe I just need to adopt the above and have a few more functions and steps like maybe_this, and use more pattern matching. It's just a question and an idea, and I'm grateful for the library and your helpful replies and clear examples of alternatives.
from opus.
Related Issues (16)
- Halting the pipeline without raising an error HOT 4
- Decouple Graphvix HOT 8
- What's the plan on Elixir version support? HOT 2
- Add a :unless param for stages HOT 1
- (ErlangError) Erlang error: "Graphvix.Graph Graphvix.Graph received unexpected message in handle_info/2: :save_state\n" HOT 1
- Disable importing instrumentation HOT 7
- Conditional link HOT 4
- Accept a function in :error_message
- Parameterize steps HOT 2
- Feature: add `.matching/2` sugar HOT 1
- question: is there any equivalent link+tee function ? HOT 2
- Add an accessor pattern HOT 2
- Add "transform" to Link
- Bad instrumentation for skip stage HOT 1
- Graphics used on Pipeline Execution wiki page HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opus.