Comments (3)
Thank you @c-arstensen for the full example. I did found an issue on with DeduplicateItems
when ION data contains date (which is the case for data coming from kafka).
from kestra.
Hi @c-arstensen, can you please share a more complete example to reproduce the error, more specifically, how you got the ION file from {{ trigger.uri }}
? I did some tests using a simple ion file and everything works fine. Thank you.
from kestra.
In my case the trigger.uri is coming from io.kestra.plugin.kafka.Trigger
- The flow looks like this
triggers:
- id: consumetrigger
type: io.kestra.plugin.kafka.Trigger
groupId: kestra-nonprodmap
properties:
auto.offset.reset: latest
bootstrap.servers: "{{ secret('KAFKA_HOST') }}"
sasl.jaas.config: >-
org.apache.kafka.common.security.scram.ScramLoginModule required
username="{{ secret('KAFKA_USER') }}"
password="{{ secret('KAFKA_PASSWORD') }}";
sasl.mechanism: SCRAM-SHA-512
security.protocol: SASL_PLAINTEXT
topicPattern: ^.*event.cnc.map
keyDeserializer: STRING
valueDeserializer: JSON
# Very Important! The default pollDuration of 2.0 seconds is not long enough
pollDuration: 10.0
tasks:
# This fails - I can get it to work by passing trigger.uri into a jsonWriter and then reading outputs.json.uri into dedup....
- id: dedup
type: io.kestra.core.tasks.storages.DeduplicateItems
from: "{{ trigger.uri }}"
expr: "{{ key }}"
The messages from our topics are relatively complicated JSON structures and each message is guaranteed to have a key.
from kestra.
Related Issues (20)
- WaitFor with 'maxIterations' is interupted too early
- Speed up loading of plugin icons (perhaps add caching?) in the Plugins UI
- Make the blueprints view more compact
- No password length constraint
- Crush upon filter: end date before start date in URI
- Reviews Onboarding HOT 14
- Pagination issue
- Dynamic execution labels from triggers
- Default image for Powershell should have WSMan installed by default
- Add multiline text input support
- Worker logs WARN: Worker Evaluate Failed with error 'null' when a flow with realtime triggers is deleted
- Move task runner out of beta
- Review Welcome page
- Use taskrunner in the various CLI tasks
- Enable/Disable Realtime Triger from the UI should not be possible HOT 2
- QA NamespaceFiles tasks HOT 2
- Provide a UI field allowing to copy a full Webhook URL e.g. on the Flow Overview page HOT 1
- Docker volmune mounting broken
- terminationGracePeriod is not respected (leading to infinite wait for worker task to finish)
- Sftp Plugin Missing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kestra.