Giter VIP home page Giter VIP logo

Comments (3)

fhussonnois avatar fhussonnois commented on June 1, 2024 2

Thank you @c-arstensen for the full example. I did found an issue on with DeduplicateItems when ION data contains date (which is the case for data coming from kafka).

from kestra.

fhussonnois avatar fhussonnois commented on June 1, 2024

Hi @c-arstensen, can you please share a more complete example to reproduce the error, more specifically, how you got the ION file from {{ trigger.uri }}? I did some tests using a simple ion file and everything works fine. Thank you.

from kestra.

c-arstensen avatar c-arstensen commented on June 1, 2024

In my case the trigger.uri is coming from io.kestra.plugin.kafka.Trigger - The flow looks like this

triggers:
  - id: consumetrigger
    type: io.kestra.plugin.kafka.Trigger
    groupId: kestra-nonprodmap
    properties:
      auto.offset.reset: latest
      bootstrap.servers: "{{ secret('KAFKA_HOST') }}"
      sasl.jaas.config: >-
        org.apache.kafka.common.security.scram.ScramLoginModule required 
        username="{{ secret('KAFKA_USER') }}" 
        password="{{ secret('KAFKA_PASSWORD') }}";
      sasl.mechanism: SCRAM-SHA-512
      security.protocol: SASL_PLAINTEXT
    topicPattern: ^.*event.cnc.map
    keyDeserializer: STRING
    valueDeserializer: JSON
    # Very Important! The default pollDuration of 2.0 seconds is not long enough
    pollDuration: 10.0

tasks:
 # This fails - I can get it to work by passing trigger.uri into a jsonWriter and then reading outputs.json.uri into dedup....
  - id: dedup
    type: io.kestra.core.tasks.storages.DeduplicateItems
    from: "{{ trigger.uri }}"
    expr: "{{ key }}"

The messages from our topics are relatively complicated JSON structures and each message is guaranteed to have a key.

from kestra.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.