kafka-streams-fan-in-example's Introduction

Example Kafka Streams application with fan-in

 ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐ 
 │   strings   │   │    ints     │   │   doubles   │   │    longs    │ 
 └──────┬──────┘   └──────┬──────┘   └──────┬──────┘   └──────┬──────┘ 
        │                 │                 │                 │        
        │                 │                 │                 │        
        │                 │                 │                 │        
        │                 │                 │                 │        
        │                 │                 │                 │        
        │                 │     ╔═════╗     │                 │        
        │                 │     ║merge║     │                 │        
        │                 │     ╚═════╝     │                 │        
        └─────────────────┴────────┬────────┴─────────────────┘        
                                   │                                   
                                   │                                   
                                   │╔════════════════╗                 
                                   │║aggregate-by-key║                 
                                   │╚════════════════╝                 
                                   │                                   
                                   │                                   
                                   │                                   
                                   ▼                                   
                            ┌─────────────┐                            
                            │ merge-parts │                            
                            └──────┬──────┘                            
                                   │                                   
                                   │╔═════════════════════════════════╗
                                   │║filter : containsAllRequiredParts║
                                   │╚═════════════════════════════════╝
                                   │╔═════════════════════════════════╗
                                   │║        map : mergeParts         ║
                                   │╚═════════════════════════════════╝
                                   │                                   
                                   ▼                                   
                            ┌─────────────┐                            
                            │merge-results│                            
                            └─────────────┘

How to run?

A docker compose file is provided that sets up the following infrastructure

Zookeeper
Kafka
Schema Registry

as well as boots up the two main classes in this project

FanInExample
ExampleProducer

and a console consumer for looking at the output.

Running this constellation of applications can thus simple be done via

docker-compose up -d

after which you can look at the logging for a particular piece via

docker-compose logs -f avro_console_consumer

kafka-streams-fan-in-example's People

Contributors

Stargazers

Watchers

kafka-streams-fan-in-example's Issues

`filter` on `KTable` is misused

Based on the discussion in #1, we realise that the filter in https://github.com/CasperKoning/kafka-streams-fan-in-example/blob/master/src/main/scala/xyz/casperkoning/poc/FanInExample.scala#L60 is not doing what it should

Fanout example

Thank you for such great fan-in example!

I have a couple of questions:

How does filtering semantics work? Does the example produce an aggregated result as long as all required parts are available?
Do you think Kafka Streams is convenient for implementing fan-out pattern with routing rules? I'm looking for alternative to header exchange in RabbitMQ that allows to route given message to multiple output queues based on message attributes.

Recommend Projects