timothyrenner / kafka-streams-ex Goto Github PK

View Code? Open in Web Editor NEW

64.0 8.0 23.0 351 KB

A collection of examples and use-cases for Kafka Streams

License: MIT License

Java 87.66% Shell 12.34%

kafka-streams-ex's Introduction

Kafka Streams Examples

This repository contains examples of use cases (ranging from trivial to somewhat complex) of Kafka Streams.

Each example is in it's own directory. The repository contains the following examples:

Exclamation: Trivial example that reads from the console consumer and appends two exclamation points.
Exclamation Advanced: Slightly more complicated version of Exclamation that "alerts" on highly exclamated messages.
Hopping Windows: Example demonstrating the behavior of hopping windows by counting the elements on a single key.
Tumbling Windows: Example demonstrating the behavior of tumbling windows by counting the elements on a single key.
Processor: Example demonstrating the processor API, state stores, and custom serializers.
Instrumented Processor: A stripped down version of the processor example that logs the values in the state store - designed to run in two nodes (or just two terminals) to show what happens under failover conditions.
Not Looking at Facebook: Implementation of a streaming pipeline for notifying users when they aren't looking at Facebook.
KTable: Literally a KTable.
Windowed Delay: Demonstration of event-time ordering.

kafka-streams-ex's People

Contributors

Stargazers

Watchers

kafka-streams-ex's Issues

Can't compile

[WARNING] 
java.lang.ClassNotFoundException: io.github.timothyrenner.kstreamex.notification.NotLookingAtFacebook
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:281)
        at java.lang.Thread.run(Thread.java:745)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.951 s
[INFO] Finished at: 2017-03-21T16:05:20-07:00
[INFO] Final Memory: 12M/379M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project not-looking-at-facebook: An exception occured while executing the Java class. io.github.timothyrenner.kstreamex.notification.NotLookingAtFacebook -> [Help 1]

➜  not-looking-at-facebook git:(master) java -version         
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

➜  not-looking-at-facebook git:(master) update-alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*  1           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-8.b14.fc25.x86_64/jre/bin/java)
 + 2           /usr/java/jdk1.8.0_121/jre/bin/java

What am I doing wrong?

Stale State Overwrite in Processor Example

There's a bug in the Processor API example in which the in-memory hash map that's periodically committed to the state store could clobber an updated state for stale keys. This happens when there's more than one processor running and one of them goes down. Essentially this is how it goes:

Two processors run (against two partitions) with their respective keys in the state stores / hash maps.
Processor 1 goes down, so the state store gets initialized on processor 2. Still all good since there's no overlap in keys.
Processor 2 goes back up, gets it's partition back. Still all good because processor 1 was updating all values.
Processor 2 goes down again, and processor 1 gets its state store and partition. However, processor 1 has the old values in it's in-memory hash map for the partition 2 keys, which get written to the state store when punctuate gets called, thereby erasing the state processor 2 had updated it with the stale values from when it last went down.

Solution: Ditch the in-memory hash map. There isn't a workaround here. If we want it durable, it needs to be managed. I still want to use the punctuate method for stuff though so I'll need to modify the example.

timothyrenner / kafka-streams-ex Goto Github PK

kafka-streams-ex's Introduction

Kafka Streams Examples

kafka-streams-ex's People

Contributors

Stargazers

Watchers

Forkers

kafka-streams-ex's Issues

Can't compile

Stale State Overwrite in Processor Example

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent