Giter VIP home page Giter VIP logo

bank-transactions-queries's Introduction

Bank Transactions: Stateful Processing in Kafka Streams

The purpose of this tutorial is for us to learn about stateful processing in Kafka Streams. The youtube video associated with this tutorial can be found here: YouTube Video. To follow this tutorial, clone this repository and switch to the branch tutorial_start.

git clone https://github.com/Programming-with-Mati/bank-transactions.git

Once the repository is cloned run:

git checkout tutorial_start

To run a working copy of the code, go to Running the project

Our Application

Mati Bank Ltd. is a bank that holds its customer's bank balances in a Kafka Streams application. We are going to develop a topology that will process bank transactions and will stream updates of every new balance for a given user. The application will also stream all the rejected transactions into a different topic for error handling purposes.

The Data Model

DataModel

Bank Transaction

The BankTransaction will hold the data regarding the transactions that will be processed. It will hold an amount that can be either positive or negative, representing credits or debits, respectively.

Bank Balance

The BankBalance class will store the account balance for the user, and it will also hold information about the latest transaction.

The Topology

DataModel In the topology, we are going to:

  1. Stream the topic bank-transactions where the key is the balanceId
  2. `groupByKey the stream
  3. Use the aggregate operation to aggregate the all the transactions that have the same key into a single account balance. The balances will be stored in a State Store. This operation will return a KTable
  4. Transform the KTable into a KStream. This means that any change occurring in the table will be streamed as an event
  5. Stream the events into the bank-balances topic
  6. map the BankBalance into BankTransaction by keeping the last transaction
  7. filter the transactions to keep only the REJECTED transactions
  8. stream the rejected transactions into the rejected-transactions topic

The code

Given we have done several tutorials on Kafka Streams, I'm providing a stub of the project with a lot of the classes implemented.

The class BankBalanceApp is the entry point of our application, were we create our KafkaStreams application.

The model package contains the model classes with all its members. We have a method called process in BankBalance which we will impement as part of this tutorial.

The class BankBalanceTopology has an empty method buildTopology() which we will also implement.

There is a BankTransactionProducer which we will use to run some tests manually.

Finally, there is a test class called BankBalanceTopologyTest, with two tests already implemented. We will have to make these tests pass as part of our tutorial.

Running the project

Starting Kafka

First, we need to start Kafka. For that we have a docker-compose.yml file that will create the necessary resources for us. It will start a Zookeeper instance and a Kafka broker. It will also create the necessary topics using the script found in the create-topics.sh file.

docker compose -f ./docker-compose.yml up

Building and starting the application

./mvnw compile exec:java -Dexec.mainClass="com.github.programmingwithmati.kafka.streams.wordcount.VoiceCommandParserApp"

bank-transactions-queries's People

Contributors

matiasdsalerno avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.