Giter VIP home page Giter VIP logo

event-driven-architecture's Introduction

Event Driven Architecture

Full Microservices Python Pub/Sub with RabbitMQ


๐Ÿš€ Build Status:


๐Ÿ’ก My idea

A simple excuse learn and use Python as Pub/Sub and a message broker, in this case RabbitMQ, to provisioning infrastructure triggered by event like "buy a small module" and finally monitoring all infrastructure.
I love IoT, for this reason, this PoC is designed to simulate a "SasS Product".
At the end of all this, it will provision my small IoT modules. ๐Ÿ‘พ


๐Ÿ”ฅ Supposed problem

๐Ÿ’€ I need to manage a lot inputs, all of them, will be to do different tasks, like messages, deployments, and more. Obvioulsly I will re-use data to make another jobs generate custom events, and finally ,will be use Grafana to see some analysis and tendences.
๐Ÿ’€ Some apps have to get information but a common problem is have or develop lot of products in with different technologies like, NodeJs,Python,Php.
๐Ÿ’€ I would like to have a shared origin to get data and avoid RE-build or make connectors or apis for connect different components in different tech/languages
๐Ÿ’€ All developers need to know what version must be fixed, or I need a human resource to manage version number. (I would like avoid manage it manually)


๐Ÿ Objetive

โœ”๏ธ Create a simple api to centralize all "inputs" and organize work-loads by queues.
โœ”๏ธ Each microservice consume his our queue and if need, can consume others to.
โœ”๏ธ Each microservice do a specific tasks, this is to avoid have "JUMBO-Pods".
โœ”๏ธ All microservice generate logs for future monitoring, analysis and make improvements or troubleshoot.
โœ”๏ธ All logs must to be expose in stdout, to avoid write data in the container. This way enable me to use my pods with ReadOnly Filesystem.
โœ”๏ธ Automate All tasks via microservice API-CALL environments.
โœ”๏ธ Generate a big scalability and security isolating each tasks in small actions/calls.
โœ”๏ธ Avoid tech dependences or "human-tech dependence". Each human can enjoy his own Tech/Language (...No, no java, please! ๐Ÿ˜‚ ).
โœ”๏ธ The standard (input/output) will be JSON because its an open standard and is easy to implement and easy to parse.
โœ”๏ธ To manage version numbers I created a Semantic Version GithubAction

Extra features


โœ”๏ธ Safe data: My apps use tokens or passwords, and I need to manage them safely and then I need to commit all data without leaking my secrets. ;)
โœ”๏ธ Vendor lock-in: In my case, I prefer have a infrastructure can be used and implemented in any cloud provider that runs a kubernetes cluster, in an "on-premise" client environment or even in a development environment like my laptop.
โœ”๏ธ Vulnerability Scans: Every time a Developer or SRE build a docker image must be scanned to find possible vulnerabilities. If it happen that image have to mark with different tag. Vulnerability Scans



Workflow logic



For automatic semantic release logic

graph LR
    A(git push) --> B>GitHub Action]
    B --> C[Get old version ]
    C --> D[1.0.0]
    B --> | git commit -m text: ...| E{Parse Trigger}
    E --> |patch: ...| F((1.0.1))
    E --> |minor: ...| G((1.1.0))
    E --> |major: ...| H((2.0.0))
    E --> |test: ... | I((1.0.0-wbp9lays))
    E --> |alpine ...| J((1.0.0-alpine))

Loading

Architecture design



Json data model (example)

{                                              /* Posible Inputs */ 
"client":"cliente02",                          /* Client Name / Identification */ 
"namespace":"cliente02",                       /* kubernetes namespace = client */
"environment":"Development",                   /* Dev / Stage / Prod */
"archtype":"SaaS",                             /* SaaS / Edge / On-Prem */
"hardware":"Dedicated",                        /* Classic (No extra cost allocated) / Dedicated (Extra cost allocated) */
"product":"Product-A",                         /* Product-A / -B / -C / -N */ 
"MessageAttributes": { 
  "event_type": { 
    "Type": "String",     
    "Value": "mycompany.producer.event.client.published"   /* (Dinamic) Company.App.messageType.client.EventAction */
    }, 
  "published_on": "2023.01.2.23.02.642883101",         /* +%Y.%m.%d.%H.%M.%N */ 
  "trace_id": "9a2ae9de-3f82-4f55-966b-47df50ff51ff",  /* uniq random string  */
  "retrace_intent": "0"                                /* how many reintents */
  }, 
  "Metadata": { 
    "host": "hostname",                       /* microservice */
    "origing": "Cloud",                       /* Cloud / On-Prem */
    "publisher": "producer"                   /* publisherType */
  } 
} 

TraceID from deployment workflow to pod annotations


This trace_id is super usefull when you need to see the deployment trace, and then I use it like reference tag and/or annotations in pods. trace_id is generated on first api-call in deployment process and is allocated to all the pods. If you use grafana or similar you can trace all steps and associate it to deployment or even in each pod deployed with this process



Producer (client portal)



DBClients UI (webserver)



Kubernetes Logs




Alerts and Messages



event-driven-architecture's People

Contributors

japseabery avatar jpradoar avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

event-driven-architecture's Issues

Buy a service using a credit card.

Tasks and or objetives:

  1. As a client I have to be able to buy a service using a credit card.
  2. As a owner I have to be able to validate data and cards.

DBwriter dont print trace_id

The DBwriter don't show trace_id, it is not an important part of deployment flow, but when you need to trace a transaction or trace data to do a troubleshoot it is important.
And neither can you can not obtain all trace when you filter a transaction in grafana dashboard.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.