internethealthreport / kafka-toolbox Goto Github PK
View Code? Open in Web Editor NEWCollection of producers, consumers, and basic tools for IHR kafka cluster
License: GNU General Public License v3.0
Collection of producers, consumers, and basic tools for IHR kafka cluster
License: GNU General Public License v3.0
Create README.md file for atlas/producers directory specifying -
Create README.md file for peeringdb/producers directory specifying -
don't loop forever in psql consumer
get data and push it once then terminate? so it can be run frequently by a cron job
handle cases where part of the data was already consumed (commit only pushed data)
Create a README.md file for all of the directories other than bgp/producers
File should contain-
@romain-fontugne I was planning to create an entire docker environment for our kafka server. But what I have observed is we have around 6-7 producers and a bunch of consumers in psql/consumers. Do you have any existing approach as to how to club these producers and consumers? I have 2 options:
Creating separate containers for each producer and consumer which will allow us to scale specific components if they generate data with high velocity and have fine-grained control over the components. But this can create a little bit of chaos as well since we'll have to deploy 20+ containers.
If we want a simpler deployment and management experience, we could combine all of the producers into one container and all of the consumer into another container. Just the fact I feel is it would limit our ability to scale individual components independently.
But entirely your call. We can move forward accordingly.
Is your feature request related to a problem? Please describe.
Currently we cannot detect anomalies in the number of dependent networks because these value is computed just before storing it in the database.
Describe the solution you'd like
Create README.md file for atlas/producers directory specifying -
Using requirements.txt can specify the version of a certain module. Pip install will download the newest module by default, but it may be incompatible with the system. Also, we can use requirements.txt in Dockerfile.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.