internethealthreport / kafka-toolbox Goto Github PK

Create README.md file for atlas/producers directory specifying -

1. Requirements
2. Usage
3. Deprecated
   Any other useful information will be appreciated.
   For reference see bgp/producers

• make a producer for long lasting atlas measurement ids
• traceroute producers should get updated msmids

Using requirements.txt can specify the version of a certain module. Pip install will download the newest module by default, but it may be incompatible with the system. Also, we can use requirements.txt in Dockerfile.

Create README.md file for peeringdb/producers directory specifying -

1. Requirements
2. Usage
3. Deprecated
   Any other useful information will be appreciated.
   For reference see bgp/producers

• Add a producer for AS names
• Consumers (e.g. psql/ASHegemony.py) should get names there

don't loop forever in psql consumer
get data and push it once then terminate? so it can be run frequently by a cron job

handle cases where part of the data was already consumed (commit only pushed data)

Is your feature request related to a problem? Please describe.
Currently we cannot detect anomalies in the number of dependent networks because these value is computed just before storing it in the database.

Describe the solution you'd like

• We should compute the number of dependent networks from AS hegemony values in Kafka and push the results to Kafka.
• So we can also run an anomaly detector for it.
• And we'll need a new consumer to store these values to the postures database.

Create README.md file for atlas/producers directory specifying -

1. Requirements
2. Usage
3. Deprecated
   Any other useful information will be appreciated.
   For reference see bgp/producers

@romain-fontugne I was planning to create an entire docker environment for our kafka server. But what I have observed is we have around 6-7 producers and a bunch of consumers in psql/consumers. Do you have any existing approach as to how to club these producers and consumers? I have 2 options:

1. Creating separate containers for each producer and consumer which will allow us to scale specific components if they generate data with high velocity and have fine-grained control over the components. But this can create a little bit of chaos as well since we'll have to deploy 20+ containers.

2. If we want a simpler deployment and management experience, we could combine all of the producers into one container and all of the consumer into another container. Just the fact I feel is it would limit our ability to scale individual components independently.

But entirely your call. We can move forward accordingly.

Create a README.md file for all of the directories other than bgp/producers

File should contain-

1. Requierements
2. Usage
3. Deprecated
   at least. And any other useful information will be appreciated.
   For better understanding refer to bgp/producers Readme