Add anomaly detector for AS hegemony cone (count of downstream networks) about kafka-toolbox HOT 3 OPEN

Hi Romain,

While reviewing the enhancement and going through the repository, I think that we need to add logic to compute the number of dependent networks in the psql/consumers/ASHegemony.py script.
The logic can be implemented as -

def compute_dependent_networks(as_hegemony_values):
    # Dictionary to store the count of dependent networks for each AS
    dependent_networks_count = {}

    # Iterate through AS hegemony values
    for as_value in as_hegemony_values:
        asn = as_value['asn']
        dependent_networks = as_value.get('dependent_networks', [])

        # Update the count for each dependent network
        for network in dependent_networks:
            if network not in dependent_networks_count:
                dependent_networks_count[network] = 1
            else:
                dependent_networks_count[network] += 1

    return dependent_networks_count

Additionally, creating a new consumer script to consume messages from the topic where the computed results are pushed, and adding a PostgreSQL schema for storing the count requests.
Well , I am new to Kafka as well as IHR and since I don't have hands-on experience with it, can you please guide me more towards implementing the solution

from kafka-toolbox.

Hi,

A similar code is already in psql/consumers/ASHegemony.py:

We need to decouple the two functions that this script has, pushing ASHegemony values to Postgres and computing the number of dependent networks. So I think we should start with a new script that reads ASHegemony values from Kafka and store the number of dependent networks back into kafka.

from kafka-toolbox.

Okay! On it

from kafka-toolbox.