network-analytics / mdt-dialout-collector Goto Github PK

I have been trying out this tool, looks very promissing..

there one thing though, is there a way I can make it serialize it into a json blob and not a json string where it escapes all the double quotes?

My use case is that I feed this into Kafka, and then ingest it into my ClickHouse database directly using a Kafka-Engine and then a materialized view and JSONEachRow which makes it easy for me to pick the objects I want and insert it into my table.

It sorta complicates things a bit if its not a "true" json object, but a string instead.. Can I turn change the behavior somewhere?

Esben

• I've tried this example https://grpc.io/docs/languages/cpp/quickstart/#build-the-example and it works.
• Cloned https://github.com/scuzzilla/mdt-dialout-collector.git and converted to .zip file and and unzip on my Linux machine.
• cmake ran successfully
  cmake.txt
• Getting an error while running make,
  make.txt
• Now it got failed at 3% in make,
  make1.txt
• System configuration
  systemconfig.txt
• @scuzzilla Let me know if you need any other information.

https://github.com/scuzzilla/mdt-dialout-collector/blob/main/src/mdt_dialout_core.cc#L110

In our lab environment, the certificate being used by Kafka is not always a valid one (either self-signed or the CN/SAN does not match the hostname). Kafka has a property that can be set, "enable.ssl.certificate.verification" but this is not available to the gRPC Kafka configuration. We are looking for something like the following for the pmtelemetryd-grpc-dialout.conf

## Kafka configuration ###
bootstrap_servers = "jedha-kafka-kafka-mtlsext-0.kafka.svc:9096,jedha-kafka-kafka-mtlsext-1.kafka.svc:9096,jedha-kafka-kafka-mtlsext-2.kafka.svc:9096";
topic = "matterhorn.ietf.grpc-raw";
enable_idempotence = "true";
client_id = "mdt-dialout-collector";
# valid options are either plaintext or ssl
security_protocol = "ssl";
ssl_key_location = "/etc/pmacct/ssl/kafka.key";
ssl_certificate_location = "/etc/pmacct/ssl/kafka.crt";
ssl_ca_location = "/etc/pmacct/ssl/ca.crt";
enable_ssl_certificate_verification = "false";  <---- New Kafka config property

Thanks,
Sean

When running the install.sh script, it determines the number of vCPUs by the following line:

readonly available_vcpu=$(egrep 'processor' /proc/cpuinfo | wc -l)

Later in the script, the value is used to determine the number of jobs to run for the 'make' command:

make -j`echo $((${available_vcpu} - 1))`

When running in Kubernetes/OpenShift, "egrep 'processor' /proc/cpuinfo | wc -l" gives the total number of processors on the node running the build as /proc/cpuinfo is not scoped by cgroup. The result is that it runs with as many jobs as there are cores (in our case, 72) which blows the RAM usage through the roof.

I was able to fix this by changing the value to 2 manually in the Dockerfile but if this could be added as an argument to install.sh (or available to override via an env variable) that would allow the build to work more effectively in a k8s-based environment

Thanks,
Sean

delete pid file on specific signals ...

In src/bridge/grpc_collector_bridge.h it is defined struct Payload. Such naming is very generic and, when used in 3rd party tools (ie. pmacct), it may be source of namespace conflicts. I would recommend to rename it to something more specific, ie. grpcPayload or something like that.

    typedef struct {
        char *event_type;
        char *serialization;
        char *writer_id;
        char *telemetry_node;
        char *telemetry_port;
        char *telemetry_data;
    } __attribute__ ((packed)) Payload;

We have already dealt with this after some conversations with Salvatore et al but we wanted to add a new issue to formally track it

When running pmtelemetryd with the mdt-dialout-collector library, the code cannot bind to a network interface defined by the 'iface' config key when the container is not running as root (which is a security requirement for our environment). Salvatore recommended trying to comment out the following code in src/core/mdt-dialout-core.cc:

    if (setsockopt(fd, SOL_SOCKET, SO_BINDTODEVICE,
        iface.c_str(), strlen(iface.c_str())) != 0) {
        spdlog::get("multi-logger")->
            error("[CustomSocketMutator()]: Unable to bind [{}] "
            "on the configured socket(s)", iface);
        std::abort();
    }

Removing this code worked and allowed the daemon to start up without any issues.

Could the 'iface' config key be made optional (or ignored) when deploying this into a container on Kubernetes/OpenShift? This would probably require a start arg or config key to identify when deploying in a container as well

Thanks,
Sean

develop a solution to be able to read the required configuration parameters from filesystem

Add apt install -y libfmt-dev to list of libraries required to Build & install the collector libraries (Integrate the collector)

make sure that the app behavior is logged with the right level of accuracy

the ZMQ socket uri should be dynamically configurable

I recommend replacing the instances of /root with $HOME.

Also i wonder if cd /opt are real necessary or can be changed to cd /tmp.

install.sh should be included in the CI/CD pipeline

The supplied example configuration is quite involved, probably to showcase all existing config directives, but it's probably not immediately applicable in a simple setup arranged for a proof-of-concept. I recommend to move the list of all config keys in a separate doc and provide a simpler example (or a few of them of increasing complexity). Maybe starting with:

writer_id = "mdt-dout-collector-01";
client_id = "mdt-dialout-collector";

iface = "eth0";

ipv4_socket_cisco = "0.0.0.0:10007";
ipv4_socket_huawei = "0.0.0.0:10008";
ipv4_socket_juniper = "0.0.0.0:10009";

replies_cisco = "10";
replies_juniper = "100";
replies_huawei = "1000";

cisco_workers = "1";
juniper_workers = "1";
huawei_workers = "1";

console_log = "true";
spdlog_level = "debug";
log_level = "0";

bootstrap_servers = "localhost:9093";
topic = "json.topic";
security_protocol = "plaintext";
enable_idempotence = "true";

introducing a more robust errors handling mechanism