Project Itinerix has been created as a PoC for my PhD where I use graph processing applied to tracing data.

This is an integration project that takes State-of-the-Art Cloud Monitoring tools in aim to create an end-to-end trace processing platform within a Kubernetes Cluster. The scope of this repository covers:

1. Creation of The Kubernetes Cluster
2. Installation & Configuration of a Service Mesh in aim to gain Observability at the network level within the Cloud Application.
3. Installation & Configuration of Cloud Native Monitoring Tools (Jaeger and Prometheus) following OpenTelemetry recommendation as much as possible.
4. Installation & Configuration of a Notebook engine feed with Monitoring Tools as Data-Source.
5. Installation & Configuration of a Neo4j Graph Database to store and process graph created within the Notebook
6. Installation & hacking tunning a sample Cloud Application compatible with tracing.

This work has been inspired by the Medium publications of Pavol Loffay, a RedHat Developer working on the OpenTelemetry Working Group:

• Data Analytics with Jaeger: a.k.a Traces tell us more
• Jaeger Data Analytics with Jupyter Notebooks

The key idea behind this project is take profit of the heavily connected nature of tracing data to create and maintain, at runtime, a property graph modeling the performance of a Cloud-Application.

Nowadays more and more applications are developed to be Cloud-Native, which means some constraints:

• The application is developed as a fully distributed system
• Components are disposable and may be volatile
• The application is built on top of abstractions layers managed by third-parties that may not be monitored

Monitoring such systems involves addressing new challenges in the APM community. Indeed, for Cloud-Applications, ensuring bottlenecks are identified is a critical criteria for delivering the service and scaling. Also, the ability to identify the root-cause in an error propagation scenario is also crucial to patch and recover the system. These challenges involve maintaining a global view of a rapidly evolving distributed system.

This is why tracing became an important topic among the companies doing their business in the Clouds [1, 2, 3, 4, 5]. Recent initiatives like OpenTelemetry aims to normalize, and also to provide an implementation, on how trace data is passed from the monitoring system to the APM. As a result, we can expect tracing data to follow some well-defined schema [6] and to respect some semantic conventions [7, 8].

However, state-of-the-art trace-based APM lack maturity and fail to provide a global view of the system. Their main purpose is helping the developer to understand interactions between the components while debugging and optimizing their code [9].

This illustration does not mention the technology used for the service Mesh, indeed as of today the twon main Service Mesh are evaluated:

1. Linkerd: Implemented Linkerd proxies send OpenCensus formated data to local OpenTelemetry agent that then forwards to a OpenTelemetry Collector
2. Istio: Still Implementing in aim to get number of bytes exchanged in each network communications.

This projects relies on some Infrastructure as Code (IaC) tools in order to get the deployment as automated as possible, please install these tools by following the documentation of each links:

1. Terraform >= v0.13: Terraform is the IaC tool compatible with the widest spectrum of Cloud providers. It is used to automate the various stages of the deployment and to have a reproducible way to create the platform.
2. kubectl >= v1.19: kubectl is the command line interface used to apply Kubernetes manifests to a Kuberntes cluster but also to debug applications deployed on the cluster.
3. Kustomize >= v3.8: Kustomize is a tool that is used to merge multiple Kubernetes manifests files. It is heavyly used in this repository to keep configuration logically organized. Eventhough Kustomise is now bundled into kubectl, different behaviours have been observed between the standalone version and the bundled version when using advanced strategig merges, the standalone version has been kept because of the quicker pace of update the tool receive.
4. gcloud: if deploying the Kubernetes Cluster on Google Cloud (which is the only target cluster currently supported btw), both Terraform and kubectl will require the gcloud cli.
5. GNU Make: proabaly already installed on your machine, Make is used to bootstrap some kustomize commands or to retreive publically available Kubernetes Manifests, it does not bring any value out of this small scope, but avoids copy-pasting public code in the repository.

In order to deploy the Kubernetes cluster on Google Cloud, a Google Account is required, you might want more information Google Cloud Free Tier. Follow this basic tutorial to ensure gcloud is correcly authenticated (pick Local shell).

This section is Work in Progress

Itinerix deployment is a 2-stages process, each having its own directory within the project:

1. Deploying common components with a Terraform project hosted within the ./infrastructure directory: This deploys on GCP a Zonal Kuberntestes cluster with the Linkerd Service Mesh, a Jaeger All-in-One instance and various other modules that configure the plateform.

2. Deploying Test Application: @TODO