This Project is all about monitoring the performance of ML Models using various tools like:

1. Prometheus: A monitoring system that wil monitor the performance of ML Models and various other metrics like CPU, Memory, Disk andf manily data drift.

2. Grafana: A visulization tool which will take the metrics from Prometheus and will display them in a graph.

3. Evidently: A tool which will capture the whole ml model performance from data drift to concept drifting and a lot.

4. MongoDB: A database that will capture the prediction in real time and drop it in the DB in the form of documents.

5. Prefect: A tool pretty much similar to Apache Airflow, which majorly used for creating data analytics pipelines and automate task and workflows.

6. Docker-Compose: A part of docker to launch multiple containers pretty much easily.

1. Docker
2. Docker-Compose
3. MongoDB

1. Create a conda enviroment conda create -n monitoring python=3.8 -y
2. Run conda activate monitoring
3. Install all the dependencies pip install -r requirements.txt
4. Run prepare.py to download data, the data is present in Cloundfront CDN and it will be feteched from their and dropped in the root direc.

Start all the services run:

• docker compose up

The compose will start the following services:

1. prometheus - TSDB for metrics
2. grafana - Visual tool for metrics
3. mongo - MongoDB, for storing raw data, predictions, targets and profile reports
4. evidently_service - Evindently RT-monitoring service
5. prediction_service - main service, which makes predictions

Run the following command to send data to the service:

python send_data.py

The script will send each row from the parquet file to the service and perform predictions the prediction will be stored in the target.csv with a unique id generated by uuid4.

A general format is: 885da328-a41b-4c94-8fbd-f54e14f94cab 20.2

All side jobs are running and their are two services running one is prediction_service and the other is evidently_service.Both communicate with the mongo service to store the data.

Run the following command to perform batch inference:

python prefect_example.py

prefect orion start

1. Upload target.csv to MongoDB
2. Load a reference dataset into MongoDB
3. Fetch both reference and target datasets from MongoDB
4. Perform batch inference
5. Generate Evidently HTML report

So, this is the end of the project.

• Would like to thank DataTalkClub for the help in creating this project.