Welcome to the Ultimate DevOps Monitoring Project! This repository contains the configuration and setup for a comprehensive monitoring stack using Prometheus, Node Exporter, Alertmanager, and Blackbox Exporter. The project demonstrates how to monitor systems and applications effectively, collect vital metrics, and set up alerting mechanisms to ensure system reliability.

• Introduction
• Project Architecture
• Prerequisites
• Installation
• Configuration
• Usage
• Alerting
• Contributing

This project is designed to provide hands-on experience with setting up and managing a robust monitoring infrastructure. It includes:

• Node Exporter: For collecting hardware and OS-level metrics.
• Prometheus: As the central monitoring system.
• Alertmanager: To handle alerts and notifications.
• Blackbox Exporter: For probing endpoints to check their availability and performance.

• VM-1: Hosts Node Exporter, which exposes system-level metrics to Prometheus.
• VM-2: Hosts Prometheus, Alertmanager, and Blackbox Exporter, responsible for scraping metrics, managing alerts, and probing endpoints.

Before you begin, ensure you have the following installed on both VMs:

• wget
• tar
• Appropriate permissions to download, extract, and execute binaries.

1. Download Node Exporter:

   wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz

2. Extract Node Exporter:

   tar xvfz node_exporter-1.8.1.linux-amd64.tar.gz

3. Start Node Exporter:

   cd node_exporter-1.8.1.linux-amd64
   ./node_exporter &

1. Download Prometheus:

   wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz

2. Extract Prometheus:

   tar xvfz prometheus-2.52.0.linux-amd64.tar.gz

3. Start Prometheus:

   cd prometheus-2.52.0.linux-amd64
   ./prometheus --config.file=prometheus.yml &

1. Download Alertmanager:

   wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz

2. Extract Alertmanager:

   tar xvfz alertmanager-0.27.0.linux-amd64.tar.gz

3. Start Alertmanager:

   cd alertmanager-0.27.0.linux-amd64
   ./alertmanager --config.file=alertmanager.yml &

1. Download Blackbox Exporter:

   wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.25.0/blackbox_exporter-0.25.0.linux-amd64.tar.gz

2. Extract Blackbox Exporter:

   tar xvfz blackbox_exporter-0.25.0.linux-amd64.tar.gz

3. Start Blackbox Exporter:

   cd blackbox_exporter-0.25.0.linux-amd64
   ./blackbox_exporter &

Define your global settings, scrape intervals, and targets in the prometheus.yml file:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['<Node_Exporter_IP>:9100']

  - job_name: 'blackbox'
    metrics_path: /probe
    static_configs:
      - targets:
          - http://prometheus.io
          - https://prometheus.io
          - http://<Your_Target_IP>:8080/
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <Blackbox_Exporter_IP>:9115

Set up alerting routes, receivers, and inhibition rules in alertmanager.yml:

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email-notifications'

receivers:
  - name: 'email-notifications'
    email_configs:
      - to: [email protected]
        from: [email protected]
        smarthost: smtp.gmail.com:587
        auth_username: your_email
        auth_password: "your_smtp_password"
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Once everything is set up, you can start monitoring your systems. Navigate to Prometheus’s web UI (http://<Prometheus_IP>:9090) to explore metrics and set up alerts.

To test the Blackbox Exporter, navigate to http://<Blackbox_Exporter_IP>:9115/probe?target=http://yourtarget.com.

The project includes predefined alert rules in alert_rules.yml for common scenarios such as:

• InstanceDown: Triggered when a target is unreachable.
• WebsiteDown: Triggered when a website probe fails.
• HostOutOfMemory: Triggered when available memory is low.
• HostOutOfDiskSpace: Triggered when disk space is running out.

These alerts are configured to send notifications via Alertmanager.

Contributions are welcome! Please fork the repository and submit a pull request.