This should be enough to set up centralized logging and monitoring fairly easily. Prometheus will be configured to monitor every host in the inventory (see roles/prometheus/templates/prometheus.yml.j2). Loki agents will point to the loki server(configured as loki_server variable). All of these services are installed as binaries with systemd.

• Prometheus
• Prometheus Alertmanager
• Prometheus Node Exporter
• Loki
• Promtail

For the prometheus config to work correctly, the ansible inventory hostname of each server must be a DNS name that your prometheus server can resolve. Current supported distributions are Debian 10/11 and Ubuntu 20/22. It may work on other distros, but it has not been tested.

- name: bootstrap monitoring server
  hosts: monitoring
  become: True
  vars:
    loki_server: loki_server.local
  roles:
    - fahcsim.grafana_stack.prometheus
    - fahcsim.grafana_stack.loki
    - fahcsim.grafana_stack.prometheus_node_exporter
    - fahcsim.grafana_stack.prometheus_alert_manager
    - fahcsim.grafana_stack.promtail
    - fahcsim.grafana_stack.grafana

Each service can be updated by changing the <service>_version variable, as long as a corresponding version exists in the project's github repo.

prometheus_version: 2.42.0 prometheus_retention_time: "15d" extra_prometheus_config: <additional prometheus config goes here - If you want to add custom prometheus configuration, put it in this variable.

alert_manager_version: 0.25.0 pushover: true - This variables defines whether or not to include the Pusover config section. pushover_token: <pushover token> pushover_key: <pushover user key> extra_alertmanager_config: <additional alertmanager config goes here - If you're not using pushover, or have additional config, put it in this variable.

loki_version: 2.7.3

node_exporter_version: 1.5.0

promtail_version: 2.7.3 loki_server: localhost - change this to whatever host you want to use as the loki server

set the variable node_exporter_binary_local_dir to the path of the node exporter binary you copied locally

- name: bootstrap monitoring server
  hosts: monitoring
  become: True
  vars:
    extra_prometheus_config: |
      rule_files:
        - /etc/prometheus/rules.d/alert.rules.yml
  roles:
    - fahcsim.grafana_stack.prometheus

Several variables can be used to customize the config file for promtail:

• custom_server_config - If defined this will replace the "server" section of the config with whatever is defined in the variable.
• custom_scrape_configs - If defined this will replace the "scrape_configs" section of the config with whatever is defined in the variable.
• loki_server - If defined this will replace the loki server's address (if using this, you must also define the port as shown below)
• loki_port - If defined this will replace the loki server's port (if using this, you must also define the address as shown above)