Fluent Beats is a set of observability pipelines for Docker Containers and Linux Hosts, compatible with Elastic observabilily features based on FluentBit

• Docker

• Docker Swarm Mode
• AWS ECS

Fluent Beats uses the Elastic Common Schema to send data that can be ingested and consumed in key Kibana contexts

• Analytics
  • Dashboard
  • Discovery
  • Canvas
  • etc
• Observability
  • Alerts
  • Logs
  • Infrastructure
    • Metrics Explorer
    • Hosts

• Observability
  • Infrastructure
    • Inventory
      • AWS and K8s filters are not supported
  • Uptime
    • Uptime Monitors
      • Only container over TCP (translated from healthchecks)

• APM
• User Experience

This service is designed to provide lightweight observability capababilities for microservices running as Docker containers.

Its was designed to be deployed as a Docker Swarm Global Service; or AWS ECS equivalent; in order to receive any type of observability information from other services.

Internally it translates all metrics and logs to Elastic ECS, which is the standard schema used by Elasticsearch

To work properly it requires:

• Port bindings:
  • 2020/tcp: Used to expose internal HTTP server
  • 24224/tcp: Used to receive Logs from Docker Fluentd Driver
  • 8125/udp: Used to receive Metrics/APM using extended StatsD datagrams (StatsD + Tags)
• Volume bindings:
  • /var/run/docker.sock: Used to call Docker engine API
  • /var/lib/docker/containers: Used to detect all container running in the host
  • /proc: Used to extract host metrics (cpu, memory and load)

The service can be configured based on a set of environment variables that basically define limits for the Fluent Bit engine.

These variables can be easily stored as Environment File and exported as Docker Configs

The supported variables are:

• APM
• Docker Container Info
• Docker Container Stats
• Docker Container Uptime
• Docker Logs
• Docker System
• Host Metrics

The project provides some Kibana dashboards, equivalent to the original ones provided by Elastic Beats.

• Details about provided dashboards

The project contains a test stack, it requires properly configuration of Elasticsearch secrets by creating 2 files.

• /test/secrets/http_host.txt: contains the ES hostname
• /test./secrets/http_host.txt: contains the ES password

Once configured the test stack can be started:

./test.sh

To build Fluent-Beats, run:

./build.sh

You can optionally dive into a Fluent-Beats container using:

docker pull ghcr.io/fluent-beats/fluent-beats:latest
docker run --rm -ti --entrypoint sh ghcr.io/fluent-beats/fluent-beats

• By default we use a bare minimal set of the standard Fluent Bit plugins/features
• The Fluent Bit version used by this project is 1.8.11
  • In order to apply Docker Secrets extraction, the Fluent Bit Docker image version must include shell support (debug flavor).
• Versions up to 1.8.11 are shipped as much bigger Docker images and don't provide any useful feature for this particular solution

Fluent Beats logo was adapted from hati-royani hummingbird icon logo