geerlingguy / ansible-role-fluentd Goto Github PK

View Code? Open in Web Editor NEW

34.0 3.0 29.0 21 KB

Ansible role - Fluentd (td-agent)

Home Page: https://galaxy.ansible.com/geerlingguy/fluentd

License: MIT License

Jinja 100.00%

ansible fluentd logging monitoring efk ansible-role containers

ansible-role-fluentd's Introduction

Ansible Role: Fluentd

An Ansible Role that installs Fluentd on RedHat/CentOS or Debian/Ubuntu. This role installs td-agent, which is a standalone version that doesn't require Ruby to be installed on the system separately. See differences between td-agent and Fluentd here.

Requirements

N/A

Role Variables

Available variables are listed below, along with default values (see defaults/main.yml):

fluentd_version: 3

The td-agent version to install. See more details about the differences between v2, v3, and v4.

fluentd_package_state: present

The td-agent Fluentd package state; set to latest to upgrade or change versions.

fluentd_service_name: td-agent
fluentd_service_state: started
fluentd_service_enabled: true

Controls the Fluentd service options.

fluentd_plugins:
  - fluent-plugin-elasticsearch

# Alternative format:
fluentd_plugins:
  - name: fluent-plugin-elasticsearch
    version: '4.0.6'
    state: present

A list of Fluentd plugins to install.

fluentd_conf_sources: |
  [see defaults/main.yml for default content]

fluentd_conf_filters: |
  [see defaults/main.yml for default content]

fluentd_conf_matches: |
  [see defaults/main.yml for default content]

The configuration which will be placed into the td-agent.conf file which controls how Fluentd listens for, filters, and routes log data. The defaults set up some basic options which can direct data to Treasure Data, but you should override these values with what's appropriate for your logs.

For example, if you want to monitor an Apache HTTP server's access log, you would add a source:

fluentd_conf_sources: |
  <source>
    @type tail
    @id input_tail
    <parse>
      @type apache2
    </parse>
    path /var/log/httpd-access.log
    tag apache.access
  </source>

And then you could route apache log entries to Elasticsearch using:

fluentd_conf_matches: |
  <match apache.**>
    @type elasticsearch
    host log.example.com
    port 9200
    user myuser
    password mypassword
    logstash_format true
  </match>

Note that Elasticsearch would require the Fluentd plugin fluent-plugin-elasticsearch to be installed.

Dependencies

N/A

Example Playbook

- hosts: search

  vars:
    fluentd_conf_sources: |
      <source>
        @type tail
        @id input_tail
        <parse>
          @type apache2
        </parse>
        path /var/log/httpd-access.log
        tag apache.access
      </source>

  roles:
    - geerlingguy.fluentd

License

MIT / BSD

Author Information

This role was created in 2020 by Jeff Geerling, author of Ansible for DevOps.

ansible-role-fluentd's People

Stargazers

Watchers

ansible-role-fluentd's Issues

Error when comparing fluentd_version variable

If fluentd_version variable is defined programatically depending on something else. For example:

fluentd_version: "{{ (3 if ansible_os_family == 'RedHat' and ansible_distribution_major_version == '7' else 4) | int }}"

It's always casted as string, making the following check in tasks/main.yml to fail and always consider it as >=4

- name: Determine fluent-gem executable location for td-agent < v4
  set_fact:
    fluent_gem_executable: /opt/td-agent/embedded/bin/fluent-gem
  when: fluentd_version < 4

- name: Determine fluent-gem executable location for td-agent v4
  set_fact:
    fluent_gem_executable: /opt/td-agent/bin/fluent-gem
  when: fluentd_version >= 4

However the installation is done as version 3, raising the following error:

TASK [geerlingguy.fluentd : Determine fluent-gem executable location for td-agent < v4] *************************************************************************************************************************
skipping: [dc1-elastic1]
skipping: [dc1-elastic2]

TASK [geerlingguy.fluentd : Determine fluent-gem executable location for td-agent v4] ***************************************************************************************************************************
ok: [dc1-elastic1]
ok: [dc1-elastic2]

TASK [geerlingguy.fluentd : Ensure Fluentd plugins are installed.] **********************************************************************************************************************************************
failed: [dc1-elastic2] (item=fluent-plugin-elasticsearch) => {"ansible_loop_var": "item", "changed": false, "cmd": "/opt/td-agent/bin/fluent-gem query -n '^fluent-plugin-elasticsearch$'", "item": "fluent-plugin-elasticsearch", "msg": "[Errno 2] No such file or directory", "rc": 2}
failed: [dc1-elastic1] (item=fluent-plugin-elasticsearch) => {"ansible_loop_var": "item", "changed": false, "cmd": "/opt/td-agent/bin/fluent-gem query -n '^fluent-plugin-elasticsearch$'", "item": "fluent-plugin-elasticsearch", "msg": "[Errno 2] No such file or directory", "rc": 2}

I'd say it could be changed for the following to force the casting to int before making the comparison:

- name: Determine fluent-gem executable location for td-agent < v4
  set_fact:
    fluent_gem_executable: /opt/td-agent/embedded/bin/fluent-gem
  when: (fluentd_version | int) < 4

- name: Determine fluent-gem executable location for td-agent v4
  set_fact:
    fluent_gem_executable: /opt/td-agent/bin/fluent-gem
  when: (fluentd_version | int) >= 4

<source> @type forward port 24224 not working

Hi @geerlingguy,
I am trying to setup Fluentd Collector (aggregator) on a server, but :

  <source>
    @type forward
    @id input_forward
    port 24224

not working

Playbook:

hosts: linux
become: yes
vars:
fluentd_version: 4
fluentd_conf_sources: |

@type http
@id input_http
port 8888

@type forward
@id input_forward
port 24224
bind 0.0.0.0

...
pre_tasks:
- name: Allow Fluentd port
  community.general.ufw:
  rule: allow
  port: 24224
  proto: tcp
  roles:
- geerlingguy.fluentd

Azure Terraform VM:
resource "azurerm_linux_virtual_machine" "infra_fluentd-01" {
...
source_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-focal"
sku = "20_04-lts-gen2"
version = "latest"
}
}
...
resource "azurerm_network_security_group" "bpa_infra_vsg_fluentd-01" {
...
security_rule {
name = "fluentdrule"
priority = 1003
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = ""
destination_port_range = "24224"
source_address_prefix = ""
destination_address_prefix = "*"
}
}

means port 24224 is allowed in Ubuntu 20.04 ufw and Azure NSG

VM got ip = 172.23.17.41
ping & ssh port are open

dry run is ok:
cd /opt/td-agent/bin
sudo ./fluentd --dry-run -c /etc/td-agent/td-agent.conf

td-agent service is ok:
systemctl status td-agent
● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
Loaded: loaded (/lib/systemd/system/td-agent.service; enabled; vendor preset: enabled)
Active: active (running)

but
telnet 172.23.17.41 24224
Trying 172.23.17.41...
telnet: connect to address 172.23.17.41: Connection refused

either from laptop or from other VMs in same Azure subnet

sudo lsof -i:24224
not showing that td-agent uses 24224 port

'<' not supported between instances of 'AnsibleUnsafeText' and 'int'

Hi there,

I've come across the following error when deploying:

TASK [geerlingguy.fluentd : Determine fluent-gem executable location for td-agent < v4] *************************************************************************************************************************
fatal: [dc2-frontend1]: FAILED! => {"msg": "The conditional check 'fluentd_version < 4' failed. The error was: Unexpected templating type error occurred on ({% if fluentd_version < 4 %} True {% else %} False {% endif %}): '<' not supported between instances of 'AnsibleUnsafeText' and 'int'\n\nThe error appears to be in '/home/jmsierra/.ansible/roles/geerlingguy.fluentd/tasks/main.yml': line 22, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Determine fluent-gem executable location for td-agent < v4\n  ^ here\n"}
fatal: [dc2-frontend2]: FAILED! => {"msg": "The conditional check 'fluentd_version < 4' failed. The error was: Unexpected templating type error occurred on ({% if fluentd_version < 4 %} True {% else %} False {% endif %}): '<' not supported between instances of 'AnsibleUnsafeText' and 'int'\n\nThe error appears to be in '/home/jmsierra/.ansible/roles/geerlingguy.fluentd/tasks/main.yml': line 22, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Determine fluent-gem executable location for td-agent < v4\n  ^ here\n"}

Ansible environment:

jmsierra@*******:~/workspace/devcycle-ansible/****$ ansible --version
ansible [core 2.11.1] 
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/jmsierra/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jmsierra/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/jmsierra/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jmsierra/.local/bin/ansible
  python version = 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
  jinja version = 2.10.1
  libyaml = True

I believe it can be fixed by adding an explicit casting to int in tasks/main.yml:

- name: Determine fluent-gem executable location for td-agent < v4
  set_fact:
    fluent_gem_executable: /opt/td-agent/embedded/bin/fluent-gem
  when: (fluentd_version | int) < 4

- name: Determine fluent-gem executable location for td-agent v4
  set_fact:
    fluent_gem_executable: /opt/td-agent/bin/fluent-gem
  when: (fluentd_version | int) >= 4

PS: Thank you so much for your amazing contributions!

Error "apt cache update failed" in Ubuntu 20.04 LTS ARM

Hi there! I'm trying to use this package with my Ansible provisioning pipeline. Target server is running Ubuntu 20.04 LTS on a ARM Amazon server.

Provision log

PLAY [Provision machines] *****************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************
ok: [x.x.x.x]

TASK [geerlingguy.fluentd : include_tasks] ************************************************************************************************************************
skipping: [x.x.x.x]

TASK [geerlingguy.fluentd : include_tasks] ************************************************************************************************************************
included: /root/.ansible/roles/geerlingguy.fluentd/tasks/setup-Debian.yml for x.x.x.x

TASK [geerlingguy.fluentd : Add required dependencies.] ***********************************************************************************************************
ok: [x.x.x.x]

TASK [geerlingguy.fluentd : Add td-agent apt key.] ****************************************************************************************************************
ok: [x.x.x.x]

TASK [geerlingguy.fluentd : Add td-agent repository.] *************************************************************************************************************
fatal: [x.x.x.x]: FAILED! => {"changed": false, "msg": "apt cache update failed"}

PLAY RECAP ********************************************************************************************************************************************************
x.x.x.x              : ok=4    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0

Steps to reproduce

Add geerlingguy.fluentd to requirements, run install command.
Add geerlingguy.fluentd role to role array.
Run provision.

Custom variables

fluentd_plugins:
  - fluent-plugin-newrelic

fluentd_conf_sources: |
  <source>
    @type tail
    <parse>
      @type none
    </parse>
    path /var/log/apache2/error.log
    tag apache.error
  </source>
  <source>
    @type tail
    <parse>
      @type none
    </parse>
    path /var/log/apache2/example.com/error.log
    tag apache.vhost.example.error
  </source>

fluentd_conf_filters: |
  <filter apache.vhost.*>
    @type record_transformer
    <record>
      service_name ${tag}
      hostname "#{Socket.gethostname}"
    </record>
  </filter>

fluentd_conf_matches: |
  <match **>
    @type newrelic
    license_key ...
    base_uri https://log-api.eu.newrelic.com/log/v1
  </match>

No package matching 'gnupg2' is available

I have an error when install fluentd:

TASK [geerlingguy.fluentd : Add required dependencies.] ***********************************************************************************************************************************************************
fatal: [54.93.207.197]: FAILED! => {"changed": false, "msg": "No package matching 'gnupg2' is available"}

It was resolved with pre_task:

- hosts: fluentd
  become: yes
  pre_tasks:
    - name: Update repositories cache
      apt:
        update_cache: yes
  roles:
    - role: geerlingguy.fluentd
      tags:
        - fluentd

Unable to overwrite yum repository template

I noticed that this role works perfectly for Amazon Linux 2 if I could just overwrite the yum repository template. However for now this is not possible as the template name is hardcoded in the task.

Following repo template works for amazon linux 2

[treasuredata]
name=TreasureData
baseurl=http://packages.treasuredata.com/3/amazon/2/\$releasever/\$basearch
gpgcheck=1
gpgkey=https://packages.treasuredata.com/GPG-KEY-td-agent

CI failing on Ubuntu/Debian

yamllint .
ansible-lint
./tasks/setup-Debian.yml
  16:121    warning  line too long (156 > 120 characters)  (line-length)

./.github/stale.yml
  4:1       warning  missing document start "---"  (document-start)
  36:121    warning  line too long (185 > 120 characters)  (line-length)
  38:121    warning  line too long (177 > 120 characters)  (line-length)
  44:121    warning  line too long (161 > 120 characters)  (line-length)
  48:121    warning  line too long (181 > 120 characters)  (line-length)
  50:121    warning  line too long (170 > 120 characters)  (line-length)
  56:121    warning  line too long (150 > 120 characters)  (line-length)

Fluentd filters should go before matches

Currently if something matches (like solr.**), then you try to add a filter (on solr.**), it is placed below the match, so it is never evaluated. The order needs to be flipped.