patrickjahns / ansible-role-promtail Goto Github PK

See also: robertdebock/galaxy-action#12

Information: https://forum.ansible.com/t/failed-role-import-to-galaxy-ng/1534

Im trying to set up a simple example of deploying Promtail and running into a weird problem where the generated YAML scrape configuration isn't written out like my playbook describes. I cant think of why this would happen, hoping for an idea or two on how to get around this.

Here's my playbook:

On my target machine, this is the resulting /etc/promtail/promtail.yml

Notice how Jobname and Entry Parser are reversed? Promtail fails to start with this error after running the playbook. Labels and targets are generated backwards as well.

Hello!
I try use ansible-role-promtail for Docker Container Logging using Promtail

I try genarate config from https://gist.github.com/ruanbekker/c6fa9bc6882e6f324b4319c5e3622460

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: containers
  entry_parser: raw

  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  # --log-opt tag="{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"
  pipeline_stages:

  - json:
      expressions:
        stream: stream
        attrs: attrs
        tag: attrs.tag

  - regex:
      expression: (?P<image_name>(?:[^|]*[^|])).(?P<container_name>(?:[^|]*[^|])).(?P<image_id>(?:[^|]*[^|])).(?P<container_id>(?:[^|]*[^|]))
      source: "tag"

  - labels:
      tag:
      stream:
      image_name:
      container_name:
      image_id:
      container_id:

How configure pipeline_stages in this role?

This is a proposal. Using the code below it is possible to give the promtail user read access to /var/log regardless of the mode/owner. Is this something desirable for this role? Should it be configurable which paths get this access?

- name: Set ACL for promtail on /var/log
  acl:
    path: /var/log
    entity: promtail
    rights: rX
    recursive: yes
    state: present

There are few files related to ansible-lint (.yamllint and .ansible-lint), but is it really used? If so, what is the exact procedure to run it? Is tox somehow related to that?

It should be possible to change the maximum of allown open files for the systemd promtail service.

It can be achieved by add the LimitNOFILE parameter to the [Service] section.

The name of the variable could be something like promtail_limit_open_files.

Example:

[Service]
Restart=always
RestartSec=5
TimeoutSec=5
LimitNOFILE={{ promtail_limit_open_files }}
User={{ promtail_system_user }}
Group={{ promtail_system_group }}
ExecStart=/usr/local/bin/promtail -config.file={{ promtail_config_file }} -log.level={{ promtail_log_level }}

The release pipeline uses some old ways of passing information which will be deprecated in June 2023. See https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

ToDo:

• update https://github.com/patrickjahns/ansible-role-promtail/blob/master/.github/workflows/release.yml#L30-L38
• replace/update avakar/tag-and-release@v1 as it seems unmaintained

When I try to call the roll with `promtail_version: "latest", the installation task fails with the following:

TASK [patrickjahns.promtail : Assert usage of systemd as an init system] *************************************************************************************
ok: [host] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [patrickjahns.promtail : Get latest release] ************************************************************************************************************
fatal: [host]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

I'm fairly new to ansible so unfortunately I can't really say what exactly is going wrong/how to retrieve more information.
Specifying the version directly works.

Currently the role hardcodes amd64 - but this is not needed and can easily be extended to also include arm architecture

Sometimes (and in the future) it becomes necessary to add options that the role creator did not provide
For example, I need override backoff_config. But I don't find any global level variable.
Other people have also mentioned limits_config, etc.

In defaults/main.yml variable promtail_config_expand_env have a string type. And condition in templates/service.j2 have no bool filter. As result with the default vars option -config.expand-env will be true. When promtail_config_expand_env is false without quotes, option -config.expand-env will be false.

Also templating -config.expand-env={{ "true" if promtail_config_expand_env else "false" }} can be simplified to -config.expand-env={{ promtail_config_expand_env | string }}

ciao, please note that i'm currently in the process to get acquainted with loki and promtail. i was just wondering why the default for promtail_config_positions is an empty mapping. the application's default is /var/log/positions.yml which it can't write to when deployed with this role.

Due to promtail not having support for journald (grafana/loki#1459). I have to build the binary locally and upload to a file server. This means that the checksum part fails when changing promtail_dist_url.

To get around this I added a variable called promtail_check_checksum and replaced

I don't know if this is something worth adding as a PR or if I should just leave the role edited locally?

Is there any reason to use run_once in the following task:

The reason I am asking is, it causes Ansible to issue a warning when strategy: free is used:

...
TASK [patrickjahns.promtail : Assert usage of systemd as an init system] ***********************************************************************************************************
ok: [myserver] => changed=false 
  msg: All assertions passed
[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list.
...

Version 1.10.0 did not complete it's CI/CD pipeline and therefore it has not been successfully pushed to Galaxy. The commit reference seems to need [RELEASE], but that is there.

From our CI:

specified version (1.10.0) of patrickjahns.promtail was not found in the list
of available versions ([{'id': 115455, 'url': '', 'related': {},

The Ansible Galaxy release still shows only the 1.9.1 version - which is what we were using previously.

Update:

From looking at the Github actions that are carried out to manage releases there's a bit of regex foo:

Run VERSION=$(echo "${COMMIT_MSG}" | grep -Po '\d.\d.\d')

And the current ${COMMIT_MSG}:

  env:
    COMMIT_MSG: Merge pull request #52 from patrickjahns/release/next
  
  [RELEASE] 1.10.0

Since we're now on double digits, we are unable to match via this regex. The alternative would be: grep -Po '\d.\d\d.\d').

To fix this, an grep OR in the regex could work? This would mean it would read:

Run VERSION=$(echo "${COMMIT_MSG}" | grep -Po '(\d.\d.\d)|(\d.\d\d.\d)'

Example is here: https://regex101.com/r/WouePL/1

Happy to try to raise a PR for this if you think this is suitable.

Hi!

I just found the latest tag not to be working. Specifying the release version instead of using latest, works without any issue. Might be related to closed issue #65?

TASK [patrickjahns.promtail : Get latest release] -> works
TASK [patrickjahns.promtail : Set promtail version to 2.4.1] -> works
TASK [patrickjahns.promtail : Get checksum list] -> fails:
task path: /home/user/.ansible/roles/patrickjahns.promtail/tasks/preflight.yml:29 fatal: [192.168.1.34]: FAILED! => { "msg": "An unhandled exception occurred while running the lookup plugin 'url'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Received HTTP error for https://github.com/grafana/loki/releases/download/vlatest/SHA256SUMS : HTTP Error 404: Not Found" }

To me it seems that there is no SHA256SUMS file with the latest tag, only specified versions (e.g. https://github.com/grafana/loki/releases/download/v2.4.1/SHA256SUMS).

The correct URL is: https://github.com/grafana/loki/releases/latest/download/SHA256SUMS (as per the docs: https://docs.github.com/en/repositories/releasing-projects-on-github/linking-to-releases#linking-to-the-latest-release)

According to https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html#running-on-a-single-machine-with-run-once run_once means the task runs only on first node in batch. This fails when this role is included using e.g.

    - role: ansible-role-promtail
      tags: promtail
      when: promtail_enabled

when not all nodes have promtail_enabled: True.

e.g., I'd like to use "-print-config-stderr", plus a few others, but can't with this role; this variable can be a list, and if it's defined, they can be looped over when templating the systemd unit file.

The variable could be "promtail_additional_arguments" or "promtail_additional_runtime_flags" and default to []

I consider this a very important feature, so the user isn't limited by what is defined in the role. Thanks, great role!

Hello,

I'm trying to parse template with to_nice_yaml, and my config are :

Variable content :

opromtailScrapeConfig:
      - job_name: system
        static_configs:
        - targets:
            - localhost
          labels:
            job: messages
            __path__: /var/log/syslog
      - job_name: syslog
        syslog:
          listen_address: 0.0.0.0:1514
          idle_timeout: 60s
          label_structured_data: yes
          labels:
            job: "syslog-ng"
          use_incoming_timestamp: true
        relabel_configs:
          - source_labels: ['__syslog_message_hostname']
            target_label: 'host'

Template config :

scrape_configs:
  {% if opromtailScrapeConfig is defined %}
  {{ opromtailScrapeConfig | to_nice_yaml(indent=2) | indent(2, False) }}
  {% endif %}

{% if opromtailTargetConfig is defined %}
target_config:
  {{ opromtailTargetConfig | to_nice_yaml(indent=2) | indent(2, False) }}
{% endif %}

My output is :

scrape_configs:
    - job_name: system
    static_configs:
    - labels:
        __path__: /var/log/syslog
        job: messages
      targets:
      - localhost
  - job_name: syslog
    relabel_configs:
    - source_labels:
      - __syslog_message_hostname
      target_label: host
    syslog:
      idle_timeout: 60s
      label_structured_data: true
      labels:
        job: syslog-ng
      listen_address: 0.0.0.0:1514
      use_incoming_timestamp: true

Expected behavior :

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'

Seems to befilter's issue :(

Any idea ?

In order to keep up-to-date with upstream changes, lets add a sync script / bot that will check for newer upstream releases and then create a pull request with the according version bump

I'm trying to use the tags to only perform promtail installation, without having the defined systemd service enabled or started
(using promtail_install tags).

The problem is, that the tasks responsible for defining the systemd service and promtail config (that are tagged by promtail_install) notify a handler, that enables and (re)starts the service (regardless of being tagged).

Looks like this is a current ansible limitation.
The workaround might be to use when clauses around the tasks and handler that would check some variable, e.g. promtail_service_start

my use case:
I'm including this role in a play, that creates an OSP instance image. I need the promtail to be ready but not started/enabled, as users will pass their custom enable_promtail flag together with their own scrape configs.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency setuptools to v69.5.1
• chore(deps): update dependency ansible-compat to v24
• chore(deps): update dependency molecule to v24
• chore(deps): update dependency setuptools to v70
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

First of all, this ansible role rocks! But I needed loki in addition to promtail, and there is a lot of common stuff.

I started a fork, but maybe you would be receptive to a PR. If you don't want anybody to mess with this repo and/or think that a loki installation should be in a standalone repo, I respect that and no need to keep reading :)

My main idea is to only share certain key shared configuration things (such as system_user and system_group, maybe config_dir and tmp_dir, probably install_dir) to a loki_ prefix of variables. Things like version can be kept separated but defaulted like this:

loki_version: "2.2.1"
promtail_version: "{{ loki_version }}"

(I can conceive the need to have different loki version and promtail version, and it may avoid unnecessary breaking changes for a lot of users, maybe?)

With some flags named loki_install and promtail_install the user can choose whether to install only promtail, only loki, or both. Loki and promtail are distributed under the same releases page (on grafana/loki github) so they are quite "a pack". Still, users may want to deploy them separately.

Does it make sense? If you don't want it here but have some suggestions, I would be happy to get your feedback.

Travis has become quite cumbersome and slow - move the ansible molecule tests to github acitons so this is not taking forever to test when we perform minor changes on the role

Hi,

New to ansible and was trying to use this role. When I set promtail_version: latest in my vars the role errors out with the following.

Now in another run, I set ANSIBLE_DEBUG=true since it seemed like thats how I can enable no_log=false but I can't really make heads or tail what the error is in the very verbose output.
But ff I set that var to a number it works fine since it uses another task step to download the binary.

So my question is, any guidance on how to debug this better so I can fix the problem?

The latest loki 2.7.0 release does not contain the correct version information (see: grafana/loki#7669 )
Reenable the tests again, when the issue has been resolved upstream

I updated role from 1.18.0 to 1.19.1. Now role no longer works for me.

TASK [patrickjahns.promtail : Ensure unzip package is available] ************************************************************************************************************************************************
ok: [test]

TASK [patrickjahns.promtail : Create promtail group] ************************************************************************************************************************************************************
ok: [test]

TASK [patrickjahns.promtail : Create the promtail user] *********************************************************************************************************************************************************
fatal: [test]: FAILED! => changed=false
  msg: |-
    usermod: user promtail is currently used by process 546
  name: promtail
  rc: 8

PLAY RECAP ******************************************************************************************************************************************************************************************************
test                       : ok=27   changed=0    unreachable=0    failed=1    skipped=17   rescued=0    ignored=0

Ubuntu 22.04.1 LTS. Looks like it's caused by #138 reverting for now.

In order to track latency, grafana labs has developed canary. It would make sense to deploy canary together with this role
https://grafana.com/blog/2019/07/18/lokis-path-to-ga-loki-canary-early-detection-for-missing-logs/

In the step that ensure unzip is installed to the managed hosts you are setting update_cache: yes.

Would it be a reasonable ask to let administrators handle update_cache themselves, so apt-get update is only executed once per run, instead of possibly multiple times, due to this role forcing it?