Giter VIP home page Giter VIP logo

fluentd-docker-image's Introduction

Fluentd Docker Image

Build Status Docker Stars Docker Pulls

What is Fluentd?

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.

www.fluentd.org

Fluentd Logo

Supported tags and respective Dockerfile links

Current images (Edge)

These tags have image version postfix. This updates many places so we need feedback for improve/fix the images.

Current images use fluentd v1 series.

Old v1.4 images

This is for backward compatibility. Use "Current images" instead.

v0.12 images

Support of fluentd v0.12 has ended in 2019. We don't recommend v0.12 for new deployment.

You can use older versions via tag. See tag page on Docker Hub.

We recommend to use debian version for production because it uses jemalloc to mitigate memory fragmentation issue.

Using Kubernetes?

Check fluentd-kubernetes-daemonset images.

The detail of image tag

This image is based on the popular Alpine Linux project, available in the alpine official image, and Debian images.

For current images

edge

Latest released version of Fluentd.

vX.Y-A

Latest version of vX.Y Fluentd branch.

A will be incremented when image has major changes.

When fluentd version is updated, A is reset to 1.

vX.Y.Z-A.B

Concrete vX.Y.Z version of Fluentd. This tag is recommeded for the production environment.

A will be incremented when image has major changes. B will be incremented when image has small changes, e.g. library update or bug fixes.

When fluentd version is updated, A.B is reset to 1.0.

debian included tag

The image based on Debian Linux image. You may use this image when you require plugins which cannot be installed on Alpine (like fluent-plugin-systemd).

armhf included tag

The armhf images use ARM base images for use on devices such as Raspberry Pis.

Furthermore, the base images enable support for cross-platform builds using the cross-build tools from resin.io.

In order to build these images natively on ARM devices, the CROSS_BUILD_START and CROSS_BUILD_END Docker build arguments must be set to the shell no-op (:), for example:

docker build --build-arg CROSS_BUILD_START=":" --build-arg CROSS_BUILD_END=":" -t fluent/fluentd:v1.3-onbuild-1 v1.3/armhf/alpine-onbuild

(assuming the command is run from the root of this repository).

For older images

These images/tags are kept for backward compatibility. No update anymore and don't use for new deployment. Use "current images" instead.

stable, latest

These tags are obsolete, already removed to avoid confusing. Use edge, vX.Y-A or vX.Y.Z-A.B images instead.

vX.Y

Latest version of vX.Y Fluentd branch.

vX.Y.Z

Concrete vX.Y.Z version of Fluentd.

onbuild included tag

onbuild images are deprecated. Use non-onbuild images instead to build your image. New images, v1.5 or later, don't provide onbuild version.

debian included tag, armhf included tag

Same as current images.

How to use this image

To create endpoint that collects logs on your host just run:

docker run -d -p 24224:24224 -p 24224:24224/udp -v /data:/fluentd/log fluent/fluentd:v1.3-debian-1

Default configurations are to:

  • listen port 24224 for Fluentd forward protocol
  • store logs with tag docker.** into /fluentd/log/docker.*.log (and symlink docker.log)
  • store all other logs into /fluentd/log/data.*.log (and symlink data.log)

Providing your own configuration file and additional options

fluentd arguments can be appended to the docker run line

For example, to provide a bespoke config and make fluentd verbose, then:

docker run -ti --rm -v /path/to/dir:/fluentd/etc fluent/fluentd -c /fluentd/etc/<conf> -v

The first -v tells Docker to share '/path/to/dir' as a volume and mount it at /fluentd/etc The -c after the container name (fluentd) tells fluentd where to find the config file The second -v is passed to fluentd to tell it to be verbose

Change running user

Use -u option with docker run.

docker run -p 24224:24224 -u foo -v ...

How to build your own image

You can build a customized image based on Fluentd's image. Customized image can include plugins and fluent.conf file.

1. Create a working directory

We will use this directory to build a Docker image. Type following commands on a terminal to prepare a minimal project first:

# Create project directory.
mkdir custom-fluentd
cd custom-fluentd

# Download default fluent.conf and entrypoint.sh. This file will be copied to the new image.
# VERSION is v1.7 like fluentd version and OS is alpine or debian.
# Full example is https://raw.githubusercontent.com/fluent/fluentd-docker-image/master/v1.10/debian/fluent.conf

curl https://raw.githubusercontent.com/fluent/fluentd-docker-image/master/VERSION/OS/fluent.conf > fluent.conf

curl https://raw.githubusercontent.com/fluent/fluentd-docker-image/master/VERSION/OS/entrypoint.sh > entrypoint.sh
chmod +x entrypoint.sh

# Create plugins directory. plugins scripts put here will be copied to the new image.
mkdir plugins

curl https://raw.githubusercontent.com/fluent/fluentd-docker-image/master/Dockerfile.sample > Dockerfile

2. Customize fluent.conf

Documentation of fluent.conf is available at docs.fluentd.org.

3. Customize Dockerfile to install plugins (optional)

You can install Fluentd plugins using Dockerfile. Sample Dockerfile installs fluent-plugin-elasticsearch. To add plugins, edit Dockerfile as following:

3.1 For current images

Alpine version

FROM fluent/fluentd:v1.17-1

# Use root account to use apk
USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/

USER fluent

Debian version

FROM fluent/fluentd:v1.17-debian-1

# Use root account to use apt
USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/

USER fluent

Note

These example run apk add/apt-get install to be able to install Fluentd plugins which require native extensions (they are removed immediately after plugin installation). If you're sure that plugins don't include native extensions, you can omit it to make image build faster.

3.2 For older images

This section is for existing users. Don't recommend for new deployment.

Alpine version

FROM fluent/fluentd:v1.3-onbuild-1

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install \
        fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

Debian version

FROM fluent/fluentd:v1.3-debian-onbuild-1

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install \
        fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

4. Build image

Use docker build command to build the image. This example names the image as custom-fluentd:latest:

docker build -t custom-fluentd:latest ./

5. Test it

Once the image is built, it's ready to run. Following commands run Fluentd sharing ./log directory with the host machine:

mkdir -p log
docker run -it --rm --name custom-docker-fluent-logger -v $(pwd)/log:/fluentd/log custom-fluentd:latest

Open another terminal and type following command to inspect IP address. Fluentd is running on this IP address:

docker inspect -f '{{.NetworkSettings.IPAddress}}' custom-docker-fluent-logger

Let's try to use another docker container to send its logs to Fluentd.

docker run --log-driver=fluentd --log-opt tag="docker.{{.ID}}" --log-opt fluentd-address=FLUENTD.ADD.RE.SS:24224 python:alpine echo Hello
# and force flush buffered logs
docker kill -s USR1 custom-docker-fluent-logger

(replace FLUENTD.ADD.RE.SS with actual IP address you inspected at the previous step)

You will see some logs sent to Fluentd.

References

Docker Logging | fluentd.org

Fluentd logging driver - Docker Docs

Issues

We can't notice comments in the DockerHub so don't use them for reporting issue or asking question.

If you have any problems with or questions about this image, please contact us through a GitHub issue.

fluentd-docker-image's People

Contributors

ajayk avatar ashie avatar brandl avatar cosmo0920 avatar edsiper avatar elezar avatar frsyuki avatar fujimotos avatar ganmacs avatar ghatwala avatar guessi avatar jnewbigin avatar jokester avatar kenhys avatar lmm avatar m30m avatar mazerty avatar moleskin-smile avatar nokute78 avatar orgads avatar repeatedly avatar stevenjm avatar tagomoris avatar tcnksm avatar thakkaryash94 avatar tuki0918 avatar tyranron avatar valerianpereira avatar vyta avatar yanana avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fluentd-docker-image's Issues

Cannot get example to work.

I'm trying to run the example that is included in the README and I'm getting and issue with file is not writeable. I've included the output when running the command as well as the logs from the container after it fails.

root@Bubu:~/fluent# docker run -d -p 24224:24224 -v /data:/fluentd/log fluent/fluentd
8913bd8e5221293b8011c02d7825d5b5dd47ba6420355cf959ba15fa4ff88a5a
root@Bubu:~/fluent# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@Bubu:~/fluent# docker logs 8913
2016-04-27 20:01:51 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2016-04-27 20:01:51 +0000 [info]: starting fluentd-0.12.21
2016-04-27 20:01:51 +0000 [info]: gem 'fluentd' version '0.12.21'
2016-04-27 20:01:51 +0000 [info]: adding match in @mainstream pattern="docker.**" type="file"
2016-04-27 20:01:51 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="out_file: `/fluentd/log/docker.20160427_0.log` is not writable"
2016-04-27 20:01:51 +0000 [info]: process finished code=256
2016-04-27 20:01:51 +0000 [warn]: process died within 1 second. exit.

Keeping derived images upwards-compatible

Per default gem installs user-packages under /home/fluent/.gem/ruby/2.2.0/bin:$PATH
Note the ruby version number in there...

So as the user needs to add this to his path, this will likely break as soon as we switch to another ruby version. So how do we solve this?

Options:

  • Installing them as root. We could recommend the user to install them before he switches to as less-privileged user, like we do in our base-image. (Does this have any security-implications, or break stuff?)
  • Bash scripting (ugly, but works):
    for dir in $HOME/.gem/ruby/*; do [ -d "$dir/bin" ] && PATH="${dir}/bin:${PATH}" done
  • What the ruby docs say (http://guides.rubygems.org/faqs/#user-install)
    if which ruby >/dev/null && which gem >/dev/null; then PATH="$(ruby -rubygems -e 'puts Gem.user_dir')/bin:$PATH" fi
  • Using rvm/bundler/... for package managment (additional bloat)?

Unable to install package

Hi

I'm trying to build my own container on top of this one but I'm not able to install a package with apk
Is it expected ? Am I missing something ?

I'm getting this error

ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied

I'm sorry if this is a simple mistake, this is my first container based on Alpine, I'm not familiar with it yet
Thanks
Damien

Here is my Dockerfile

FROM fluent/fluentd:v0.12.21
MAINTAINER Damien Garros <[email protected]>

RUN apk --update add grep ruby-dev

And log

Sending build context to Docker daemon   108 kB
Step 1 : FROM fluent/fluentd:v0.12.21
# Executing 2 build triggers...
Step 1 : COPY fluent.conf /fluentd/etc/
 ---> Using cache
Step 1 : COPY plugins /fluentd/plugins/
 ---> Using cache
 ---> 2174b2d46cca
Step 2 : MAINTAINER Damien Garros <[email protected]>
 ---> Using cache
 ---> 4c67b3818839
Step 3 : RUN apk --update add grep ruby-dev
 ---> Running in 993a8205131d
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied

problem adding plugin to the image

I get this stack while trying to build the modified Dockerfile from the README to install new plugin

[root@manager2 fluentd]# docker build -t aa/fluentdaggregator .
Sending build context to Docker daemon 180.2 kB
Step 1 : FROM fluent/fluentd:latest-onbuild
# Executing 2 build triggers...
Step 1 : COPY fluent.conf /fluentd/etc/
 ---> Using cache
Step 1 : COPY plugins /fluentd/plugins/
 ---> Using cache
 ---> cf7bc09254a9
Step 2 : MAINTAINER YOUR_NAME <...@...>
 ---> Running in 1e2d0e21fdff
 ---> de2f1246900c
Removing intermediate container 1e2d0e21fdff
Step 3 : WORKDIR /home/fluent
 ---> Running in 4b6a30b4ac32
 ---> a99f8f4acbb5
Removing intermediate container 4b6a30b4ac32
Step 4 : ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
 ---> Running in 6a0651a5ede5
 ---> 294b972adb14
Removing intermediate container 6a0651a5ede5
Step 5 : USER root
 ---> Running in fa18f5ce97cf
 ---> 592201f0f7dc
Removing intermediate container fa18f5ce97cf
Step 6 : RUN apk --no-cache --update add sudo build-base ruby-dev &&     sudo -u fluent gem install fluent-plugin-elasticsearch fluent-plugin-record-reformer &&     rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem && sudo -ufluent gem sources -c &&     apk del sudo build-base ruby-dev &
& rm -rf /var/cache/apk/*
 ---> Running in 50436c80cd6f
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:
  build-base (missing):
    required by: world[build-base]
  ruby-dev (missing):
    required by: world[ruby-dev]
  sudo (missing):
    required by: world[sudo]
The command '/bin/sh -c apk --no-cache --update add sudo build-base ruby-dev &&     sudo -u fluent gem install fluent-plugin-elasticsearch fluent-plugin-record-reformer &&     rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem&& sudo -u fluent gem sources -c &&     apk del sudo build-base
 ruby-dev && rm -rf /var/cache/apk/*' returned anon-zero code: 3

here is the Dockerfile (exactly similar as the one in README)

FROM fluent/fluentd:latest-onbuild
MAINTAINER YOUR_NAME <...@...>
WORKDIR /home/fluent
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH

# cutomize following "gem install fluent-plugin-..." line as you wish

USER root
RUN apk --no-cache --update add sudo build-base ruby-dev && \

    sudo -u fluent gem install fluent-plugin-elasticsearch fluent-plugin-record-reformer && \

    rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem && sudo -u fluent gem sources -c && \
    apk del sudo build-base ruby-dev && rm -rf /var/cache/apk/*

EXPOSE 24284

USER fluent
CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

I created a plugins folder and pulled fluent-plugin-elasticsearch and fluent-plugin-record-reformer in it.

Why not running fluent as root?

I have the problem that fluent can't write the pos files of the tail plugin into the volume I mount for tailing. The reason is that fluentd doesn't run as root inside the container. Is there a reason for not running fluentd as root?

docker images v0.14.2 & v0.14.6 are broken

Hi,

I have issue with running docker images with version v0.14.2 & v0.14.6 but everything works fine with v0.14.1.

v0.14.2

docker run -p 24224:2224 fluent/fluentd:v0.14.2

but getting this error:

2016-09-14 14:46:32 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2016-09-14 14:46:32 +0000 [info]: starting fluentd-0.14.2
2016-09-14 14:46:32 +0000 [info]: spawn command to main: /usr/bin/ruby -Eascii-8bit:ascii-8bit /usr/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins --under-supervisor
2016-09-14 14:46:32 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2016-09-14 14:46:32 +0000 [info]: starting fluentd-0.14.2 without supervision
2016-09-14 14:46:32 +0000 [info]: gem 'fluentd' version '0.14.2'
2016-09-14 14:46:32 +0000 [info]: adding match in @mainstream pattern="docker.**" type="file"
2016-09-14 14:46:32 +0000 [info]: adding match in @mainstream pattern="**" type="file"
2016-09-14 14:46:32 +0000 [info]: adding filter pattern="**" type="stdout"
2016-09-14 14:46:32 +0000 [info]: adding source type="forward"
2016-09-14 14:46:32 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    @id input1
    @label @mainstream
    port 24224
  </source>
  <filter **>
    @type stdout
  </filter>
  <label @mainstream>
    <match docker.**>
      @type file
      @id output_docker1
      path "/fluentd/log/docker.*.log"
      symlink_path "/fluentd/log/docker.log"
      append true
      time_slice_format "%Y%m%d"
      time_slice_wait 1m
      time_format "%Y%m%dT%H%M%S%z"
      buffer_path /fluentd/log/docker.*.log
      <buffer time>
        flush_mode lazy
        retry_type exponential_backoff
        path /fluentd/log/docker.*.log
        timekey_wait 1m
        timekey 86400
      </buffer>
      <parse>
        time_format %Y%m%dT%H%M%S%z
      </parse>
      <inject>
        time_format %Y%m%dT%H%M%S%z
      </inject>
    </match>
    <match **>
      @type file
      @id output1
      path "/fluentd/log/data.*.log"
      symlink_path "/fluentd/log/data.log"
      append true
      time_slice_format "%Y%m%d"
      time_slice_wait 10m
      time_format "%Y%m%dT%H%M%S%z"
      buffer_path /fluentd/log/data.*.log
      <buffer time>
        flush_mode lazy
        retry_type exponential_backoff
        path /fluentd/log/data.*.log
        timekey_wait 10m
        timekey 86400
      </buffer>
      <parse>
        time_format %Y%m%dT%H%M%S%z
      </parse>
      <inject>
        time_format %Y%m%dT%H%M%S%z
      </inject>
    </match>
  </label>
</ROOT>
2016-09-14 14:46:32 +0000 [warn]: section <parse> is not used in <match docker.**> of out_file plugin
2016-09-14 14:46:32 +0000 [warn]: section <parse> is not used in <match **> of out_file plugin
2016-09-14 14:46:32 +0000 [error]: unexpected error error_class=NameError error="uninitialized constant Fluent::Compat::HandleTagAndTimeMixin\nDid you mean?  Fluent::Compat::HandleTagNameMixin\n               Fluent::HandleTagNameMixin"
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/compat/output.rb:655:in `start'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/root_agent.rb:138:in `block in start'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/root_agent.rb:117:in `block (2 levels) in lifecycle'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/agent.rb:125:in `block (2 levels) in lifecycle'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/agent.rb:124:in `each'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/agent.rb:124:in `block in lifecycle'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/agent.rb:117:in `each'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/agent.rb:117:in `lifecycle'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/root_agent.rb:116:in `block in lifecycle'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/root_agent.rb:113:in `each'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/root_agent.rb:113:in `lifecycle'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/root_agent.rb:137:in `start'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/engine.rb:211:in `start'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/engine.rb:175:in `run'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/supervisor.rb:624:in `run_engine'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/supervisor.rb:409:in `block in run_worker'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/supervisor.rb:553:in `main_process'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/supervisor.rb:404:in `run_worker'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/lib/fluent/command/fluentd.rb:271:in `<top (required)>'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-09-14 14:46:32 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.2/bin/fluentd:5:in `<top (required)>'
  2016-09-14 14:46:32 +0000 [error]: /usr/bin/fluentd:23:in `load'
  2016-09-14 14:46:32 +0000 [error]: /usr/bin/fluentd:23:in `<main>'
2016-09-14 14:46:32 +0000 [error]: unexpected error error="uninitialized constant Fluent::Compat::HandleTagAndTimeMixin\nDid you mean?  Fluent::Compat::HandleTagNameMixin\n               Fluent::HandleTagNameMixin"
  2016-09-14 14:46:32 +0000 [error]: suppressed same stacktrace
2016-09-14 14:46:33 +0000 [info]: Worker 0 finished unexpectedly with status 1

v0.14.6

docker run -p 24224:2224 fluent/fluentd:v0.14.6

but getting this error:

/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- json (LoadError)
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/config/literal_parser.rb:19:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/config/element.rb:18:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/config.rb:18:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:20:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/command/fluentd.rb:19:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.6/bin/fluentd:5:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'

symlink_path is absolute, so link is broken when accessed from host

When following the instructions in the readme, the symlink created by fluentd via symlink_path /fluentd/log/docker.log is broken when read from the host. If the symlink were relative it would work. It looks like this would require a change upstream, but reporting here since it's a problem primarily when using the docker image.

Building from custom Dockerfile fails with docker-compose

When trying to use a custom Dockerfile with docker-compose, I get the following error:

Building fluentd
Step 1 : FROM fluent/fluentd:latest-onbuild
# Executing 2 build triggers...
Step 1 : COPY fluent.conf /fluentd/etc/
ERROR: Service 'fluentd' failed to build: lstat fluent.conf: no such file or directory

I'm using the sample Dockerfile in repo:

FROM fluent/fluentd:latest-onbuild
MAINTAINER YOUR_NAME <...@...>
WORKDIR /home/fluent
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH

USER root
RUN apk --no-cache --update add sudo build-base ruby-dev && \

    sudo -u fluent gem install fluent-plugin-secure-forward && \

    rm -rf /home/fluent/.gem/ruby/2.3.0/cache/*.gem && sudo -u fluent gem sources -c && \
    apk del sudo build-base ruby-dev && rm -rf /var/cache/apk/*

EXPOSE 24284

USER fluent
CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

My docker-compose.yml entry for fluentd:

fluentd:
    build: ./docker/fluentd
    restart: always

UID of fluentd user

We already have users at UID 1000. Any way to make this not hardcoded as such?

Basically, I'm having trouble matching the UIDs of the users in and out of the container when I map /fluentd/log to the host. I get permission denied when launching the container:

[error]: config error file="/fluentd/etc/fluent.conf" error="out_file: `/fluentd/log/docker.20160609_0.log` is not writable"

where

docker run -d -p 24224:24224 -v /var/log/fluentd:/fluentd/log fleuntd/fluentd

and
rwxrwxrwx /var/log/fluentd

Inside the container

/fluentd $ lsd
total 20
drwxr-xr-x    5 fluent   fluent      4.0K Jun  8 00:19 ./
drwxr-xr-x   19 root     root        4.0K Jun  9 23:43 ../
drwxr-xr-x    2 fluent   fluent      4.0K Jun  8 00:09 etc/
drwxr-xr-x    2 root     root        4.0K Jun  9 22:32 log/
drwxr-xr-x    2 fluent   fluent      4.0K Jun  8 00:19 plugins/

Latest Image fails when extended

Very simple repro:

Using:

FROM fluent/fluentd:latest
USER root
RUN apk --no-cache --update add sudo

Results:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
Bus error

Replacing the base image with alpine:3.4 (same as fluent/fluentd:latest) fixes the problem -- you can even see it fetch the same file without an issue.

I did just rebuild from master here and it worked fine so perhaps it's the image pushed to Docker hub that just needs a cache refresh.

Cannot use on-build image

I am trying to use the on-build image to build a customized image of fluentd.

My Dockerfile is copied verbatim from the README:

FROM fluent/fluentd:latest-onbuild
MAINTAINER your_name <...>
USER fluent
WORKDIR /home/fluent
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
RUN gem install fluent-plugin-secure-forward
EXPOSE 24284
CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

When I run docker build ., I get:

Sending build context to Docker daemon 57.34 kB
Step 1 : FROM fluent/fluentd:latest-onbuild
# Executing 2 build triggers...
Step 1 : COPY fluent.conf /fluentd/etc/
lstat fluent.conf: no such file or directory

Docker is 1.11.2 running on Ubuntu 16.04 64-bit.

Status of using jemalloc?

Hi,

I've been seeing the fluentd forwarder agent consuming quite a lot of memory and as I run it in a Kubernetes environment, it's gets OOM killed and restarted (which is a good Kubernetes thing to do). Still, it looks like if we were using jemalloc, memory wouldn't be fragmented as much.

However, I see that it was reverted in 94691d0 and I don't quite understand the reason why.

What's the status of using jemalloc with this image today?

Thanks,

Permission problem when forwarding to aggregator

I am using the fluent container on each host and I have an aggregator container on one host (192.168.77.22). (of course the host having the aggregator will have to co-exist with a fluentd client)

each fluent container has this config

<source>
  type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.*>
  type stdout
</match>

<match *.*>
  type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 1s
  phi_threshold 16
  hard_timeout 60s
  <server>
    name manager2
    host 192.168.77.22
    port 24225
  </server>
</match>

the fluent aggregator has this config

<source>
  type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.*>
  type stdout
</match>

the trick is that I am starting the fluentd aggregator using this command:
I forward the port from 24225 outside the container to 24224 inside the container

docker run -p 24225:24224 -v /etc/fluentd/fluent_aggregator.conf:/fluentd/etc/fluent_aggregator.conf -e FLUENTD_CONFIG=fluentd_aggregator.comf fluent/fluentd:latest

This allows me to have the fluent aggregator coexisting with one fluent client on a host.

my fluentd client are receiving lots of logs but my aggregator is receiving none.
I connected inside the fluent aggregator container and try this:

~ $ ping 192.168.77.22
PING 192.168.77.22 (192.168.77.22): 56 data bytes
ping: permission denied (are you root?)

this is quite difficult to debug because of that.
I placed the aggregator and the client on the same network and I tried using the host alias like such:

  <server>
    name manager2
    host fluentdaggregator
    port 24225
  </server>

I could not test from inside the fluentd container but from another container I could do:

[root@manager0 vagrant]# docker exec -it telegraf sh
# ping fluentdaggregator
PING fluentdaggregator (10.0.0.6) 56(84) bytes of data.
From 29692e088307 (10.0.0.12) icmp_seq=1 Destination Host Unreachable
From 29692e088307 (10.0.0.12) icmp_seq=2 Destination Host Unreachable
From 29692e088307 (10.0.0.12) icmp_seq=3 Destination Host Unreachable
From 29692e088307 (10.0.0.12) icmp_seq=4 Destination Host Unreachable
--- fluentdaggregator ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4002ms

How can I debug this?
What is wrong with my code?

Confused as to versioning

If I pull the latest tag, and I pull v0.12.19 they are two different images. One is ubuntu based and one is alpine based (yay!). I like to use discrete versions in my Dockerfiles so there are no surprises when I build.

Feature request: Provide image based on something other than Alpine due to DNS issues

Alpine seems to not play well with Kubernetes. See https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md#dns and SumoLogic/fluentd-kubernetes-sumologic#8.

There have even been images built to get around this DNS issue like: https://github.com/janeczku/docker-alpine-kubernetes.

So it would be nice to create a tag with each release based on a different image than Alpine. E.g. One of these: https://hub.docker.com/_/buildpack-deps/

Minimise image footprint

I'm currently trying to setup containerized log aggregation on my coreos-cluster, unfortunately the docker pull of this container takes way longer than expected and most other services waiting for it to come online just time out.

I don't have any ideas how to resolve this in a clean fashion other than using a stripped down fluentd-image.

There are multiple ways to achieve this, the easiest and also most efficient would be not to use a full fledged ubuntu as base, but rather Alpine Linux:
https://github.com/gliderlabs/docker-alpine

I think most people expect this container to be lightweight and fast, just like fluent.

Ubuntu tag fails to build

Step 16 : RUN gem install fluentd -v 0.12.19
 ---> Running in d5029c9ae7bb
Building native extensions.  This could take a while...
Successfully installed cool.io-1.4.4
Building native extensions.  This could take a while...
Successfully installed http_parser.rb-0.6.0
Building native extensions.  This could take a while...
ERROR:  Error installing fluentd:
        string-scrub requires Ruby version < 2.1, >= 1.9.3.

Also seems like it's crazy out of date?

Failed Mounting Multiple Volume

Host Os: Windows
Docker Command :
docker run -p 24224:24224 -e FLUENTD_CONF=fluent.conf -v //c//Users//user//fluentd//logs://fluentd//log -v //c//Users//user//fluentd//config://fluentd//etc fluent/fluentd

Logs :

#.... blah blah blah and then fluentd config   
  <source>
type forward
port 24224
bind 0.0.0.0
  </source>
  <match *.*>
    type stdout
  </match>
</ROOT>
2016-07-01 08:38:43 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2016-07-01 08:38:43 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2016-07-01 08:38:43 +0000 [error]: unexpected error error_class=Errno::EADDRINUSE error=#<Errno::EADDRINUSE: Address in use - bind(2) for "0.0.0.0" port 24224>
  2016-07-01 08:38:43 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/server.rb:57:in `initialize'
  2016-07-01 08:38:43 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/server.rb:57:in `new'

if i only mount 1 volume it works well
can anyone help me to find out why fluentd tries to listen to the socket twice when i tries to mount 2 volume?

Log File Not writeable

@tagomoris @repeatedly Strangely enough I have created a directly with full RW permissions and it still errors out for me, can someone help pls?

screenshot 2016-02-08 06 59 30

The other ticket with the same name was closed hence why I craeted this.

gem install fails for Latest Image

[kujira:~/dockers/fluentd]$cat Dockerfile
FROM fluent/fluentd
RUN ["gem", "install", "fluent-plugin-forest", "--no-rdoc", "--no-ri"]

Step 3 : RUN gem install fluent-plugin-forest --no-rdoc --no-ri
 ---> Running in bb8243553dc8
/usr/lib/ruby/2.3.0/psych.rb:377: [BUG] Segmentation fault at 0x007f92b34008e8
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux-musl]

Dockerfile should chown directories as root

The way this Dockerfile is written makes it impossible, in practice, to use with Kubernetes, which mounts all volumes as root.

Since you specify USER to drop privileges, and then mkdir/chown the log directory inside the Dockerfile, it's too late to do anything with the volume by the time the container is run.

The fix โ€” and this applies to any Docker image โ€” is to start as root and to drop privileges in a script invoked by ENTRYPOINT.

Here is an example from the official PostgreSQL image (entrypoint script is here). That script uses gosu to drop privileges, but I prefer su-exec myself.

Another reason why dropping privileges with USER is a bad idea is that it makes it impossible to exec into the container as root, which in turn makes it impossible to do certain things (gdb, strace) inside the container.

Can't install plugin that need native extension with Alpine based image

I am trying to install fluent-plugin-flume in 0.12.21 alpine based image and receive this error:

mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

Digging in the dockerfile, I can see that after installing fluentd the necessary package to build native extension is removed:

apk del build-base ruby-dev

Is this a bug or is the intended use is to re-install those in my docker file and remove them after my plugin is installed like so (note it does require switching to root user momentarily though) ?

USER root
RUN apk --no-cache --update add \
                            build-base \
                            ruby-dev && \
    gem install fluent-plugin-flume && \
    apk del build-base ruby-dev && \
    rm -rf /tmp/* /var/tmp/* /var/cache/apk/*
USER fluent

I guess I can see why you remove those 2 packages as they are quite large but wanted to confirm.

Thanks

empty plugins directory breaks image builds

When creating my own image using FROM fluent/fluentd:latest the ONBUILD COPY cmd stops exectution at the copy command and breaks subsequent builds. Adding ant file to the directory forces correct behaviour.

$ docker build --force-rm --no-cache .
Sending build context to Docker daemon 86.53 kB
Step 0 : FROM fluent/fluentd:latest
# Executing 2 build triggers
Trigger 0, COPY fluent.conf /fluentd/etc/
Step 0 : COPY fluent.conf /fluentd/etc/
Trigger 1, COPY plugins/* /fluentd/plugins/
Step 0 : COPY plugins/* /fluentd/plugins/
Removing intermediate container 34e3b50362bf
No source files were specified

http://docs.docker.com/engine/reference/builder/#copy
perhaps ONBUILD COPY plugins/* /fluentd/plugins/ could become ONBUILD COPY plugins/ /fluentd/plugins/ ??

the image in docker hub is broken

it seems the official image in docker hub is broken !
[root@cloud /data 11:23:08]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
fluent/fluentd latest 6179c3420517 8 days ago 35.94 MB

Container exits with code 143 on SIGTERM

All recent versions of the Fluentd Docker image exit with code 143 when receiving a SIGTERM (i.e. docker stop, docker-compose stop, or docker-compose down). Is this expected behavior, or should containers be exiting with code 0 on SIGTERM?

Only fluent/fluentd:ubuntu-base exits with code 0. Coincidentally we do use the ubuntu image, but because we wrap our fluentd -v -c .... command in a shell script and use that as the container's CMD, we get exit code 137 when sending SIGTERM to our container. In addition, we are looking to move off of the ubuntu image as it's no longer supported.

$ docker ps -a

IMAGE                        COMMAND                  CREATED              STATUS
fluent/fluentd:ubuntu-base   "/bin/sh -c 'exec ..."   About a minute ago   Exited (0) 18 seconds ago
fluent/fluentd:edge-debian   "/bin/entrypoint.s..."   3 minutes ago        Exited (143) 3 minutes 
fluent/fluentd:edge          "/bin/entrypoint.s..."   3 minutes ago        Exited (143) 3 minutes
fluent/fluentd:debian        "/bin/entrypoint.s..."   4 minutes ago        Exited (143) 3 minutes
fluent/fluentd:latest        "/bin/entrypoint.s..."   4 minutes ago        Exited (143) 4 minutes

Log files are not being created when running fluentd as 1.12 swarm mode service

Fluentd container works totally fine when using "docker run" but I run into a problem when trying to run it as a 1.12 swarm mode service. Fluentd does not create any files in the mounted "log" directory when run as as service.

This works as expected:

docker run -v `pwd`/log:/fluentd/log -p 24224:24224 fluent/fluentd:v0.14.8

MacBook-Air:swate jzk$ ls -al log/
total 24
drwxr-xr-x   5 jzk  staff  170 23 Okt 18:22 .
drwxr-xr-x  14 jzk  staff  476 23 Okt 16:15 ..
-rw-r--r--   1 jzk  staff  200 23 Okt 18:22 docker.b53f8aae520c74f7fd631a67e3698f31f.log
-rw-r--r--   1 jzk  staff   68 23 Okt 18:22 docker.b53f8aae520c74f7fd631a67e3698f31f.log.meta
lrwxrwxrwx   1 jzk  staff   57 23 Okt 18:22 docker.log -> /fluentd/log/docker.b53f8aae520c74f7fd631a67e3698f31f.log

This does not create any log files:

docker service create --name logger --replicas 1 --mount type=bind,source=pwd/log,target=/fluentd/log -p 24224:24224 fluent/fluentd:v0.14.8

Fluentd starts up as service without any errors:

MacBook-Air:swate jzk$ docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                  PORTS                 NAMES
f21d0543ee1b        fluent/fluentd:v0.14.8   "/bin/sh -c 'exec flu"   4 seconds ago       Up Less than a second   5140/tcp, 24224/tcp   logger.1.2u5efbe4ihekrl6bmz4ah2o4w
MacBook-Air:swate jzk$ docker service ls
ID            NAME    REPLICAS  IMAGE                   COMMAND
4mefp54i3inv  logger  1/1       fluent/fluentd:v0.14.8

MacBook-Air:swate jzk$ docker logs f21d0543ee1b
2016-10-23 16:24:31 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2016-10-23 16:24:31 +0000 [info]: starting fluentd-0.14.8
2016-10-23 16:24:31 +0000 [info]: spawn command to main: /usr/bin/ruby -Eascii-8bit:ascii-8bit /usr/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins --under-supervisor
2016-10-23 16:24:31 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2016-10-23 16:24:31 +0000 [info]: starting fluentd-0.14.8 without supervision
2016-10-23 16:24:31 +0000 [info]: gem 'fluentd' version '0.14.8'
2016-10-23 16:24:31 +0000 [info]: adding match in @mainstream pattern="docker.**" type="file"
2016-10-23 16:24:31 +0000 [info]: adding match in @mainstream pattern="**" type="file"
2016-10-23 16:24:31 +0000 [info]: adding filter pattern="**" type="stdout"
2016-10-23 16:24:31 +0000 [info]: adding source type="forward"
2016-10-23 16:24:31 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    @id input1
    @label @mainstream
    port 24224
  </source>
  <filter **>
    @type stdout
  </filter>
  <label @mainstream>
    <match docker.**>
      @type file
      @id output_docker1
      path "/fluentd/log/docker.*.log"
      symlink_path "/fluentd/log/docker.log"
      append true
      time_slice_format "%Y%m%d"
      time_slice_wait 1m
      time_format "%Y%m%dT%H%M%S%z"
      buffer_path /fluentd/log/docker.*.log
      <buffer time>
        flush_mode lazy
        retry_type exponential_backoff
        path /fluentd/log/docker.*.log
        timekey_wait 1m
        timekey 86400
      </buffer>
      <parse>
        time_format %Y%m%dT%H%M%S%z
      </parse>
      <inject>
        time_format %Y%m%dT%H%M%S%z
      </inject>
    </match>
    <match **>
      @type file
      @id output1
      path "/fluentd/log/data.*.log"
      symlink_path "/fluentd/log/data.log"
      append true
      time_slice_format "%Y%m%d"
      time_slice_wait 10m
      time_format "%Y%m%dT%H%M%S%z"
      buffer_path /fluentd/log/data.*.log
      <buffer time>
        flush_mode lazy
        retry_type exponential_backoff
        path /fluentd/log/data.*.log
        timekey_wait 10m
        timekey 86400
      </buffer>
      <parse>
        time_format %Y%m%dT%H%M%S%z
      </parse>
      <inject>
        time_format %Y%m%dT%H%M%S%z
      </inject>
    </match>
  </label>
</ROOT>
2016-10-23 16:24:31 +0000 [warn]: section <parse> is not used in <match docker.**> of out_file plugin
2016-10-23 16:24:31 +0000 [warn]: section <parse> is not used in <match **> of out_file plugin
2016-10-23 16:24:31 +0000 [info]: listening fluent socket on 0.0.0.0:24224

Both variants "run" & "service" is tested with the following:

docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 python:alpine echo Hello

My docker (docker for mac) environment:

MacBook-Air:swate jzk$ docker -v
Docker version 1.12.1, build 6f9534c

I assume it has something to do with the new "mount" functionality of services in 1.12 vs the "-v" option of the docker run.

Any help would be really appreciated.

Fluent is no longer a default recognised user

Hi, My code was working fine till 12th May 2017 but from 15th May 2017, I started getting this error:

Step 5/8 : RUN gem install fluent-plugin-elasticsearch
---> Running in 5fa0ac7253e1
ERROR: Service 'fluentd' failed to build: linux spec user: unable to find user fluent: no matching entries in passwd file

My fluentd docker file is:

FROM fluent/fluentd:latest-onbuild
USER fluent
WORKDIR /home/fluent
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
RUN gem install fluent-plugin-elasticsearch
RUN gem install fluent-plugin-secure-forward
EXPOSE 24224
CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

Any reason why it can no longer identify fluent as a valid user and what can be done to fix this?

Customized dockerfile cannot work

Hi,

I am creating a customized dockerfile for installing plugins like elasticsearch
here is my dockerfile

FROM fluent/fluentd:ubuntu-base

USER ubuntu
WORKDIR /home/ubuntu
ENV PATH /home/ubuntu/.gem/ruby/2.2.0/bin:$PATH

# install plugins
RUN gem install fluent-plugin-secure-forward && \
    gem install fluent-plugin-elasticsearch

EXPOSE 24224

CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

However, this dockerfile cannot work for me. there is no data in /fluentd/logs/*

ubuntu@3527aba5df6d:~$ ls -al /fluentd/log/*
ls: cannot access /fluentd/log/*: No such file or directory

but if I modify base dockerfile directly (by just adding gem install commands and REMOVE onbuild commands), it works.

FROM ubuntu:14.04
MAINTAINER TAGOMORI Satoshi <[email protected]>
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.0"

RUN apt-get update -y && apt-get install -y \
              autoconf \
              bison \
              build-essential \
              curl \      
              git \
              libffi-dev \              
              libgdbm3 \
              libgdbm-dev \
              libncurses5-dev \
              libreadline6-dev \              
              libssl-dev \
              libyaml-dev \
              zlib1g-dev \              
        && rm -rf /var/lib/apt/lists/*

RUN useradd ubuntu -d /home/ubuntu -m -U
RUN chown -R ubuntu:ubuntu /home/ubuntu

# for log storage (maybe shared with host)
RUN mkdir -p /fluentd/log
# configuration/plugins path (default: copied from .)
RUN mkdir -p /fluentd/etc
RUN mkdir -p /fluentd/plugins

RUN chown -R ubuntu:ubuntu /fluentd

USER ubuntu
WORKDIR /home/ubuntu

RUN git clone https://github.com/tagomoris/xbuild.git /home/ubuntu/.xbuild
RUN /home/ubuntu/.xbuild/ruby-install 2.2.2 /home/ubuntu/ruby

ENV PATH /home/ubuntu/ruby/bin:$PATH
RUN gem install fluentd -v 0.12.19

# install plugins
RUN gem install fluent-plugin-secure-forward && \
    gem install fluent-plugin-elasticsearch

COPY fluent.conf /fluentd/etc/
#######ONBUILD COPY fluent.conf /fluentd/etc/
#######ONBUILD COPY plugins /fluentd/plugins/

WORKDIR /home/ubuntu

ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

EXPOSE 24224

### docker run -p 24224 -v `pwd`/log: -v `pwd`/log:/home/ubuntu/log fluent/fluentd:latest
CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

seems we have an issue on ONBUILD commands.

Using ONBUILD in Dockerfile to provide a way to install plugins

I this this image could benefit from this feature since fluentd use plugins heavily.

Python use this functionality to read requirement.txt and install plugin at image build time using pip.
No need to rewrite part of the Dockerfile file:

ONBUILD COPY requirements.txt /usr/src/app/
ONBUILD RUN pip install --no-cache-dir -r requirements.txt

syslog input to retain source hostname

I am running fluentd as a docker container listening for 514/udp to aggregate syslog messages sent by bare metal machines, forwarding all to graylog2 via gelf/udp.

Now in Graylog2 all messages have the generated docker hostname of the fluentd container in the source field.
How can I retain the original hostname from which the syslog message has been sent?

<source>
 @type syslog
 port 514
 tag syslog
</source>

<match **>
    @type gelf
    host 192.168.1.15
    port 12201
    protocol udp
    flush_interval 2
</match>

much appreciated

Error response from daemon: Cannot start container [8] System error: not a directory

I'm hitting the following issue:

# docker run -it -p 24224:24224 -v /opt/fluent.conf:/fluentd/etc -e FLUENTD_CONF=fluent.conf --name="fluentd" fluent/fluentd:latest
Timestamp: 2015-08-18 18:27:30.223988366 +0100 IST
Code: System error

Message: not a directory

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@37

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@52

---
Error response from daemon: Cannot start container 6b35e175f2542c959f0b95557ea368f1a4eaad157a6a4a40ae49fed20c6c8e46: [8] System error: not a directory

Network access problem from image

I created an image using the example Dockerfile but added in an extra plugin - fluent-plugin-splunk.

This plugin errors out when running the container with a "Connection reset by peer".

As a troubleshooting step, I logged in to the container and i was able to ping my host. I then installed telnet and tried to telnet to my Splunk port - this failed.

I then started a standard centos container for testing and i was able to successfully telnet to my Splunk server port.

I then changed my Dockerfile to use the Debian fluent image and faced the same problem.

It seems strange that from other containers i can reach my Splunk server but from the fluentd images i am unable to connect (even though ping works).

Log file is not writable

Get the following when I try to start this container:

2015-07-21 21:27:36 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2015-07-21 21:27:36 +0000 [info]: starting fluentd-0.12.11
2015-07-21 21:27:36 +0000 [info]: gem 'fluentd' version '0.12.11'
2015-07-21 21:27:36 +0000 [info]: adding match in @Mainstream pattern="docker.**" type="file"
2015-07-21 21:27:36 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="out_file: /fluentd/log/docker.20150721_0.log is not writable"
2015-07-21 21:27:36 +0000 [info]: process finished code=256
2015-07-21 21:27:36 +0000 [warn]: process died within 1 second. exit.

Running as root and pretty sure I have the latest version:
docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
fluent/fluentd latest 212de73247af 6 weeks ago 491.4 MB

Can't build custom images

Can't build custom images. ๐Ÿ˜ข


My Dockerfile (by README)

FROM fluent/fluentd:latest
# MAINTAINER your_name <...>
USER fluent
WORKDIR /home/fluent
ENV PATH /home/fluent/ruby/bin:$PATH
RUN gem install fluent-plugin-secure-forward
EXPOSE 24284
CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

out:

$ docker build .
Sending build context to Docker daemon 13.82 kB
Step 1 : FROM fluent/fluentd:latest
# Executing 2 build triggers...
Step 1 : COPY fluent.conf /fluentd/etc/
Step 1 : COPY plugins /fluentd/plugins/
 ---> 3bff29583bc7
Removing intermediate container d217e6fa7948
Removing intermediate container 02ae73ca52bb
Step 2 : USER fluent
 ---> Running in 25eaac160014
 ---> d7a82ba6b76e
Removing intermediate container 25eaac160014
Step 3 : WORKDIR /home/fluent
 ---> Running in 0840f0fe527e
 ---> c779c0953c4d
Removing intermediate container 0840f0fe527e
Step 4 : ENV PATH /home/fluent/ruby/bin:$PATH
 ---> Running in c16476527348
 ---> a8f36ac6492b
Removing intermediate container c16476527348
Step 5 : RUN gem install fluent-plugin-secure-forward
 ---> Running in 8be70d171708
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /usr/lib/ruby/gems/2.2.0 directory.
The command '/bin/sh -c gem install fluent-plugin-secure-forward' returned a non-zero code: 1

Create custom Fluent Docker Image 0.12.24 -> 0.12.26

When creating your own docker image for Fluentd and would like to use the 0.12.26 then some config on the custom image should be changed. Otherwise you will get the notification that plugins cannot be found.

I needed to add these 2 lines to my build of an image based on v0.12.26 of the DockerHub image.

ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH
ENV GEM_PATH /home/fluent/.gem/ruby/2.3.0:$GEM_PATH

Docker Tags

Hi,

Is it possible to obtain the "latest" image from docker hub using a tag other than "latest"? Otherwise said, I'd like to pin my version on the current one and not get the next update automatically. Is there a missing tag?

How to Run this fluentd docker image as non-root user

I am using the Apline version https://github.com/fluent/fluentd-docker-image#apline-version

When I run this, it is running as ROOT

After running this as docker image

docker exec -it b3c565091160 /bin/sh

cat /etc/passwd

fluent:x:1000:1000::/home/fluent:

And

/home/fluent # ps -ef
PID   USER     TIME   COMMAND
    1 root       0:00 {fluentd} /usr/bin/ruby /usr/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins
    8 root       0:12 {fluentd} /usr/bin/ruby /usr/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins
   22 root       0:00 /bin/sh
   28 root       0:00 ps -ef
/home/fluent # whoami
root

How can run this Fluentd as fluent user, specifically user 1000 instead of ROOT?

td-agent availability

Hi guys, i'm starting my journey at fluentd this weekend. Do you guys can tell me with the td-agent is already part of this image? If not how could i install it properly?

Fluentd seem to be stuck when opening my log file

I installed fluentd on my machine to tail some log file, so far everything works great.
I wanted to use this docker image (actually I made my own based on this one, the only difference is I also gem install that elasticsearch plugin).
When I run my container, it seem to be stuck at (I set the log_level to trace):

2017-06-14 10:34:42 +0000 [info]: starting fluentd-0.14.17 pid=13
2017-06-14 10:34:42 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluentd.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2017-06-14 10:34:43 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.5'
2017-06-14 10:34:43 +0000 [info]: gem 'fluentd' version '0.14.17'
2017-06-14 10:34:43 +0000 [info]: adding match pattern="scheduler.**" type="elasticsearch"
2017-06-14 10:34:43 +0000 [trace]: #0 registered output plugin 'elasticsearch'
2017-06-14 10:34:44 +0000 [trace]: #0 registered buffer plugin 'memory'
2017-06-14 10:34:44 +0000 [warn]: #0 'type' is deprecated parameter name. use '@type' instead.
2017-06-14 10:34:44 +0000 [info]: adding source type="tail"
2017-06-14 10:34:44 +0000 [trace]: #0 registered parser plugin 'regexp'
2017-06-14 10:34:44 +0000 [trace]: #0 registered parser plugin 'multiline'
2017-06-14 10:34:44 +0000 [trace]: #0 registered input plugin 'tail'
2017-06-14 10:34:44 +0000 [info]: #0 starting fluentd worker pid=17 ppid=13 worker=0
2017-06-14 10:34:44 +0000 [debug]: #0 buffer started instance=70050719371560 stage_size=0 queue_size=0
2017-06-14 10:34:44 +0000 [info]: #0 following tail of /fluentd/logs/Scheduler.log
2017-06-14 10:34:44 +0000 [debug]: #0 flush_thread actually running
2017-06-14 10:34:44 +0000 [debug]: #0 enqueue_thread actually running
2017-06-14 10:34:44 +0000 [trace]: #0 enqueueing all chunks in buffer instance=70050719371560
2017-06-14 10:34:45 +0000 [trace]: #0 enqueueing all chunks in buffer instance=70050719371560
2017-06-14 10:34:46 +0000 [trace]: #0 enqueueing all chunks in buffer instance=70050719371560
2017-06-14 10:34:47 +0000 [trace]: #0 enqueueing all chunks in buffer instance=70050719371560

On the same machine, with the same fluentd.conf file, it works outside that docker container, but doesn't inside (I can see from my Kibana stack that nothing is ever received).

I also tried to attach a shell to that container and tail that file directly from within the container and it actually works correctly...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.