Attach original tag for 'mounted-file' about kube-fluentd-operator HOT 7 CLOSED

@cw-sakamoto
I've implemented the add_labels feature described earlier. You can give it a try with this image jvassev/kube-fluentd-operator:v1.6.0-17-gbdc4593. It's built from bdc4593.

Just put add_labels sub-component=compXXX to your mounted_files and then you can distinguish between different log files from the same container.

from kube-fluentd-operator.

Hi,
In general kube-fluentd-operator tries to step away from the fluentd tag. Instead, it promotes Kubernetes labels.

In your case however all the logevents come from a single pod meaning they share the same labels. So what can be done here is let users attach new labels to the @type mounted-file logs ike this:

<source>
  @type mounted-file
  path /var/log/hello1.log
  labels msg=hello
  add_labels app=app1
  tag hello1-log
</source>

<source>
  @type mounted-file
  path /var/log/hello2.log
  labels msg=hello
  add_labels app=app2
  tag hello2-log
</source>

Then you use the $labels macro to select the logs:

<match $labels(app=app1)>
  @type test123
</match>

<match $labels(app=app2)>
  @type test456
</match>

Indeed, this would be easy to add - do you think this would help you in your setup?.
However, it would be more "kubernetes-idiomatic" to spread those processes producing different log files into independent pods.

from kube-fluentd-operator.

@cw-sakamoto In my previous comment I assumed you have two log files produced by the same container in a single pod.
If the logs files are produced by different containers then you can solve your case even with the current version by using the _container meta-label:

<match $labels(msg=hello, _container=cont1)>
  @type test123
</match>

<match $labels(msg=hello, _container=cont2)>
  @type test456
</match>

from kube-fluentd-operator.

@jvassev Thanks for your comments.

the same container in a single pod.

You are right. I assume the same container.

It can be solved by #26 (comment) , however I thought that it is difficult to distinguish from label of k8s itself by this writing style.

In general kube-fluentd-operator tries to step away from the fluentd tag. Instead, it promotes Kubernetes labels.

Under this basic idea, I think that my proposal(add fluentd tag) is difficult, so think a little more...🤔

from kube-fluentd-operator.

This is just idea...add $labels meta_label _file(or _path) like _container

$labels(msg=hello, _file=/var/log/hello.log)

or

$labels(msg=hello, _path=/var/log/hello.log)

What do you think this idea?

from kube-fluentd-operator.

I thinks it's better to use add_labels because labels values have strict format validation. A path can contain =, ,, which will confuse the $labels macro

from kube-fluentd-operator.

The add_labels feature is now merged.

from kube-fluentd-operator.