no hostname/nodename information logged about kube-fluentd-operator HOT 10 CLOSED

Reading this: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#environment-variables-for-kubernetes

I think it makes sense in general to expose the nodeName as K8S_NODE_NAME to the fluentd process (if only as an optimisation).

I just thought that could also be used to annotate the systemd unit logs.

from kube-fluentd-operator.

So this sort of works:

      - command:
        - /bin/config-reloader
       ...
        - --meta-key=metadata
        - --meta-values=node=$(NODE_NAME)
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName

from kube-fluentd-operator.

Hi @databus23,
Information about kubernetes is nested under the "kubernetes" key - it's a core feature of the fluent-plugin-kubernetes_metadata_filter plugin.

For example, a log event would look like this and what you need is kubernetes.host:

{
    "kubernetes": {
        "container_image_id": "docker-pullable://.......",
        "container_name": "...",
        "host": "ip-XXXXX.us-west-2.compute.internal",
        "pod_id": "3772c64f-7e0b-11e8-ae9f-06dea036ca4c",
        "container_image": "xxxx",
        "namespace_name": "xxxxx",
        "pod_name": "xxxxx-998d48b69-skbhg",
        "labels": {
            "app": "test",
            "environment": "hello"
        }
    }
}

from kube-fluentd-operator.

@jvassev But this is not available for logs from system units. How can I know which kubelet.service (which node) a log line comes from?

from kube-fluentd-operator.

You're right, I didn't consider systemd at all. I think 58d1869 should fix it.

from kube-fluentd-operator.

I've just published the image jvassev/kube-fluentd-operator:v1.4.0-5-g58d1869 with the change applied. Can you try with it by upgrading the daemonset?

from kube-fluentd-operator.

@jvassev No it doesn't. I tried that, _HOSTNAME just contains localhost

from kube-fluentd-operator.

Are you sure the machines have their hostname configured properly? I've checked on several debian hosts and it looks good.

You can try journalctl -o json-pretty and see the raw data before it reaches fluentd.

from kube-fluentd-operator.

d819652 enables the K8S_NODE_NAME, thanks for the suggestion!
The chart version is bumped to 0.2.2.

from kube-fluentd-operator.

@jvassev Ok, you got me. My cloud VMs had a metadata problem and default to localhost for the hostname...

from kube-fluentd-operator.