newrelic / kubernetes-logging Goto Github PK

View Code? Open in Web Editor NEW

7.0 21.0 25.0 65 KB

DEPRECATED: use https://github.com/newrelic/helm-charts

License: Apache License 2.0

Smarty 100.00%

kubernetes-logging's Introduction

DEPRECATION NOTICE

This repository has been deprecated. Use https://github.com/newrelic/helm-charts instead.

New Relic Kubernetes logging

Welcome to the New Relic Fluent Bit output plugin for Kubernetes! There are only a few quick steps to getting this working in your cluster.

Usage

Deploying as Helm chart (Recommended)

Clone this repo
Deploy the chart using your New Relic license key: helm install --set licenseKey=(your-license-key),clusterName=(your-cluster-name) ./helm/newrelic-logging
If you are in the EU, make sure that the license key you are using is an EU key. You will not need to specify the EU logging endpoint -- it will be inferred from the license key
Check New Relic for your logs

Applying k8s manifests manually

Clone this repo
Configure the plugin. In new-relic-fluent-plugin.yml:
- Specify your New Relic license key in the value for LICENSE_KEY
- Specify your Kubernetes cluster name in the value for CLUSTER_NAME
- If you are in the EU:
  - Override the ENDPOINT environment variable to https://log-api.eu.newrelic.com/log/v1
  - Make sure that the license key you are using is an EU key
From this directory, run kubectl apply -f . on your cluster
Check New Relic for your logs

Find and use your data

For how to find and query your data in New Relic, see Find log data.

For general querying information, see:

Configuration notes

We default to tailing /var/log/containers/*.log. If you want to change what's tailed, just update the PATH value in new-relic-fluent-plugin.yml.

Parsing

We currently support parsing json and docker logs. If you want more parsing, feel free to add more parsers in fluent-conf.yml.

Here are some parsers for your parsing pleasure.

[PARSER]
    Name   apache
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   apache2
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   apache_error
    Format regex
    Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Legal

This project is provided AS-IS WITHOUT WARRANTY OR SUPPORT, although you can report issues and contribute to the project here on GitHub.

kubernetes-logging's People

Contributors

Stargazers

Watchers

kubernetes-logging's Issues

Feature Request - Custom Fluentd config

We'd like to have two Fluentd outs, one to New Relic, and the other to our archival storage. If custom fluentd.conf are possible today, the documentation isn't clear on how to do so. If not, then I'm happy to write a PR if you point me in the write direction.

Fluent Plugin

Has there been any discussion about releasing some of this functionality as a Fluent Plugin? It would make integrating with current fluentd setups much easier, as it could just be tacked on to existing Fluentd configurations/ parsing pipelines.

Thanks,
Austin

Helm Hub

Hi there,

Has newrelic thought about integrating your Helm charts with the new hub.helm.sh platform? Would make it much easier to work with your charts.

Thanks!
Austin

Fluent bit failing to reach Kubernetes API server - Amazon EKS

Deployed the helm chart via Pulumi (similar to Terraform). Daemonset running, RBAC created however the pods are failing to reach the Kubernetes API server:

│ Copyright (C) Treasure Data                                                                                                                                         │
│                                                                                                                                                                     │
│ [2019/12/09 22:54:41] [ info] [storage] initializing...                                                                                                             │
│ [2019/12/09 22:54:41] [ info] [storage] in-memory                                                                                                                   │
│ [2019/12/09 22:54:41] [ info] [storage] normal synchronization mode, checksum disabled                                                                              │
│ [2019/12/09 22:54:41] [ info] [engine] started (pid=1)                                                                                                              │
│ [2019/12/09 22:54:41] [ warn] [filter_kube] merge_json_log is deprecated, enabling 'merge_log' option instead                                                       │
│ [2019/12/09 22:54:41] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443                                                              │
│ [2019/12/09 22:54:41] [ info] [filter_kube] local POD info OK                                                                                                       │
│ [2019/12/09 22:54:41] [ info] [filter_kube] testing connectivity with API server...                                                                                 │
│ [2019/12/09 22:54:41] [ warn] [filter_kube] could not get meta for POD ip-10-208-83-172.us-west-2.compute.internal                                                  │
│ [2019/12/09 22:54:41] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020

RBAC is set to 'yes', serviceaccount is 'yes'.

Service account:

Name:                newrelic-logging
Namespace:           newrelic
Labels:              app=newrelic-logging
                     app.kubernetes.io/managed-by=pulumi
                     chart=newrelic-logging-1.0.0
                     heritage=Tiller
                     release=newrelic-logging
Annotations:         kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"labels":{"app":"newrelic-logging","app.kubernetes.io/managed-by":"pulumi","chart":...
Image pull secrets:  <none>
Mountable secrets:   newrelic-logging-token-4bfrt
Tokens:              newrelic-logging-token-4bfrt
Events:              <none>

Cluster Role Binding

Name:         newrelic-logging
Labels:       app=newrelic-logging
              app.kubernetes.io/managed-by=pulumi
              chart=newrelic-logging-1.0.0
              heritage=Tiller
              release=newrelic-logging
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"labels":{"app":"newrelic-logging","app.kubernetes.io...
Role:
  Kind:  ClusterRole
  Name:  newrelic-logging
Subjects:
  Kind            Name              Namespace
  ----            ----              ---------
  ServiceAccount  newrelic-logging  newrelic

Cluster Role

Name:         newrelic-logging
Labels:       app=newrelic-logging
              app.kubernetes.io/managed-by=pulumi
              chart=newrelic-logging-1.0.0
              heritage=Tiller
              release=newrelic-logging
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"labels":{"app":"newrelic-logging","app.kubernetes.io/manage...
PolicyRule:
  Resources   Non-Resource URLs  Resource Names  Verbs
  ---------   -----------------  --------------  -----
  namespaces  []                 []              [get list watch]
  pods        []                 []              [get list watch]

Add the ability to add additional attributes to the Logger

It would be great if we could easily add global attributes to the Logger.

E.g.
Cluster: {Kubernetes Cluster Name}
Region: {Kubernetes Cluster hosted region}
and more...

If we could achieve that by customising the Helm Chart, or by adding Environment Variables, that would be great.

Feature Request - Add ability to tag the resources you want the logs captures (or opt-out)

Installation and setup with this was easy and straightforward, nice work!

It'd be nice to opt-in or out of New Relic collecting logs based on metadata tags we assign to our K8 resources, or specify a K8 namespace to collect logs from. We have some third-party managed resources we do not care to collect logs from (e.g. Azure's Key Vault).

Fail to work in OKD 3.11 (any idea to fix this problem is very helpful.)

Hello. I am working with OpenShift OKD 3.11 Kubernetes Cluster and is trying to install kubernetes-logging. I have confirmed that Kubernetes Logging can display log in newrelic for GKE (Google Kubernetes Engine) but is not working for OKD 3.11.

Below is the error I get when I apply to OKD 3.11.

apply with helm => fail to display log in newrelic.

$ helm install --set licenseKey=?????? ./helm/newrelic-logging
NAME:   geared-hog
LAST DEPLOYED: Fri Dec  6 16:56:52 2019
NAMESPACE: newrelic-logging
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                         AGE
geared-hog-newrelic-logging  0s

==> v1/ClusterRoleBinding
NAME                         AGE
geared-hog-newrelic-logging  0s

==> v1/ConfigMap
NAME                                           DATA  AGE
geared-hog-newrelic-logging-fluent-bit-config  5     1s

==> v1/Secret
NAME                                TYPE    DATA  AGE
geared-hog-newrelic-logging-config  Opaque  1     1s

==> v1/ServiceAccount
NAME                         SECRETS  AGE
geared-hog-newrelic-logging  2        1s

==> v1beta2/DaemonSet
NAME                         DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
geared-hog-newrelic-logging  0        0        0      0           0          <none>         0s


NOTES:

Your deployment of the New Relic Kubernetes Logging is complete. You can check on the progress of this by running the following command:

    kubectl get daemonset -o wide -w --namespace newrelic-logging geared-hog-newrelic-logging

apply in different method => fail to display log in newrelic.

$ oc apply -f .
configmap/fluent-bit-config unchanged
daemonset.extensions/newrelic-logging unchanged
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read unchanged
clusterrole.rbac.authorization.k8s.io/fluent-bit-read unchanged
serviceaccount/newrelic-logging unchanged

daemonset

$ oc describe daemonset geared-hog-newrelic-logging
Name:           geared-hog-newrelic-logging
Selector:       app=newrelic-logging,release=geared-hog
Node-Selector:  <none>
Labels:         app=newrelic-logging
                chart=newrelic-logging-1.0.0
                heritage=Tiller
                release=geared-hog
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=newrelic-logging
                    release=geared-hog
  Service Account:  geared-hog-newrelic-logging
  Containers:
   newrelic-logging:
    Image:      newrelic/newrelic-fluentbit-output:1.1.2
    Port:       <none>
    Host Port:  <none>
    Command:
      /fluent-bit/bin/fluent-bit
      -c
      /fluent-bit/etc/fluent-bit.conf
      -e
      /fluent-bit/bin/out_newrelic.so
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      ENDPOINT:     https://log-api.eu.newrelic.com/log/v1
      SOURCE:       kubernetes
      LICENSE_KEY:  <set to the key 'license' in secret 'geared-hog-newrelic-logging-config'>  Optional: false
      BUFFER_SIZE:  256000
      MAX_RECORDS:  500
      LOG_LEVEL:    info
      PATH:         /var/log/containers/*.log
    Mounts:
      /fluent-bit/etc from fluent-bit-config (rw)
      /var from var (rw)
  Volumes:
   fluent-bit-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      geared-hog-newrelic-logging-fluent-bit-config
    Optional:  false
   var:
    Type:          HostPath (bare host directory volume)
    Path:          /var
    HostPathType:  
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  19s (x13 over 39s)  daemonset-controller  Error creating: pods "geared-hog-newrelic-logging-" is forbidden: unable to validate against any security context constraint: [provider anyuid: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used]

This chart's DaemonSet apiVersion is deprecated and removed

The DaemonSet resource is no longer served from apps/v1beta2 as of Kubernetes 1.16.

You can use Capabilities.KubeVersion in your helm chart to check if the target cluster is 1.9 or higher and use the apps/v1 if so.

new FILTER "grep" rule doesn't work

I have changed the filter-kubenetes.con part in fluent-conf.yml:

  filter-kubernetes.conf: |
    [FILTER]
        Name record_modifier
        Match *
        Record cluster_name ${CLUSTER_NAME}

    [FILTER]
        Name           kubernetes
        Match          kube.*
        Kube_URL       https://kubernetes.default.svc.cluster.local:443
        Merge_JSON_Log Off

    [FILTER]
        Name grep
        Match *
        Exclude message HealthChecker

And want to exclude log messages like:
172.18.110.109 - - [29/May/2020:13:02:35 +0000] "GET / HTTP/1.1" 200 2804 "-" "ELB-HealthChecker/2.0" "-"

But seem it doesn't work - such log messages still coming to NR Logs page.
Also I checked:

Exclude log HealthChecker
the same behavior.
Please advice - is the Filter rules works in New Relic Fluent Bit output plugin for Kubernetes?