Use Prometheus and Grafana for Pi monitoring about turing-pi-cluster HOT 17 CLOSED

I have the node_exporter role just about up and running. Doing some final testing on it now.

from turing-pi-cluster.

Grafana example deployment (I may just use this): https://github.com/kubernetes/ingress-nginx/tree/master/deploy/grafana

from turing-pi-cluster.

Trying to get it running with kube-prometheus, but I've run into prometheus-operator/kube-prometheus#545

from turing-pi-cluster.

It looks like there are actually a lot of different bits and pieces that are not compatible with ARM/Pi, so next step is trying out https://github.com/carlosedp/cluster-monitoring instead. See related blog post: Creating a full monitoring solution for ARM Kubernetes Cluster.

from turing-pi-cluster.

I hit an issue with Getting /bin/jb: not found when I try running make vendor. But resolved it by setting GOPATH: /home/pirate/go in the make command environment differently.

from turing-pi-cluster.

New issue: During 'make', I get 'gojsontoyaml: not found'.

from turing-pi-cluster.

Another new issue: Prometheus operator giving RBAC error with latest commit.

from turing-pi-cluster.

And: CPU Temperature alert giving 'No data' on Raspberry Pi CM3+.

from turing-pi-cluster.

All those issues are fixed, though now I'm not getting any node metrics :D

from turing-pi-cluster.

And one more: 'arm-exporter' isn't running on master node, only workers.

from turing-pi-cluster.

I have this all working now:

Two open PRs are in the works or the upstream repo. The two changes that are required currently until carlosedp/cluster-monitoring#41 and carlosedp/cluster-monitoring#42 are merged:

Add the following in the armexporter-daemonset.yaml DaemonSet spec:

    spec:
      affinity:                                                                           
        nodeAffinity:                                                  
          requiredDuringSchedulingIgnoredDuringExecution:              
            nodeSelectorTerms:                                         
            - matchExpressions:                                        
              - key: kubernetes.io/arch                                
                operator: In                                           
                values:                                                
                - arm                                                  
                - arm64 
      tolerations:
      - operator: Exists

And remove the nodeSelector from the same spec.

from turing-pi-cluster.

Another issue, just an annoyance more than anything: CPU Temperature monitor giving Pod IPs instead of node IPs, so DNS names don't display.

from turing-pi-cluster.

Affinity PR was fixed up and merged, woohoo!

from turing-pi-cluster.

One other note: It seems that Prometheus dies when the worker on which it's scheduled gets any extra load, and starts swapping like mad. Swap is soooooo slow on the Pi's eMMC or microSD, so load goes north of 9-10, then the node becomes unresponsive.

I opened a support issue How to change Prometheus' scrapeInterval?, to see if maybe there's a way to easily work around this by scraping less often.

from turing-pi-cluster.

Sheesh. Every little bug, fixed in a couple days. Thanks so much to @carlosedp for helping out!

from turing-pi-cluster.

Testing a final rebuild on the latest commit, if it works, I can close this out, woohoo!

from turing-pi-cluster.

It's all working quite well now.

from turing-pi-cluster.