Support default/secure metrics-server installation about doks HOT 17 OPEN

1.19 users are presumably affected by an incompatibility with Docker 18 as described in kubernetes/kubernetes#94281.

We plan to release a DOKS 1.19 update (ideally today) that is going to address the problem by moving to Docker 19.03.

from doks.

With secure TLS usage of the kubelet API now possible (see the update to #6), the --kubelet-insecure-tls parameter is not needed anymore.

Users are down to having to specify --kubelet-preferred-address-types=InternalIP at this point, which is the last item to tackle prior to the default metrics-server configuration working out of the box.

from doks.

@mbrodala this may indeed be an issue on our end. I filed an internal bug report so that we can look into the matter more closely.

Thanks for bringing this to our attention. I'm going to report back to this issue once we've identified and fixed the problem.

from doks.

the stats still don't show within the dashboard web UI.

Unfortunately, that's unrelated to the latest release. It's still on the agenda to get it fixed as well.

from doks.

@timoreimann as a 1.19 user I can confirm I'm affected by that (or something else) and currently don't have pod metrics through metrics-server

from doks.

FYI:
1.19.3-do.2 was just released and should fix the problem. Please report back if that's not the case.

Originally posted by @timoreimann in digitalocean/digitalocean-cloud-controller-manager#150 (comment)

Can confirm this is now fixed for me

from doks.

I just set up the metrics-server with the components.yaml from the v0.3.6 release and made sure to inject the --kubelet-preferred-address-types=InternalIP flag as mentioned here. (Using kustomize and a JSON patch for this)

The cluster is running on 1.17.5-do.0 currently.

Everything seems to be running just fine and I get no errors. With kubectl top node i see some metrics. However I see no change in the DOKS dashboard as suggested in official dashboard docs:

This is my view instead:

Am I missing something?

from doks.

I just installed metrics-server with doctl kubernetes cluster create --1-clicks="metrics-server" .... When I run kubectl top pods --all-namespaces I get:

W1031 04:06:49.790056 1123183 top_pod.go:265] Metrics not available for pod default/external-dns-68cf9b5c56-4bjqs, age: 40m32.790034709s
error: Metrics not available for pod default/external-dns-68cf9b5c56-8bjqs, age: 40m32.790034709s

The metrics pod logs something like this:

I1031 03:03:25.598399       1 manager.go:148] ScrapeMetrics: time: 5.606118ms, nodes: 1, pods: 0
E1031 03:03:26.647337       1 reststorage.go:160] unable to fetch pod metrics for pod cert-manager/cert-manager-cainjector-6d59c8d4f7-7l274: no metrics known for pod
E1031 03:03:26.647589       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-operator-6cb976fbcf-rc8lv: no metrics known for pod
E1031 03:03:26.647701       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-rl67k: no metrics known for pod
E1031 03:03:26.647793       1 reststorage.go:160] unable to fetch pod metrics for pod cert-manager/cert-manager-webhook-578954cdd-9vwph: no metrics known for pod
E1031 03:03:26.647843       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-4f2vb: no metrics known for pod
E1031 03:03:26.647900       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-76bcfddf46-rkfvt: no metrics known for pod
E1031 03:03:26.647957       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/do-node-agent-wx74s: no metrics known for pod
E1031 03:03:26.648048       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/csi-do-node-skgr8: no metrics known for pod
E1031 03:03:26.648137       1 reststorage.go:160] unable to fetch pod metrics for pod default/external-dns-68cf9b5c56-8bjqs: no metrics known for pod
E1031 03:03:26.648196       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-operator-6cb976fbcf-z5vmk: no metrics known for pod
E1031 03:03:26.648288       1 reststorage.go:160] unable to fetch pod metrics for pod cert-manager/cert-manager-86548b886-j655n: no metrics known for pod
E1031 03:03:26.648387       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-76bcfddf46-lsvm5: no metrics known for pod
E1031 03:03:26.648433       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-5b8f47666-m6sn5: no metrics known for pod
E1031 03:03:26.648487       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-state-metrics-6f4b6669f9-zhgw2: no metrics known for pod
E1031 03:03:26.648544       1 reststorage.go:160] unable to fetch pod metrics for pod ingress-nginx/ingress-nginx-controller-98cb87fb7-bfvvr: no metrics known for pod

from doks.

Required work items are referenced above.

from doks.

@mbrodala is this the dashboard we integrate in the DigitalOcean cloud control panel, or a separate deployment you manage on your own?

from doks.

@timoreimann this is the original dashboard provided by DOKS.

from doks.

@timoreimann Any news on this?
I'm also facing the exact same issue - metrics server works properly (with the mentioned modifications) & I can see the stats using the top command, but the (built-in) dashboard isn't showing them in the UI.

from doks.

The dashboard metrics are served by a different, separate sidecar, which we yet have to integrate. I created #21 to track the effort.

from doks.

@feluxe I have just the same issue with you

from doks.

@feluxe @johnwook and I have the same issue as you guys too.

from doks.

Same here. kubectl top ... works with the new update, but the stats still don't show within the dashboard web UI.

from doks.

Hi this is still biting me on 1,22. I have to change the endpoints to be InternalIP and enable the apiService

from doks.