Error while sending request to Stackdriver ... unsupported protocol scheme """ about k8s-stackdriver HOT 16 OPEN

The problem is in kubernetes/kubernetes#54783. This change includes changes to configuration that 1.7 was not supporting. I'm going to prepare a fix, but we will need to wait until next release.

Sorry for inconvenience.

from k8s-stackdriver.

Any update on this? I'm running a new GKE v1.14.10-gke.27 cluster (which is deploying prometheus-to-sd:v0.4.2) and the problem is still occuring

from k8s-stackdriver.

Sure.
Btw. I have create a PR for 1.7 that should fix an issue. Hopefully it's going to be included into the next patch release. kubernetes/kubernetes#58959

from k8s-stackdriver.

@loburm Can you please kindly give an update or is there any temporary workaround/fix for this problem as this is open for too long.

from k8s-stackdriver.

Experiencing the same problem myself.

from k8s-stackdriver.

Been experiencing this issue as well. Fairly annoying, seems like a stackdriver bug.

from k8s-stackdriver.

Actually looks like configuration of prometheus-to-sd is broken. Parameters in yaml files are not replaced by correct values.

Investigating...

from k8s-stackdriver.

Looks like 1.8 is not affected, I was able to find this problem only on 1.7. If anyone has this problem on other cluster please notify me.

from k8s-stackdriver.

In GKE context, dos this manifest as stackdriver monitoring being unable to see metrics for GKE pods?

from k8s-stackdriver.

@tomoe not sure that I have understood your question. Heapster should work without any problem. The only problem that in GKE we are not able to collect system metrics related to the fluentd.

from k8s-stackdriver.

Thanks @loburm. Pls disregard my question. I think I was confusing heapster with fluentd.

from k8s-stackdriver.

This doesn't seem to be limited to 1.7.

We are observing this on multiple GKE clusters. The one most recently created runs 1.11.6-gke.2 with kube-dns using gcr.io/google-containers/prometheus-to-sd:v0.2.3 with the following config:

/monitor
  --component=kubedns
  --target-port=10054
  --stackdriver-prefix=container.googleapis.com/internal/addons
  --api-override=https://monitoring.googleapis.com/
  --whitelisted-metrics=probe_kubedns_latency_ms,probe_kubedns_errors,dnsmasq_misses,dnsmasq_hits
  --pod-id=$(POD_NAME)
  --namespace-id=$(POD_NAMESPACE)
  --v=2 

Error Message every 10s:

E0116 18:52:47.901658 1 stackdriver.go:58] Error while sending request to Stackdriver Post /v3/projects/our-organisation-name/timeSeries?alt=json: unsupported protocol scheme ""

from k8s-stackdriver.

I believe that's due to fact that kubedns is using very old image of prometheus-to-sd. I'll create internal issue for folks to fix it. Thank you for noticing.

from k8s-stackdriver.

same issue Error while sending request to Stackdriver Post /v3/projects/***/timeSeries?alt=json: unsupported protocol scheme """

from k8s-stackdriver.

Still seeing this on 1.12.6-gke.10

Also accompanied by the error

Error while fetching metric descriptors for kubedns: Get /v3/projects/together-k8s-staging/metricDescriptors?alt=json&filter=metric.type+%3D+starts_with%28%22%7B%7B+prometheus_to_sd_prefix+%7D%7D%2Faddons%2Fkubedns%22%29:

from k8s-stackdriver.

@loburm
On 1.12.6-gke.10 I see prometheus-to-sd:v0.4.2 (so version released after your last comment).
Do you have any update on this?

from k8s-stackdriver.