Sidecar stopped submitting stats to StackDriver after prometheus eviction on GKE about stackdriver-prometheus-sidecar HOT 11 CLOSED

Thanks for the detailed report. That indeed sounds like the sidecar got stuck somehow.

Is the Prometheus server scraping the sidecar for metrics? Could you check what the prometheus_sidecar_samples_processed and prometheus_sidecar_samples_produced metrics report over time? That would tell us whether it is the WAL is actually being processed.

That Prometheus's CPU usage drops when the sidecar is shut down would indicate that the sidecar potentially does a lot of requests for metric and target metadata against the API. That's at least the only way I can see the sidecar affect Prometheus.
The HTTP request metrics of Prometheus and the sidecar respectively should show whether an endpoint is hit a lot.

It's probably easiest if you post screenshots of some of those metrics here.

from stackdriver-prometheus-sidecar.

I added the sidecar as a target but those metrics do not show up.

The only metric I can see with the prometheus_sidecar_ prefix is prometheus_sidecar_dropped_series, I'm attaching the graph below and a manual scrape of the metrics endpoint of the sidecar.

Sidecar Metrics:
sd_metrics.txt

from stackdriver-prometheus-sidecar.

I just realized that this last information that I posted may be useless. Before giving up I deleted the stackdriver_sidecar.json and let is work overnight to see if it made a difference, that may be causing the errores that are being reported on the graph.

from stackdriver-prometheus-sidecar.

Hi, @lkoba, any update on the answers to @fabxc's questions above?

from stackdriver-prometheus-sidecar.

No. I posted the information requested but didn't research much more, I ended up using prometheus-to-sd as a workaround to get the information I needed on StackDriver. Thanks for your help.

from stackdriver-prometheus-sidecar.

Hi @lkoba, thanks for the graph and the metric dump.

This indicates that the metrics the sidecar could not find a matching target via Prometheus's API for the metrics it read from disk. It also clearly does a lot of requests to Prometheus target API endpoint.

Could you also provide is with the curl output from /api/v1/targets?
Your full configuration file for Prometheus would also be helpful.

from stackdriver-prometheus-sidecar.

Here's the information you requested:

targets.json.txt

prometheus.yml.txt

Please let me know if you need anything else.

from stackdriver-prometheus-sidecar.

Hi @lkoba,

we've likely identified the issue and will push a fix soon. Once it's in, you might want to give this another shot.

from stackdriver-prometheus-sidecar.

Thanks! Just let me know when, and I'll test it.

from stackdriver-prometheus-sidecar.

Hi @lkoba,

We've identified an issue that negative cache in targets doesn't work as we expected, and it may make sidecar keep querying targets API and block the traffic. We fixed this issue with #91.

We've released a new release v0.4.0 including #91, and you may want to try it out.

Thanks again for reporting the issue!

from stackdriver-prometheus-sidecar.

Hi @lkoba,

Just want to follow up on this thread. If you ever test the new image, let us know whether you encounter any issue. My team will be happy to hear your feedback!

from stackdriver-prometheus-sidecar.