Backend metrics snapshotting is not compatible with semantics expected by Prometheus about vespa HOT 5 OPEN

This is work in progress.

from vespa.

I am trying to put all your current metrics into Prometheus Gauges, just to create some simple graphs in Grafana.

I have seen that the metric names is a mix of _-separated, .-separated and camel cased names. Some also contains '[' and ']'.

For prometheus I have to convert all this into a pure lowercased _-separated string.

from vespa.

I also notice that some metrics disappear when the cluster is idle, e.g. 95percentile of query_latency in container metrics. This makes it harder to track metrics into an external system.

from vespa.

@Oracien do you know if this is still a problem, after the latest changes to vespa prometheus integration?

from vespa.

@kkraune most of these issues have been resolved as far as I know. Counters do seem to be monotonically increasing over time and the format of metrics does appear to be metric_name{service="something-this", ...}. However, the last issue is still prevalent, if you are not running queries, then relevant query metrics such as rate does not appear. This should probably be fixed, I would guess it is something as simple as a filter (element != 0).

from vespa.