Prometheus ClassCastException while scraping (Micrometer 1.13 / SB 3.3) about micrometer HOT 4 CLOSED

If you are using a framework like Spring Boot, Quarkus, Micronaut, etc, those controllers are already instrumented so you don't need to re-instrument them with @Timed.

Btw, you can apply @Counted where you don't care about the duration and @Timed where you do (@Counted is also cheaper). In Prometheus, metrics under the same name should have the same type, the previous behavior was because Micrometer was able to generate basically any output which is not the case anymore.

from micrometer.

Thanks for the issue and thanks Andy for the reproducer!

Workaround

It seems this is only an issue with the Prometheus 1.x Client, using the 0.x Client does not seem to have this issue. As a workaround you can downgrade to the 0.x client by using micrometer-registry-prometheus-simpleclient instead of micrometer-registry-prometheus.
Please see the release notes of Micrometer 1.13.0 as well as the Migration Guide. In a nutshell: we upgraded micrometer-registry-prometheus from Prometheus Client 0.x to 1.x but also kept the old client as-is under a different artifact: micrometer-registry-prometheus-simpleclient.
(See a code example below.)

Investigation Notes

Here's another reproducer also proving that the 0.x Client does not have this issue (commented out):

io.micrometer.prometheusmetrics.PrometheusMeterRegistry registry = new io.micrometer.prometheusmetrics.PrometheusMeterRegistry(io.micrometer.prometheusmetrics.PrometheusConfig.DEFAULT);
// io.micrometer.prometheus.PrometheusMeterRegistry registry =  new io.micrometer.prometheus.PrometheusMeterRegistry(io.micrometer.prometheus.PrometheusConfig.DEFAULT);

Timer.builder("test").tag("index", "1").publishPercentileHistogram(false).register(registry).record(Duration.ofMillis(100));
Timer.builder("test").tag("index", "2").publishPercentileHistogram( true).register(registry).record(Duration.ofMillis(100));
System.out.println(registry.scrape());

Doing such thing seems quite weird to me, it produces an output like this:

# HELP test_seconds  
# TYPE test_seconds summary
test_seconds_count{index="1",} 1.0
test_seconds_sum{index="1",} 0.1
test_seconds_bucket{index="2",le="0.001",} 0.0
[...]
test_seconds_bucket{index="2",le="+Inf",} 1.0
test_seconds_count{index="2",} 1.0
test_seconds_sum{index="2",} 0.2

This means that test_seconds is neither a Prometheus Summary (it has buckets) nor a Prometheus Histogram (it does not have buckets). The TYPE says summary which is not true. I think this was possible since with the 0.x Client Micrometer had bigger freedom to do things (even if those things were invalid) and it was able to produce "invalid" results. The Prometheus 1.x Client is much restrictive so some things are not possible anymore (in general I think this is a good thing).

Let me reach out to the Prometheus Client maintainer but right now I feel the output should not be considered as valid and this should be fix in your code by

• either turning histograms on for every Timer/LongTaskTimer/DistributionSummary with the same name.
• or use a different name for meters where histograms are on/off

from micrometer.

Thanks for investigation; I've applied your mentioned workaround with adding histogram to all those cases.

Example / Context / Reasoning behind this: We use @Timed for all external methods of, for example, Controller to get information about througput - to group them all, all use e.g. controller_1 as name and add additional tags like "method=get" and so on to identify each method within the service. To reduce "costs" of storing operations where we know that they are fast / cheap we skip the histogram flag to reduce the amount of scraped data points.

from micrometer.

After talking to the maintainer of the Prometheus Client, there is one more workaround you can do right now: in Prometheus the histogram is valid if it has a single bucket: +Inf. You can "approximate" this behavior with the current Micrometer API without any modifications by having two buckets using .serviceLevelObjectives(...):

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
Timer.builder("test").tag("index", "1")
    .serviceLevelObjectives(Duration.ofNanos(1)) // or you can use any meaningful value, like 100 ms
    .register(registry)
    .record(Duration.ofMillis(100));
Timer.builder("test").tag("index", "2")
    .publishPercentileHistogram( )
    .register(registry)
    .record(Duration.ofMillis(100));

System.out.println(registry.scrape());

Doing this means that sometimes you will have all the buckets and sometimes you will have only two buckets (not sure if it is good enough in your case). Unfortunately @Timed can't do SLOs right now but we can consider adding it, we already have a PR: #5145 please let us know if this is something you would use (by commenting on it/adding a 👍🏼).

I think I still not recommend doing the above since the buckets are different and because of that they are not aggregatable which kind of defeats the purpose of the histogram and I would still go with having summary and histogram with different names but since you are already doing this, you probably already aware of the consequences.

With this, let me close this issue, please let us know if these don't help and we can reopen.

from micrometer.