Comments (4)
If you are using a framework like Spring Boot, Quarkus, Micronaut, etc, those controllers are already instrumented so you don't need to re-instrument them with @Timed
.
Btw, you can apply @Counted
where you don't care about the duration and @Timed
where you do (@Counted
is also cheaper). In Prometheus, metrics under the same name should have the same type, the previous behavior was because Micrometer was able to generate basically any output which is not the case anymore.
from micrometer.
Thanks for the issue and thanks Andy for the reproducer!
Workaround
It seems this is only an issue with the Prometheus 1.x Client, using the 0.x Client does not seem to have this issue. As a workaround you can downgrade to the 0.x client by using micrometer-registry-prometheus-simpleclient
instead of micrometer-registry-prometheus
.
Please see the release notes of Micrometer 1.13.0 as well as the Migration Guide. In a nutshell: we upgraded micrometer-registry-prometheus
from Prometheus Client 0.x to 1.x but also kept the old client as-is under a different artifact: micrometer-registry-prometheus-simpleclient
.
(See a code example below.)
Investigation Notes
Here's another reproducer also proving that the 0.x Client does not have this issue (commented out):
io.micrometer.prometheusmetrics.PrometheusMeterRegistry registry = new io.micrometer.prometheusmetrics.PrometheusMeterRegistry(io.micrometer.prometheusmetrics.PrometheusConfig.DEFAULT);
// io.micrometer.prometheus.PrometheusMeterRegistry registry = new io.micrometer.prometheus.PrometheusMeterRegistry(io.micrometer.prometheus.PrometheusConfig.DEFAULT);
Timer.builder("test").tag("index", "1").publishPercentileHistogram(false).register(registry).record(Duration.ofMillis(100));
Timer.builder("test").tag("index", "2").publishPercentileHistogram( true).register(registry).record(Duration.ofMillis(100));
System.out.println(registry.scrape());
Doing such thing seems quite weird to me, it produces an output like this:
# HELP test_seconds
# TYPE test_seconds summary
test_seconds_count{index="1",} 1.0
test_seconds_sum{index="1",} 0.1
test_seconds_bucket{index="2",le="0.001",} 0.0
[...]
test_seconds_bucket{index="2",le="+Inf",} 1.0
test_seconds_count{index="2",} 1.0
test_seconds_sum{index="2",} 0.2
This means that test_seconds
is neither a Prometheus Summary (it has buckets) nor a Prometheus Histogram (it does not have buckets). The TYPE
says summary which is not true. I think this was possible since with the 0.x Client Micrometer had bigger freedom to do things (even if those things were invalid) and it was able to produce "invalid" results. The Prometheus 1.x Client is much restrictive so some things are not possible anymore (in general I think this is a good thing).
Let me reach out to the Prometheus Client maintainer but right now I feel the output should not be considered as valid and this should be fix in your code by
- either turning histograms on for every
Timer
/LongTaskTimer
/DistributionSummary
with the same name. - or use a different name for meters where histograms are on/off
from micrometer.
Thanks for investigation; I've applied your mentioned workaround with adding histogram to all those cases.
Example / Context / Reasoning behind this: We use @Timed
for all external methods of, for example, Controller to get information about througput - to group them all, all use e.g. controller_1 as name and add additional tags like "method=get" and so on to identify each method within the service. To reduce "costs" of storing operations where we know that they are fast / cheap we skip the histogram flag to reduce the amount of scraped data points.
from micrometer.
After talking to the maintainer of the Prometheus Client, there is one more workaround you can do right now: in Prometheus the histogram is valid if it has a single bucket: +Inf
. You can "approximate" this behavior with the current Micrometer API without any modifications by having two buckets using .serviceLevelObjectives(...)
:
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
Timer.builder("test").tag("index", "1")
.serviceLevelObjectives(Duration.ofNanos(1)) // or you can use any meaningful value, like 100 ms
.register(registry)
.record(Duration.ofMillis(100));
Timer.builder("test").tag("index", "2")
.publishPercentileHistogram( )
.register(registry)
.record(Duration.ofMillis(100));
System.out.println(registry.scrape());
Doing this means that sometimes you will have all the buckets and sometimes you will have only two buckets (not sure if it is good enough in your case). Unfortunately @Timed
can't do SLOs right now but we can consider adding it, we already have a PR: #5145 please let us know if this is something you would use (by commenting on it/adding a 👍🏼).
I think I still not recommend doing the above since the buckets are different and because of that they are not aggregatable which kind of defeats the purpose of the histogram and I would still go with having summary and histogram with different names but since you are already doing this, you probably already aware of the consequences.
With this, let me close this issue, please let us know if these don't help and we can reopen.
from micrometer.
Related Issues (20)
- io.micrometer:micrometer-observation:1.11.5 getting error with micrometer-test with version 1.12.0-SNAPSHOT with java version 17 & 21 HOT 5
- Make measured event duration accessible from outside of DefaultMeterObservationHandler HOT 3
- process_start_time_seconds HELP description inconsistency between Prometheus and micrometer HOT 1
- Make metric export filters applicable only to Dynatrace and not Spring Boot Admin (Spring Actuator) HOT 9
- Metrics in "/actuator/prometheus" are not consistent in a multi nodes environment (kubernetes) HOT 3
- Bump org.jetbrains.kotlin.jvm from 1.9.23 to 1.9.24
- Bump org.jetbrains.kotlin.jvm from 1.9.23 to 1.9.24
- zgcGenerationalTest is flaky on CI
- Support for removing metrics registered by CaffeineCacheMetrics HOT 12
- Gauge API like counter / timer HOT 6
- Use Duration for @Timed SLOs instead of double HOT 2
- Add "cancelled" information to the GrpcServerObservationContext HOT 3
- Make PrometheusMeterRegistryTest less flaky
- Apply test-retry settings to all tests
- Upgrade opentelemetry-collector-contrib to 0.104.0
- Value from environment variable `OTEL_EXPORTER_OTLP_ENDPOINT` is not being used HOT 2
- Add a metric for tracking heap size after last gc event HOT 2
- Migrate to io.spring.develocity.conventions:0.0.19
- Spring Boot 3.x Micrometer StatsD registry does not send metrics to UDP port when another StatsD client is active on the same port
- Excessive TLAB memory allocations from MetricsTurboFilter HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from micrometer.