Comments (10)
With the current implementation of CKMS that seems widespread (literally copied from place to place including Prometheus and Netflix), the quantiles are updated after each 500 sample batch. So it largely depends on how many samples are coming in within a particular time window.
But the flat nature of it in this graph just shows how CKMS is more stable and accurate at the expense of computational complexity. I imagine if you want an alert on one of these quantiles, you would set the threshold high enough that the perturbation in a successive approximation approach like Frugal is probably irrelevant.
At any rate, feel free to play with the sample from which this graph was generated.
from micrometer.
Prometheus uses CKMS, but we can evaluate the instrumentation cost of a range of quantile algorithms against this. Netflix also uses Frugal2U for load balancing. Great illustration of the effect over time here.
from micrometer.
Frugal is substantially faster, but as a successive approximation algorithm benefits from good initial estimates for faster convergence. There is a legitimate question about how to arrive at those initial estimates.
QuantilesBenchmark.ckmsQuantiles avgt 30 1233.518 ± 105.064 ns/op
QuantilesBenchmark.frugal2uQuantiles avgt 30 82.686 ± 2.720 ns/op
from micrometer.
I'm not familiar with CKMS/Frugal. Are those algorithms? If the output is pretty much identical, is it something worth pulling upstream?
I know a common footnote with using Summaries in Prometheus is in regards to potential 'cost' of the client side calculations.
Very cool to see the substantial difference.
from micrometer.
The average time benchmark does hide one somewhat pernicious characteristic about CKMS, which is the effect of the algorithm's "batch observations and calculate every so often" approach on worst case performance. The implementation in Netflix Ocelli and Prometheus batches 500 observations before computing a result. p=0.999 samples show the effect of this:
QuantilesBenchmark.ckmsQuantiles:p0.999 sample 1435648.000 ns/op
QuantilesBenchmark.frugal2uQuantiles:p0.999 sample 6693.648 ns/op
Yikes.
from micrometer.
An example of both algorithms at work simultaneously on two summaries recording the same values.
from micrometer.
The flat nature of CKMS seems to imply it lags.
If you don't mind, I would be interested to see if the numbers doubled at one point, how long would it take until CKMS actually register it?
from micrometer.
Ultimately, I built in 4 different quantile algorithms and selected the GK-based sliding window algorithm as the underlying implementation for @Timed
because it requires no tuning per quantile (so is simplest for annotation use) and is otherwise most similar to the native Prometheus implementation which is CKMS wrapped in a sliding-window.
from micrometer.
if you want to know the number of requests in a certain period of time. How can you achieve that with the given _sum
and _count
metrics.
https://www.innoq.com/en/blog/prometheus-counters/ I was looking to apply the increase function correctly.
from micrometer.
Please ask this on stack overflow
from micrometer.
Related Issues (20)
- Possible concurrency problems HOT 5
- Inconsistenty in JVM TimedExecutors "executionTimer" HOT 1
- Prometheus ClassCastException while scraping (Micrometer 1.13 / SB 3.3) HOT 4
- Disable kotlinSourcesJar since it conflicts with sourcesJar HOT 5
- Publish recent documentation under a fixed version number HOT 4
- Introduce Metric Filtering for CloudWatch Integration HOT 1
- Spring Boot with Kafka Template with supplied tags provider breaks spring_kafka_template metric
- Metric will be skipped when producing scrape output for PrometheusMeterRegistry HOT 9
- Fix histogram consistency in PrometheusMeterRegistry HOT 7
- ClassCastException io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot vs io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot when scraping two PrometheusTimers, one with publishPercentile, the other one without after upgrade to Micrometer 1.13 HOT 3
- Timer metrics - not able to see _seconds_count and _seconds_sum metrics, _max metric is getting scrap HOT 1
- Update japicmp compatibleVersion to 1.13.0 for 1.14.x
- [gRPC + coroutines] Observation to which we're restoring is not the same as the one set as this scope's parent observation HOT 4
- ClassicHistogramBuckets creating failure HOT 2
- `ClassNotFoundException` using `Counter` from Micrometer version `1.13.1` HOT 1
- Prometheus Java client 1.x consumes more memory than 0.x HOT 13
- Performance Issue: Heap is getting full slowly as using unique tag for each new entry of the. Looks like objects are not garbage collected. HOT 3
- Dynamic MeterFilter results in wrong Meter being reused HOT 4
- System memory metrics HOT 3
- Timed annotation should consider Transactional handling HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from micrometer.