I don't have hard proof for this, but from my understanding several calls to micrometer contain locks and should be wrapped in blocking instead of delay.

Example:
https://github.com/ovotech/meters4s/blob/master/core/src/main/scala/com/ovoenergy/meters4s/Reporter.scala#L312

This delay calls after some indirection the following method in micrometer:
https://github.com/micrometer-metrics/micrometer/blob/54f89f40743937a5a78fb57eb0ddb6b2e6238be4/micrometer-core/src/main/java/io/micrometer/core/instrument/MeterRegistry.java#L598

This contains a synchronized block over meterMapLock and can block the current thread. Therefore I would argue that this could lead to blocking one of the io-compute threads if several calls hit the lock in parallel.

I am raising this issue to track the problem and initial findings and intend to raise it as an issue with micrometer.

Summary

Counters incremented near application shutdown are not emitted when using StatsD. (I have not tested gauges or timers, they may also be affected).

Code Analysis

It appears the metrics are being buffered within micrometer and aren't flushed when the resources are torn down. Meters4s appears to be correctly shutting down the micrometer objects by calling .close:

Repro

The issue can be reproduced along the lines of this gist, including running a local mock DataDog UDP server.

As an example, I can see the application outputs:

[info] Emitted counter metric of name "375"
[info] Emitted counter metric of name "467"
[info] Emitted counter metric of name "567"
... # redacted
[info] Emitted counter metric of name "2268"
[info] Emitted counter metric of name "2367"
[info] Emitted counter metric of name "2468"
[info] Emitted counter metric of name "2567"
[info] Emitted counter metric of name "2668"
[info] Emitted counter metric of name "2767"
[info] Emitted counter metric of name "2867"
[info] Emitted counter metric of name "2967"
[info] Emitted counter metric of name "3067"
[info] Emitted counter metric of name "3167"
[info] Emitted counter metric of name "3267"
[info] -------------------------------
[info] stopping after 30 metrics

But the mock server does not receive the final metrics:

gordon.375:1|c|#statistic:count,tag:test
gordon.467:1|c|#statistic:count,tag:test
gordon.567:1|c|#statistic:count,tag:test
... # redacted
gordon.2067:1|c|#statistic:count,tag:test
gordon.2167:1|c|#statistic:count,tag:test
gordon.2268:1|c|#statistic:count,tag:test
gordon.2367:1|c|#statistic:count,tag:test
gordon.2468:1|c|#statistic:count,tag:test
# end of output

Reproduced On

meters4s 1.3.1, micrometer 1.9.8 (bundled with meters4s release) and 1.11.0-RC1 and latest.integration on 4th May 1930 BST

To Investigate

micrometer has open issue micrometer-metrics/micrometer#2141. It was suggested it may have been resolved by other, recently fixed, issue micrometer-metrics/micrometer#1882. However the fact that the latest micrometer builds do not resolve the problem indicates not; I haven't had time to dive into the micrometer metrics class hierarchy yet to understand what was fixed.

Workarounds

I found the following resolved the problem:

• adding a magic sleep that exceeds the configured polling time after the last metric call
• OR setting buffered = false in the StatsD config (with performance impact)