Missing labels on metrics after restarting alfresco about alfred-telemetry HOT 10 CLOSED

from alfred-telemetry.

Yes, I checked. They are also missing from the prometheus metrics endpoint on Alfresco. See screenshot below.

from alfred-telemetry.

Thanks for checking.

@kerkhofsd any clue ?

from alfred-telemetry.

Which version of Alfresco? Or more specifically: is this a version of Alfresco which includes Micrometer out of the box? (Alfresco Enterprise >= 6.1)

from alfred-telemetry.

If we assume that this is on a version of Alfresco that includes Micrometer out of the box, this might be an explanation:

• Alfresco's PrometheusMeterRegistry is added to the static global composite registry (Metrics.globalRegistry). On this MeterRegistry no MeterFilters are applied (we don't touch Alfresco's registry).
• Alfred Telemetries PrometheusMeterRegistry is added to the static global composite registry. On this MeterRegistry the MeterFilters (and hence the common tags) are applied.
• As we can see here the scrape takes the first PrometheusMeterRegistry found in the global registry. This might be the Alfresco (no common tags) or the AT (with common tags) one, based on the position of the stars.

Just a theory that might explain the behavior: to be verified.

You should be able to test this theory by toggling following property: alfred.telemetry.alfresco-integration.include-default-alfresco-registry.
If alfred.telemetry.alfresco-integration.include-default-alfresco-registry=false, Alfresco's PrometheusMeterRegistry will not be registered and I would expect the issue to disappear.

from alfred-telemetry.

It's an Alfresco v6.1.1 so it indeed includes micrometer out of the box.

With the following metrics properties in alfresco-global.properties:

• metrics.enabled=true
• alfred.telemetry.alfresco-integration.enabled=true
• alfred.telemetry.binder.cache.enabled=true

from alfred-telemetry.

Your theory is correct!

The issue is solved by adding alfred.telemetry.alfresco-integration.include-default-alfresco-registry=false

Maybe this should be the default setting?

from alfred-telemetry.

@hvancamp thanks for the confirmation.

Yes, it might be a good idea to disable this by default.

However disabling the Alfresco PrometheusMeterRegistry is only a workaround. We should make sure that we always scrape the Alfred Telemetry PrometheusMeterRegistry for our Prometheus endpoint.

from alfred-telemetry.

After having another look, I think the best approach is indeed disabling the integration of Alfresco's PrometheusMeterRegistry by default.

Furthermore, I think we should add the restriction that only a single PrometheusMeterRegistry can be available in the global registry. This means users have two options:

1. Use the Alfresco PrometheusMeterRegistry by Alfresco.

   • Both scrape endpoints (/alfresco/s/alfred/telemetry/prometheus and /alfresco/s/prometheus) will be available
   • Both endpoints contain metrics registered by Alfresco and Alfred Telemetry
   • Both endpoints won't contain any common tags or other MeterFilters. (We don't customize the Alfresco registry)

2. Use the PrometheusMeterRegistry provided and initialized by Alfred Telemetry

   • Both scrape endpoints will be available
   • The Alfresco endpoint will contain only the metrics registered by Alfresco
   • The Alfresco endpoint won't contain any common tags or other MeterFilters.
   • The AT endpoint will contain metrics registered by Alfresco and Alfred Telemetry. (At least if alfred.telemetry.alfresco-integration.enabled=true)
   • The AT endpoint will contain common tags or other MeterFilter's

That sounds acceptable, right @tgeens ?

Maybe we should rename alfred.telemetry.alfresco-integration.include-default-alfresco-registry to e.g. ... .alfresco-integration.use-default-alfresco-registry?

from alfred-telemetry.

(2) sounds very reasonable

from alfred-telemetry.