Add Audit Logs to the User Configurable API about tempo HOT 5 OPEN

Interested in taking this up, if there are any further details that can be added!

from tempo.

We should already be logging some of this information in the API layer. For example when storing new limits: https://github.com/grafana/tempo/blob/main/modules/overrides/userconfigurable/api/api.go#L104

We currently log timestamp, tenant, the new value and the new version. When we are updating we also log the current version and value.
Is there anything else you would like to see? We could also log stuff like IP address and any headers, but I'm not sure how valuable that is since Tempo is most often deployed behind a gateway.

Finding these logs should be straightforward with Loki:

{namespace=~"tempo.*"} |= "component=overrides-api" != "method=GET"

from tempo.

Ah, I didn't realize we had solid audit logging here. @mdisibio you were recently digging into a user configurable api issue, does this cover what you needed?

from tempo.

When we are updating we also log the current version and value.

@kvrhdn for clarity, on a successful update we log both the current data and the new data?

from tempo.

When we are updating we also log the current version and value.

@kvrhdn for clarity, on a successful update we log both the current data and the new data?

We only do so when we are updating the value through a PATCH operation. For POST requests we just store the value and don't retrieve the current value.

For store operations (POST):

• ✍🏻 we log the incoming data and version supplied by caller
• we send the data to the backend (which will validate the version is the most recent)
• ✍🏻 afterwards we log the stored data, the version and the new version

For updates (PATCH):

• we fetch the limits for the backend
• ✍🏻 we log current limits, version and the received patch
• if the patch is applied successfully, we send the data to the backend
• ✍🏻 afterwards we log the stored data, the version and the new version

So POST and PATCH will log slightly different data. We could maybe make this more consistent and verbose? We could, for instance, always log the current version, even for POST request. This would require an additional fetch request from the backend which is negligible honestly.

We also recommend to use versioning on the backend, so depending on how long your backend saves versions you can also use those to look through the history.

from tempo.