Comments (5)
Interested in taking this up, if there are any further details that can be added!
from tempo.
We should already be logging some of this information in the API layer. For example when storing new limits: https://github.com/grafana/tempo/blob/main/modules/overrides/userconfigurable/api/api.go#L104
We currently log timestamp, tenant, the new value and the new version. When we are updating we also log the current version and value.
Is there anything else you would like to see? We could also log stuff like IP address and any headers, but I'm not sure how valuable that is since Tempo is most often deployed behind a gateway.
Finding these logs should be straightforward with Loki:
{namespace=~"tempo.*"} |= "component=overrides-api" != "method=GET"
from tempo.
Ah, I didn't realize we had solid audit logging here. @mdisibio you were recently digging into a user configurable api issue, does this cover what you needed?
from tempo.
When we are updating we also log the current version and value.
@kvrhdn for clarity, on a successful update we log both the current data and the new data?
from tempo.
When we are updating we also log the current version and value.
@kvrhdn for clarity, on a successful update we log both the current data and the new data?
We only do so when we are updating the value through a PATCH operation. For POST requests we just store the value and don't retrieve the current value.
For store operations (POST):
- ✍🏻 we log the incoming data and version supplied by caller
- we send the data to the backend (which will validate the version is the most recent)
- ✍🏻 afterwards we log the stored data, the version and the new version
For updates (PATCH):
- we fetch the limits for the backend
- ✍🏻 we log current limits, version and the received patch
- if the patch is applied successfully, we send the data to the backend
- ✍🏻 afterwards we log the stored data, the version and the new version
So POST and PATCH will log slightly different data. We could maybe make this more consistent and verbose? We could, for instance, always log the current version, even for POST request. This would require an additional fetch request from the backend which is negligible honestly.
We also recommend to use versioning on the backend, so depending on how long your backend saves versions you can also use those to look through the history.
from tempo.
Related Issues (20)
- Guidance on traceql metrics timeout issue
- Support EKS Pod Identity feature HOT 4
- Bug in wal.go causes c.BlocksFilepath config to be overwritten in tempo.yaml HOT 5
- compile issue when make vendor-check HOT 2
- No query results for older telemetry HOT 3
- Tenant error when S3 bucket analytics turned on for S3 backend storage bucket HOT 1
- [DOC] Clarify endpoint access for consistent hash rings doc
- generator push to prometheus pushgateway always fails "HTTP status 400 Bad Request: snappy: corrupt input" HOT 2
- [DOC] Review and update the Parquet docs
- AWS S3 HTTP 4xx client errors HOT 13
- Adjust client/server span times to account for clock skew HOT 3
- Error on kubernetes (microk8s) with persistence enabled HOT 7
- Data retention for 30 days in grafana tempo do not work HOT 5
- User-configurable overrides refuses span-metrics-size as valid processor
- tempo-distributed does not support OpenShift installation out of the box because of missing securitycontextconstraint HOT 2
- aws s3 storage backend:List HOT 2
- pusher failed to consume trace data due to "context canceled" HOT 1
- Monolithic deployment with s3 storage fails with "mkdir /var/tempo/wal: permission denied" HOT 1
- Periodic errors during searching for traces HOT 1
- [DOC] Release notes for 2.6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tempo.