grafana / tempo Goto Github PK
View Code? Open in Web Editor NEWGrafana Tempo is a high volume, minimal dependency distributed tracing backend.
Home Page: https://grafana.com/oss/tempo/
License: GNU Affero General Public License v3.0
Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
Home Page: https://grafana.com/oss/tempo/
License: GNU Affero General Public License v3.0
tempo is a good name. frigg is fun, but not :)
change it to tempo.
The indexes are currently written to storage with no compression and assume a 16 byte (128 bit) key. Both of these things are bad.
Add compression. Drop the key length assumption.
All traces referenced by a single index record could be compressed.
This would occur when a block is completed in the ingester. Currently it "downsamples" according to a configurable value and only stores one index record for N objects. During this process we could also compress this group of N objects using gzip or snappy or something.
The WAL replays so quickly it hits limits. This is bad. Fix.
level=warn ts=2020-02-12T18:46:32.638180201Z caller=grpc_logging.go:37 method=/friggpb.Pusher/Push duration=87.8µs err="rpc error: code = Code(429) desc = per-user traces limit (local: 10000 global: 0 actual local: 10000) exceeded" msg="gRPC\n"
The ingester is designed to keep completed/flushed blocks for a configurable time period after it has been flushed. This time period exists to allow the queriers to pull new blocklists successfully before an ingester "forgets" a trace as well as to provide faster lookups on recent traces.
However, these completed blocks are stored in the wal folder. If an ingester restarts it will replay all of these traces locally and re-flush them to the backend.
The querier first attempts to ask ingesters for a trace id. If an ingester is temporarily unavailable this will timeout. Add a tight timer or, even better, execute it concurrently with a backend query.
Move compactor to its own module and make sure all backend compactors implement backend.Compactor
interface
There are ton of methods on the block objects. They are complicated and weird. Tighten things up.
Log and metric WAL replay.
blocksToCompact
returns a slice of the blocklist:
return blocklist[cursor : cursorEnd+1] ...
It should copy these values into new slice with a different backing array which it passes to compact.
Right now a trace has to make it to the headblock before it's searchable. Search the live traces.
We may have gotten too excited during the great butchering of Loki and removed more than was intended. In the distributor there is a method routeRequest
that breaks a PushRequest up by traceid and forwards it to the appropriate ingesters based on tokens/replication factor. After this the distributor sends
Unfortunately it lost the ability to check for concensus on write. Readd this. Look at the replicationSet.Do()
method from Cortex b/c I think it does this already.
To increase confidence in the data returned let's add a CRC to the object block.
See /friggdb/object.go
Need to think about the #32
Thanos compaction design doc:
https://docs.google.com/document/d/1U7SWi44pLsZrI2awWbz7gurn8kBHL7bmww3943mFC_w/edit?authuser=1#heading=h.v9f03rq2qq0b
currently the compactor has 0 disk space and is repeatedly logging this:
level=info ts=2020-02-27T14:06:47.382590293Z caller=compactor.go:144 msg="beginning compaction"
level=info ts=2020-02-27T14:06:47.382624782Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:71d77440-e5ae-46ae-9b61-a70f2414582b MinID:[0 0 0 0 0 0 0 0 0 0 0 10 189 238 6 54] MaxID:[0 0 0 0 0 0 0 0 127 254 241 7 100 74 22 76] TenantID:fake StartTime:2020-02-27 12:20:11.712776024 +0000 UTC EndTime:2020-02-27 12:33:06.386303856 +0000 UTC TotalObjects:103121}"
level=info ts=2020-02-27T14:06:47.430287526Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:bec15b74-6a7c-491d-89ee-fba9921a5a59 MinID:[0 0 0 0 0 0 0 0 0 0 0 31 65 182 242 69] MaxID:[0 0 0 0 0 0 0 0 127 253 53 140 68 27 44 94] TenantID:fake StartTime:2020-02-27 12:33:12.575160991 +0000 UTC EndTime:2020-02-27 12:46:36.583367246 +0000 UTC TotalObjects:103564}"
level=info ts=2020-02-27T14:06:47.469475667Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:fba54b79-0602-4428-85d8-59e1be48688f MinID:[0 0 0 0 0 0 0 0 0 0 0 11 214 90 88 248] MaxID:[0 0 0 0 0 0 0 0 127 248 235 225 79 137 18 27] TenantID:fake StartTime:2020-02-27 12:46:42.612485912 +0000 UTC EndTime:2020-02-27 13:00:06.413352755 +0000 UTC TotalObjects:103573}"
level=info ts=2020-02-27T14:06:47.519901468Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:7fadee40-0bfa-40d1-ab8f-d89d95ddb8f0 MinID:[0 0 0 0 0 0 0 0 0 0 0 140 62 163 208 249] MaxID:[0 0 0 0 0 0 0 0 127 254 196 124 171 249 176 73] TenantID:fake StartTime:2020-02-27 13:00:13.094668004 +0000 UTC EndTime:2020-02-27 13:13:06.398800446 +0000 UTC TotalObjects:104019}"
level=info ts=2020-02-27T14:06:48.009173722Z caller=compactor.go:140 msg="compaction complete"
This issue exists to track additional CI we'd like/need implemented.
If a wal replay fails just kill the block?
$ kc logs frigg-6b75444f9f-j87jk frigg
level=info ts=2020-02-20T00:24:52.765814165Z caller=config.go:130 msg=initialising module=server
level=info ts=2020-02-20T00:24:52.766143557Z caller=server.go:133 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-02-20T00:24:52.766349136Z caller=config.go:130 msg=initialising module=overrides
level=info ts=2020-02-20T00:24:52.767610908Z caller=config.go:130 msg=initialising module=store
level=info ts=2020-02-20T00:24:52.773444742Z caller=config.go:130 msg=initialising module=ring
level=info ts=2020-02-20T00:24:52.773584847Z caller=config.go:130 msg=initialising module=querier
level=info ts=2020-02-20T00:24:52.773694734Z caller=config.go:130 msg=initialising module=ingester
level=info ts=2020-02-20T00:24:52.774212314Z caller=client.go:212 msg="value is nil" key=collectors/ring index=1
level=info ts=2020-02-20T00:24:52.774324387Z caller=lifecycler.go:471 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2020-02-20T00:24:52.774424424Z caller=lifecycler.go:495 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2020-02-20T00:24:52.774707919Z caller=lifecycler.go:363 msg="auto-joining cluster after timeout" ring=ingester
level=error ts=2020-02-20T00:25:05.626133582Z caller=ingester.go:299 msg="error cleaning up headblock" error=null
level=error ts=2020-02-20T00:25:05.626265939Z caller=main.go:61 msg="error initialising Frigg" err="error initialising module: ingester: read 3261 but expected 14644"
As blocks are compacted the index sizes grow larger. Currently the codebase is not built to deal with this at all. Two implementation options:
The current backend design hits a number of limitations as the size of the index and bloom filters continue to grow due to compaction. Reorganize around the idea of page access to allow for bloom/index/objects to grow larger as time goes on. Research and determine a good page size that is cache friendly and easy to work with.
One of the big concerns is partial updates. Partial reads are possible by using GET
with the Content-Range
header, but uncertain if partial updates are possible with the PATCH
method. This will not matter to ingestion as it will cut relatively small blocks, but compaction will struggle as blocks become larger and larger.
Bloom
Index
Objects
https://en.wikipedia.org/wiki/External_memory_algorithm
https://en.wikipedia.org/wiki/Cache-oblivious_algorithm
Consider adding vendor/
to .gitignore
?
The compactor module kind of technically exists, but is not really independently configurable. Finish it.
The headblock is cut when a certain time or number of traces ends up in the block:
i.headBlock.Length() >= maxTracesPerBlock || i.lastBlockCut.Add(maxBlockLifetime).Before(now)
Add Max size in MBs and consider removing max traces. This plays into #32 so you probably want to work with that guy.
Add this metric.
Propagate Context appropriately and add tracing support.
Read this: https://blog.golang.org/context and do non-dumb things.
gcs block lists are currently polled serially. some improvements
2 is harder to pull off and less important. I think just implementing #1 for now will be fine.
This was confusing to me when I first started on Loki/Cortex. Make it something better.
The database code needs a way to log and publish metrics. We need to decide if it will be tightly integrated with the rest of tempo and just use the same logger as everything else. The other option being to pass these utilities as params.
Add a friggdb backend for S3.
The distributor and frigg-query both have god-awful protobuf translation code. From opencensus proto -> otel proto and from otel proto -> jaeger proto.
Soon (tm) the otel collector will implement this code correctly at which point we should replace ours.
Create a loki-canary-like application that continuously tests tracing.
The ingester instance has a go map that needs to sustain 1000s of updates per second. After #5 is in place for benchmarking consider alternatives such as:
Cortex and Loki can be intimidating to configure. Running ./app --help
dumps a huge number of options immediately on the operator. Research and improve this experience.
Add a configurable in memory cache to the /friggdb/backend/cache
package.
Blocklist and Bloom should be cached in memory. Hold off on Object. I would love to see GroupCache work, but Ristretto is also an option.
In different scenarios partial blocks can be stranded with neither a meta.json
nor a meta.compacted.json
. The compactor should also act as a janitor and delete these blocks from the backend.
Currently a byte slice is used everywhere for the ID. Benchmark using 2 uint64's (idHigh, idLow) instead of the byte slice to represent the 128 bit trace id.
... and airline food?
Review the limits code that just came along with the Loki package. Make it make sense for frigg.
Add CI. At the minimum it should:
Configured with a 60m maxCompactionRange the compactor is choosing blocks over 1 hour apart:
level=info ts=2020-02-27T14:56:33.118545538Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:ef16962b-85b0-4d4c-96cf-25e4efbf775b MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 0 127 53 218 50 155 24 40] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:1114175}"
level=info ts=2020-02-27T14:56:33.215026831Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:3c2f9209-2767-44cb-8c40-d893f9c082f4 MinID:[0 0 0 0 0 0 0 0 0 11 85 40 191 206 203 8] MaxID:[0 0 0 0 0 0 0 0 4 184 161 121 117 202 81 175] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:426499}"
level=info ts=2020-02-27T14:56:33.309527672Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:34292379-20a3-4f12-bc98-f305da331c1b MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 0 8 56 75 22 24 15 158] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:1152174}"
level=info ts=2020-02-27T14:56:33.403106762Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:15d841e8-a4ec-43ae-adff-71b731686a5a MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 3 236 175 12 175 121 214 105] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:18:24.28298673 +0000 UTC TotalObjects:202623}"
level=info ts=2020-02-27T15:01:48.041123907Z caller=compactor.go:266 msg="writing compacted block" block="&{Version:v0 BlockID:1c2902f1-59e0-42b5-8efc-432e854ffdd5 MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 0 13 28 120 172 46 134 68] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:1447736}"
This method is critical and untested and relatively easily testable. Add tests.
blocksToCompact
currently just finds the first 4 blocks that fall within the compaction range. This will not be sufficient for HA mode, multiple ingesters, etc.
At least add:
Compaction Duration and Compaction Stop Duration seem capped at ~8 minutes. I just saw a 2 million traces get compacted over ~25 minutes and it was reported at 8.
Increase range of these histograms. Maybe 30 minutes?
GCS backend is important and untested.
TestJanitorCleanupOrder
was commented out b/c it's flaky. Data as reported from the fileinfo objects is often not what is expected.
tempo starts up etcd goroutines even when not using the etcd client. Prevent these from being started. Possibly requires a PR to Cortex.
goroutine 57 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0001c9b40)
/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x80
goroutine 106 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0001f0a60)
/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x80
goroutine 109 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0001f0c00)
/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x80
Response codes/metrics are broken. Investigate and improve.
Some examples:
-race
catches a unsafe pointer conversion in godirwalk:
fatal error: checkptr: unsafe pointer conversion
goroutine 126 [running]:
runtime.throw(0x13ff65d, 0x23)
/usr/local/go/src/runtime/panic.go:1112 +0x72 fp=0xc00006c860 sp=0xc00006c830 pc=0x463162
runtime.checkptrAlignment(0xc000299f18, 0x136c0e0, 0x1)
/usr/local/go/src/runtime/checkptr.go:18 +0xb7 fp=0xc00006c890 sp=0xc00006c860 pc=0x434607
github.com/karrick/godirwalk.(*Scanner).Scan(0xc000326090, 0x3d)
/home/joe/go/pkg/mod/github.com/karrick/[email protected]/scandir_unix.go:144 +0x11e fp=0xc00006c968 sp=0xc00006c890 pc=0x11c7d0e
github.com/karrick/godirwalk.walk(0xc0001ae300, 0xe, 0xc0000b0bd0, 0xc00006cc10, 0x0, 0x0)
/home/joe/go/pkg/mod/github.com/karrick/[email protected]/walk.go:270 +0x40c fp=0xc00006cab8 sp=0xc00006c968 pc=0x11c92ec
github.com/karrick/godirwalk.Walk(0xc0001ae300, 0xe, 0xc00006cc10, 0xc000075240, 0x0)
/home/joe/go/pkg/mod/github.com/karrick/[email protected]/walk.go:204 +0x36b fp=0xc00006cb88 sp=0xc00006cab8 pc=0x11c8b2b
github.com/grafana/frigg/friggdb/backend/cache.clean(0xc0001ae300, 0xe, 0x1, 0x1, 0x0, 0x0, 0x0)
/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:84 +0x214 fp=0xc00006cc60 sp=0xc00006cb88 pc=0x11e6274
github.com/grafana/frigg/friggdb/backend/cache.TestJanitor(0xc0001b0900)
/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache_test.go:109 +0xaf0 fp=0xc00006ced0 sp=0xc00006cc60 pc=0x11e80f0
testing.tRunner(0xc0001b0900, 0x1426ba0)
/usr/local/go/src/testing/testing.go:992 +0x1ec fp=0xc00006cfd0 sp=0xc00006ced0 pc=0x5bd74c
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00006cfd8 sp=0xc00006cfd0 pc=0x4976b1
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1043 +0x661
goroutine 1 [chan receive]:
testing.(*T).Run(0xc0001b05a0, 0x13e7257, 0xb, 0x1426ba0, 0x1)
/usr/local/go/src/testing/testing.go:1044 +0x699
testing.runTests.func1(0xc0001b05a0)
/usr/local/go/src/testing/testing.go:1285 +0xa7
testing.tRunner(0xc0001b05a0, 0xc000241d50)
/usr/local/go/src/testing/testing.go:992 +0x1ec
testing.runTests(0xc0001d2060, 0x1d37500, 0x2, 0x2, 0x0)
/usr/local/go/src/testing/testing.go:1283 +0x528
testing.(*M).Run(0xc000190400, 0x0)
/usr/local/go/src/testing/testing.go:1200 +0x300
main.main()
_testmain.go:46 +0x224
goroutine 127 [select]:
github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor.func1(0xc00018d1a0)
/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:56 +0x6f3
created by github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor
/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:53 +0x4d
goroutine 69 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0003007e0)
/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x4b2
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0xfc
goroutine 125 [select]:
github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor.func1(0xc00018cff0)
/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:56 +0x6f3
created by github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor
/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:53 +0x4d
goroutine 32 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000183820)
/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x4b2
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0xfc
goroutine 77 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000300a00)
/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x4b2
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0xfc
****
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.