[Bug]: Trace Id mismatch between backend Cassandra tables about jaeger HOT 8 CLOSED

How should we debug this issue? Like how can we check the logs when writting entries to this Cassandra index Table https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/cassandra/schema/v002.cql.tmpl#L109?

from jaeger.

there are metrics for service_operation_index that you can inspect. And if Cassandra returns an error from insert, it should be logged:

from jaeger.

Thanks for the quick reply! But where does this line of log exist? In Jaeger-collector or Jaeger-query?

from jaeger.

writes to storage are done in the collector

from jaeger.

I only see repeated log lines like this in collector

{"level":"debug","ts":1707514221.466436,"caller":"app/span_processor.go:165","msg":"Span written to the storage by the collector","trace-id":"7250014b6827a9835a88ffb99061cdb3","span-id":"7d61cc5a23ddfded"}
{"level":"debug","ts":1707514221.2547507,"caller":"app/span_processor.go:165","msg":"Span written to the storage by the collector","trace-id":"7250014b6827a9835a88ffb99061cdb3","span-id":"6ed685f5d8b9b2bc"}
{"level":"debug","ts":1707514220.805909,"caller":"app/span_processor.go:165","msg":"Span written to the storage by the collector","trace-id":"7250014b6827a9835a88ffb99061cdb3","span-id":"3f1b83c760637957"}

For this trace 7250014b6827a9835a88ffb99061cdb3, I do see it miss writing 1 span(supposed to be 4 but I only see 3) but there isn't any log about why that span has been failed written into Cassandra. Do I have to turn something on so we can see the verbose log or the error level?

from jaeger.

I do see it miss writing 1 span(supposed to be 4 but I only see 3)

Well this is different from the problem you described above where the span was written but the service_operation_index was not. If the whole span was not written there could be other reasons, e.g. it may not have made it to the collector at all.

from jaeger.

Maybe I didn't clarify well, I think all the spans arrive in collector because I do see all the spans are being written into traces table and also the UI for this trace 7250014b6827a9835a88ffb99061cdb3 , but not all for service_operation_index table. And checking the logs from the collector, I do see 1 span is missing(so I think maybe that's the reason why it's not written into service_operation_index), but there is no error log in the collector can explain why this span is missing like Cassandra write error. All I can see is the repeated lines I show above.

from jaeger.

This was due to span dropped on OpenTelemetry side. Closed this issue.

from jaeger.