jaegertracing / jaeger Goto Github PK

Hi,

Trying to figure out if we could use Jaeger in our product tracing collecting. So far getting working by setting a single collector via -collector.host-port. It seems the way to setup multiple collectors is by some auto discovery mechanism. But I have no idea how this works? Please help.

Thanks.
-Kui

The jaeger-agent exposes the Zipkin thrift service only on a UDP port. Because thrift doesn't support a UDP transport out of a box it makes using Jaeger for Zipkin instrumented services difficult. It also makes migrating from Zipkin to Jaeger a nonstarter.

We should explore adding a HTTP or TCP thrift Zipkin receiver to jaeger-agent for compatibility and have tests with existing Zipkin instrumentation. Alternatively, we can refactor Zipkin conversions to a separate agent to reduce complexity in jaeger-agent

Sorry for hijacking issues but this looks like the only way to contact the team.

I don't know much about go. Could you share some basic details on how to start jaeger daemon/UI?

Thanks in advance!

REST API will be necessary for collecting span from a web browser.

see jaegertracing/jaeger-client-node#109

The pattern of using Jaeger Agents runnning on the same host does not fit (our) usecase of running (python) functions as FAAS in AWS Lambda.
The requirement of running a seperate process is a showstopper.
Is it possible to have the agent running in another host? or in some work-around this issue in some way?

Btw. looking forward to OsCon workshop!!! :-)

For some reason we set it to 10 seconds in some libs. This is an umbrella issue to check that all clients are sending at least every second (they will send more often if the data exceeds 64k)

UI from all-in-one image fails to start.

docker run -it --rm --network=host jaegertracing/all-in-one

Stack trace from browser:

main.7c74e600.js:2 Uncaught Error: React.PropTypes type checking code is stripped in production.
    at r (main.7c74e600.js:2)
    at Object.r (main.7c74e600.js:59)
    at Object.<anonymous> (main.7c74e600.js:45)
    at t (main.7c74e600.js:1)
    at Object.<anonymous> (main.7c74e600.js:45)
    at t (main.7c74e600.js:1)
    at Object.<anonymous> (main.7c74e600.js:25)
    at t (main.7c74e600.js:1)
    at Object.<anonymous> (main.7c74e600.js:45)
    at t (main.7c74e600.js:1)

I'm trying to run the collector module individually, and it seems it always requires cassandra, even if the -span-storage.type memory option is passed.

$ CGO_ENABLED=0 GOOS=linux installsuffix=cgo go build -o ./cmd/collector/jaeger-collector-linux ./cmd/collector/main.go
$ ./cmd/collector/jaeger-collector-linux -span-storage.type memory
{"level":"fatal","ts":1492096339.4118035,"caller":"collector/main.go:55","msg":"Unabled to set up builder","error":"MemoryStore is not provided","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/mnt/storage/jpkroehling/Projects/go/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/mnt/storage/jpkroehling/Projects/go/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/mnt/storage/jpkroehling/Projects/go/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:208\nmain.main\n\t/home/jpkroehling/Projects/go/src/github.com/uber/jaeger/cmd/collector/main.go:55"}

Changing it to be almost 1-1 with the server bootstrap code from the standalone module, the message becomes:

{"level":"fatal","ts":1492096398.6600523,"caller":"collector/main.go:52","msg":"Unabled to set up builder","error":"Cassandra not configured","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/mnt/storage/jpkroehling/Projects/go/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/mnt/storage/jpkroehling/Projects/go/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/mnt/storage/jpkroehling/Projects/go/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:208\nmain.main\n\t/home/jpkroehling/Projects/go/src/github.com/uber/jaeger/cmd/collector/main.go:52"}

There does not exist a cql file that fits the profile of the existing cassandra/spanstore package logic.

When querying for traces using serviceName, operationName and a tag with the default LIMIT of 20, some results might be omitted.

This is because of this logic which does the following:

1. Retrieve all traceIDs matching the operation name
2. Retrieve all traceIDs matching tags
3. Intersect 1 & 2

Because Cassandra doesn't guarantee ordering, this could eliminate results.

I propose that we do the following instead (or in addition to what we do now),

1. Retrieve all traceIds matching tags
2. Filter by operation name

The reason for retrieving traceIds matching tags first targets the use case when somebody is searching for a jaeger-debug-id or some other tag with low cardinality, guaranteeing them a result when it exists.

It turns out that -query.static-files requires trailing slash. Without it instead of index.html a HTTP 404 is returned and it's not clear why. It would be good to fail at startup if it's missing.

Since reverting away from SASI indices, we need to artificially increase our limit number during queries and then shed off excessive traceIDs after results from cassandra.

We don't really define our API anywhere, and we should do so. This is probably a good issue to bring up the idea of using swagger for self-documentation.

Currently buckets is a hardcode const string - (0,1,2,3,4,5,6,7,8,9).

This should be calculated at runtime. This will modify some queries and the new query behavior will need to be tested.

https://github.com/fabric8io/kubernetes-zipkin

Consider using custom theme for readthedocs like http://rexray.readthedocs.io/ (via mkdocs)

During the initial Docker/Kubernetes/OpenShift , we took some shortcuts regarding Cassandra and they need to be sorted out.

Points to solve:

• Use a more appropriate Cassandra Docker image
• Wrap the Docker entry point to run a nodetool drain on shutdown, as suggested by @jsanda
• Figure out the best approach regarding the replication strategy and datacenter configuration

Currently several implementations are reporting tags sampling.type=lowerbound, sampling.param=$rate when lower bound sampler is triggered, where $rate is the sampling rate of the probabilistic sampler. We should change it to report the actual lower bound value, for consistency.

• Node client
• Go client
• Python client
• Java client

Instead of an arbitrary "RESTful" HTTP API service, I think a better solution would be to implement the query service over GraphQL.

Here is my sample implementation of what the service would like:

https://launchpad.graphql.com/wxn5zk8mz
HTTP Endpoint: https://wxn5zk8mz.lp.gql.zone/graphql

Some of the benefits:

• Clients can request what they want, resulting in smaller payloads and less unused data. Really beneficial in the list view of traces, we don't need to guess what a client needs as designs/needs may change.
• Documentation for free (via introspection) and strongly typed API backed by a spec: https://facebook.github.io/graphql/
• A descriptive data model to be more expressive with complex queries. (more robust the URL encoded params)
• GraphQL queries can be issues via HTTP, TChannel, or any communication protocol.
• Simple computed fields and object type relationships

This will help solve the following issues as well:
https://github.com/uber/jaeger/issues/158
https://github.com/uber/jaeger/issues/123

The hotrod UI 404s with the following

go get github.com/uber/jaeger/examples/hotrod && hotrod all

Hi,

Is the adaptive sampling inside collector supported? I could not figure out how I could set/change sampling strategy via collector. Please help.

Thanks.
-kui

Whenever a baggage item is set, we would like to record the baggage inside the span where it was set. This allows increased visibility into which service set the baggage at what point in the call graph.

• Go jaegertracing/jaeger-client-go#153
• Java jaegertracing/jaeger-client-java#189
• Python jaegertracing/jaeger-client-python#54
• Node jaegertracing/jaeger-client-node#129

Currently we are using an archaic methodology for storing and querying against duration (against hourly buckets that are not timestamps). We should revisit this and fix up both the code and the cassandra schema

I would like to propose wire format change. There are two things in my mind:

• there is uber in headers: uber-trace-Id for span context and baggage prefix uberctx. This does not tell anything about jaeger. This could be changed to jaeger-
• parentId is redundant when using the new model

The change will be problematic due to a large deployment of mixed services (jaeger and zipkin thrift used simultaneously).

I'm not sure how this could be done in the safest manner. Maybe implement new default inejector/extractor which would first extract/inject old and new headers and then after some time remove old headers.

This is also related to #121

It has been 11 days since last updated.

Logs are sampled, and any more than ~100 logs lines per second would be dropped.
Change this so that all logs are output.

e.g. log-level and runtime-metrics-frequency in flags.go.

Current Behavior

jaeger-collector stores spans with span context flags set to 0.

Expected Behavior

jaeger-collector should not store spans with span contexts where the flag value is 0 because these spans failed the sampling tests.
This protects the jaeger-collector from buggy clients that assume that the collector only stores sampled spans.

Reproduction steps

To reproduce, build jaeger, and run locally as such

make build_ui
cd cmd/standalone/
go run ./main.go --span-storage.type=memory --query.static-files=../../jaeger-ui-build/build/

Then use yab to make a request as such. Use jaeger.thrift from here.

yab -A trace-id:9 -y payload.yaml

payload.yaml contains the following

---
method: "Collector::submitBatches"
peer: "localhost:14267"
request:
  batches:
    -
      process:
        service_name: flags-test
        tags:
          -
            key: jaeger.hostname
            vStr: localhost
            vType: string
      spans:
        -
          duration: 1294
          flags: 0
          operation_name: arbitrary
          parent_span_id: 0
          span_id: 1
          start_time: 1494509316000000
          tags:
            -
              key: span.kind
              vStr: server
              vType: string
          trace_id_high: 0
          trace_id_low: "${trace-id:1}"
service: jaeger-collector
thrift: jaeger.thrift

Note that flags is set to 0, yet the trace is retrievable using the UI at http://localhost:16686/trace/9

It seems I got "everything" running, except that I see a message like no peers available on the standalone logs.

The server has been started like this: go run cmd/standalone/main.go -span-storage.type memory -query.port 3001
The UI has been started with npm start
And the example with cd examples/hotrod/ ; go run main.go all

After switching the LogSpans to true and clicking on Japanese Deserts, I can see this on the example's logs:

[I] 2017-04-04T17:20:54Z HTTP request received service=frontend method=GET url=/dispatch?customer=731&nonse=0.29873183383130686
[I] 2017-04-04T17:20:54Z Getting customer service=frontend component=customer_client customer_id=731
[I] 2017-04-04T17:20:54Z HTTP request received service=customer method=GET url=/customer?customer=731
[I] 2017-04-04T17:20:54Z Loading customer service=customer component=mysql customer_id=731
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:598cecebbc62b0d6:221186770f1b11e7:1 service=customer
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:221186770f1b11e7:5c56cc1caa97d99d:1 service=customer
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:221186770f1b11e7:5c56cc1caa97d99d:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:5c56cc1caa97d99d:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Found customer service=frontend customer=&{ID:731 Name:Japanese Deserts Location:728,326}
[I] 2017-04-04T17:20:54Z Finding nearest drivers service=frontend component=driver_client location=728,326
[I] 2017-04-04T17:20:54Z Searching for nearby drivers service=driver location=728,326
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 2
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 1
count till error 0
[E] 2017-04-04T17:20:54Z redis timeout service=driver driver_id=T752378C error=redis timeout
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
[E] 2017-04-04T17:20:54Z Retrying GetDriver after error service=driver retry_no=1 error=redis timeout
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 4
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 3
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 2
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 1
count till error 0
[E] 2017-04-04T17:20:54Z redis timeout service=driver driver_id=T726422C error=redis timeout
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
[E] 2017-04-04T17:20:54Z Retrying GetDriver after error service=driver retry_no=1 error=redis timeout
count till error 4
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 3
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
count till error 2
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
[I] 2017-04-04T17:20:54Z Search successful service=driver num_drivers=10
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=driver
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:447d52ceebdcc70:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Found drivers service=frontend drivers=[{DriverID:T708733C Location:464,677} {DriverID:T780476C Location:523,750} {DriverID:T752378C Location:226,57} {DriverID:T761341C Location:719,409} {DriverID:T787199C Location:780,824} {DriverID:T745302C Location:897,908} {DriverID:T726422C Location:699,836} {DriverID:T713475C Location:4,833} {DriverID:T748433C Location:784,369} {DriverID:T743274C Location:603,664}]
count till error 1
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=226,57 dropoff=728,326
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=464,677 dropoff=728,326
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=523,750 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=464%2C677
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=523%2C750
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=226%2C57
+2.189036e+000
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:687b9b4a7623cadb:630382228a29730c:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:687b9b4a7623cadb:630382228a29730c:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:630382228a29730c:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=719,409 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=719%2C409
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:114d4d6403d035ab:29cbce8d0c38fc9e:1 service=route
+2.000000e+000
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:114d4d6403d035ab:29cbce8d0c38fc9e:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:29cbce8d0c38fc9e:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=780,824 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=780%2C824
+6.719845e+000
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:2ebf99de852ba41:3c20c4208c588fc1:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:2ebf99de852ba41:3c20c4208c588fc1:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:3c20c4208c588fc1:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=897,908 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=897%2C908
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:29566611ca466286:69cbfbac4eb59726:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:29566611ca466286:69cbfbac4eb59726:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:69cbfbac4eb59726:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=699,836 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=699%2C836
+6.614381e+000
+3.800338e+000
+7.494734e+000
+8.702753e+000
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:66957a3805a13c83:475bf32b7d802d8b:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:66957a3805a13c83:475bf32b7d802d8b:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:475bf32b7d802d8b:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=4,833 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=4%2C833
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:614d46e0a8120d8e:3680afd13cd2692c:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:614d46e0a8120d8e:3680afd13cd2692c:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:3680afd13cd2692c:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=784,369 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=784%2C369
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:6c76f6eb56c42165:36e83173bdfa4604:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:6c76f6eb56c42165:36e83173bdfa4604:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:36e83173bdfa4604:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Finding route service=frontend component=route_client pickup=603,664 dropoff=728,326
[I] 2017-04-04T17:20:54Z HTTP request received service=route method=GET url=/route?dropoff=728%2C326&pickup=603%2C664
+5.915291e+000
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:2965df47a1576fe6:12c1f6e4f2ee0c6d:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:2965df47a1576fe6:12c1f6e4f2ee0c6d:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:12c1f6e4f2ee0c6d:49db0efb135614c6:1 service=frontend
+5.578001e+000
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:47b1c5655a3e34ce:34dc248a89e3d5f0:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:47b1c5655a3e34ce:34dc248a89e3d5f0:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:34dc248a89e3d5f0:49db0efb135614c6:1 service=frontend
+3.835140e+000
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:13a899ccba5f141a:9383933e7be8001:1 service=route
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:13a899ccba5f141a:9383933e7be8001:1 service=frontend
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:9383933e7be8001:49db0efb135614c6:1 service=frontend
[I] 2017-04-04T17:20:54Z Found routes service=frontend routes=[{driver:T708733C route:0xc42047d410 err:<nil>} {driver:T780476C route:0xc420239a10 err:<nil>} {driver:T752378C route:0xc42014c960 err:<nil>} {driver:T787199C route:0xc4202f7ad0 err:<nil>} {driver:T726422C route:0xc4202f7f50 err:<nil>} {driver:T761341C route:0xc42047dad0 err:<nil>} {driver:T745302C route:0xc420176420 err:<nil>} {driver:T713475C route:0xc42016c480 err:<nil>} {driver:T748433C route:0xc42014d980 err:<nil>} {driver:T743274C route:0xc4201772f0 err:<nil>}]
[I] 2017-04-04T17:20:54Z Dispatch successful service=frontend driver=T708733C eta=2m0s
[I] 2017-04-04T17:20:54Z Reporting span 49db0efb135614c6:49db0efb135614c6:0:1 service=frontend

And this can be seen on the standalone's logs:

{"level":"error","ts":1491326454.9346092,"msg":"Could not submit zipkin batch","error":"no peers available"}
{"level":"error","ts":1491326454.934651,"msg":"Could not submit zipkin batch","error":"no peers available"}
{"level":"error","ts":1491326454.9348862,"msg":"Could not submit zipkin batch","error":"no peers available"}
{"level":"error","ts":1491326454.935086,"msg":"Could not submit zipkin batch","error":"no peers available"}
{"level":"error","ts":1491326454.9354107,"msg":"Could not submit zipkin batch","error":"no peers available"}
{"level":"error","ts":1491326454.9355452,"msg":"Could not submit zipkin batch","error":"no peers available"}
{"level":"error","ts":1491326455.935414,"msg":"Could not submit zipkin batch","error":"no peers available"}
{"level":"error","ts":1491326455.9359097,"msg":"Could not submit zipkin batch","error":"no peers available"}

I'm not sure the error message is expected, but even if it is, I can't seem to be able to see the traces on the standalone's UI (and the service select box is empty).

Dapper paper explains near-zero intrusive tracing for language like c++, java. For golang, what I know is passing down the tracing needed info via context. This means the context has to be explicitly passed down, without any interruption.This makes migrating old code base to tracing difficult, as there is no simple way to enforce that context is correctly passing down. Any suggestion on better way to achieve this?

Thanks.
-Kui

I try to add JSON content to logs like that :

span.LogFields(log.String("get_platform", string(resp)))

[...]

data := string(resp)
glog.V(0).Infof("Data: %s", data)
span.LogFields(log.String("response", data))

Output logs are :

Data: [{"biosUUID": "422523e2-4bf4-917f-a3ba-e65574cc18ef", "description": "4ef78da2-3552-11e7-99f4-0800270a9ecb", "disks": [{"capacity": 10, 
[...]

On UI, first log (get_platform) seems to be correctly printed, with key = "get_platform", and a value in JSON.
But second and third logs, "response" and "virtual_machines", key are "event" and it's not JSON content but string as values.
I don't really understand the different behavior with the same code. Only responses are differents.

Any idea how can't i fix that ?
Thanks

Hi, we are doing some research on jaeger for tracing.
Is there a way to boot from a conf file?

Assuming there is a cassandra instance we can point to, we need to provide working docker images for:

• The Jaeger Agent
• The Jaeger Collector
• The Jaeger Query Service
• This task is complete when proof can be provided of all 3 services run, connect to a cassandra instance, and can consume traffic/display UI results.

As storage interfaces are modified, we will require a make target for regenerating mocks.

hello , I want to run the "Jaeger" locally , but I can not fing "schema.cql" .

When a package has .go files but no test files, it is not counted towards the code coverage, giving misleading total. The proposal is to do something similar to yarpc/yarpc-go#1043:

• we already use .nocover files to skip packages with this file from running coverage
• we should add another validation script that will fail if a package has .go files, no .nocover file, and has no tests

When using zipkin.thrift error tag is set as a string value (maybe every value is set as a string). This may affect future data analysis pipeline.

Fix for UI was done here https://github.com/uber/jaeger-ui/pull/32

Some command line tools like yab are capable of starting a trace before sending the request. They also allow setting additional headers like jaeger-debug-id, but the Jaeger client libs ignore (at least Java client does) it since there is already a trace context.

• make sure all clients respect the header regardless of the presence of the trace
• add a test for this behavior to end-to-end test

• Cloned the jaeger repo
• make install_examples -- worked
• cd examples/hotrod -- worked
• go run ./main.go all -- failed

/Users/Chris/go/src/github.com/uber/jaeger/examples/hotrod/cmd/customer.go:28:2: code in directory /Users/Chris/go/src/github.com/uber-go/zap expects import "go.uber.org/zap"

Not familiar with Go so could be missing something totally obvious ... but thought i would log this as an issue as other noobs like me will likely hit this too.

Add integration tests for zipkin.thrift over http using popular zipkin instrumentation libraries.

From jaegertracing/jaeger-client-java#172 (comment):

name:       jaeger-rpc.{requests, request_latency}
service:    myservice 
endpoint:   fooendpoint 
result:     {ok,err}
error:      {4xx,5xx,app-error,sys-error,rate-limit,timeout,...}
transport:  {http,tchannel,...}
direction:  {inbound,outbound}

Desired workflow:

1. start docker image
2. run hotrod
3. see traces in the UI

Action items:

• Dockerfile for a single image running all backend and UI
  • Consolidate all default ports into a single range
• Add integration tests that validate the docker image is actually operational
  • curl the query service with debug flag, it should create a trace
  • curl it again looking for that trace by correlation ID
  • curl UI endpoint for trace HTML and maybe do a dumb string search for the correlation ID
• Publish the image to DockerHub

Wondering if the span context key and baggage prefix used for propagation could be changed from using 'uber' to being 'jaeger'?

For example:
https://github.com/uber/jaeger-client-node/blob/master/src/constants.js#L74-L77
https://github.com/uber/jaeger-client-java/blob/master/jaeger-core/src/main/java/com/uber/jaeger/propagation/TextMapCodec.java#L37-L40

Currently jaeger has problems with traces which exceed few hundred spans.
I noticed following:

• searching for traces fetches very large payloads - basically /api/traces returns all matching traces along with spans, tags and logs, but the UI shows only a summary like number of spans per service, duration, etc. The search response should contain only the details required to render the screen. Currently the UI becomes very sluggish when the search result contains a large trace or does not render the results at all.
• showing trace view for a very large trace has similar performance issues - the response payload is very large, the UI is sluggish or does not render.
  • In this case it would make sense to only fetch some part of the trace duration - e.g. first 200 spans. Or top 200 spans when counting distance from the root span. Some way to expand additional spans could allow to gradually navigate the trace. Or maybe some span filters would do the job.
  • another idea that comes to mind would involve dedicating some bits in the span id for "distance from root" to make it possible to search for the top spans efficiently
  • one more way to improve it would be to fetch spans without tags/log and fetch those only when "tags" or "logs" section on a span is "expanded"
• paylods for large traces are pretty large (10MB, 100MB and more) and the browser has problems with processing them

I was able to view traces that had more than 1k spans (it was laggy and took some time), but not ~5k.

The traces I encountered were usually created by long iterative processes - e.g. recalculating something on thousands of records.
Another more pathological reason is bad communication design - an example would be a process which emits thousands of messages instead of using some batching.

Related post on jaeger-tracing group: here

hi, we currently have a kafka cluster which stores zipkin spans. Does jaeger collector support consume zipkin data from kafka and transform to jaeger data model into cassandra?

Provide a mechanism for distributing reported span data from the Jaeger server. This can be used to trigger post processing of the tracing information.

As discussed here: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jaeger-tracing/2ClHbZ3BwBg/tnGVfmQbBwAJ

is there any plan to support Elasticsearch as additional backend? I will be useful for those who want to leverage there existing elasticsearch cluster. Zipkins and Hawkular APM uses elasticsearch as backend.

Under certain conditions, the collector might not return adaptive sampling strategies for certain operations which in turn means the operation will continue to use the existing sampling strategy and not be updated.

https://github.com/uber/jaeger-client-go/blob/master/sampler.go#L361 doesn't update existing strategies that are not apart of the passed in strategies.

Issue exists in all clients

• Go jaegertracing/jaeger-client-go#413
• Java
• Node
• Python
• C#
• C++