OpenTelemetry is correctly extracting SpanKind information, but our exporter is not exporting it to Cloud Trace.

Hello, I work on the Bank of Anthos sample application. In July, we added the OpenTelemetry exporter and trace propagator to 3 Python services (all using flask/gunicorn, all running inside containers on GKE).

We have determined that the OT exporter seems to be causing multiple seconds of latency per request (findings here). We upgraded OT to the latest version (0.13b0) but we're still seeing latency in the app.

My hypothesis is that there is a process in the exporter that is slow on a per-request basis, or there is something going on with our GKE environment/Trace API authentication that is causing the latency.

Wondering if you've seen anything similar, or are doing any performance testing on these libraries. Thank you!

Run the test suite on Windows as well and fix any issues

"Homepage" link in https://pypi.org/project/opentelemetry-exporter-google-cloud/ is broken. The correct URL would be https://github.com/GoogleCloudPlatform/opentelemetry-operations-python/tree/master/opentelemetry-exporter-google-cloud

If even one of the spans passed in a batch to CloudTraceExporter is malformed, when it attempts to export these, the Cloud Trace backend will complain and as a result none of the spans in the batch will be exported. A proposed solution is a divide and conquer algorithm that, given an array of spans (possibly a subarray) will attempt to export the left and right halves of the array. If it succeeds, continue, otherwise if it fails recurse on the half that failed.

def exportDivide(arr_of_spans):
left = arr_of_spans[:len(arr_of_spans)/2], right = arr_of_spans[len(arr_of_spans)/2:]
try left export
if left export failed: exportDivide(left)
try right export
if right export failed: exportDivide(right)

This will require O(klogn) calls the backend where k is the number of malformed spans and n is the number of spans. One issue with this approach is that if k is large, say k == n, then we would do O(nlogn) calls to the backend which is far too slow. The size of the stack also shouldn't be an issue since there would be log(n) recursive calls to the function.

There is a bunch of duplicated code that we need to work out how to share. This is tricky because we publish separate packages for the metrics exporter, trace exporter, and tools (propagator and resource detector). There is also BaseExporterIntegrationTest which could be useful in test packages.

The OpenTelemetry SDK ships with propagators for Zipkin and Jaeger. OpenCensus also ships with a propagator for Cloud Trace (x-cloud-trace-context): https://github.com/census-instrumentation/opencensus-python/blob/master/opencensus/trace/propagation/google_cloud_format.py

We should add a Cloud Trace propagator to this exporter.

-> this could offer customers the possibility of being compatible with existing services that are not yet instrumented using OpenTelemetry by e.g. being able to extract either header format and inject both headers formats.

The documentation should mention that SimpleExportSpanProcessor is slow and probably only useful for debugging purposes. Instead use BatchExportSpanProcessor in production. I think the OTel python documentation could also be updated to explain this or at least an FAQ.

Related #75

Instead, use OTel resource semantic conventions. See Go reference impl https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/402639db27fdf8e5143cfd8a871d40cbe10d4bd2/exporter/metric/metric.go#L365

OpenTelemetry's Resource.labels property actually copies the labels dict: https://github.com/open-telemetry/opentelemetry-python/blob/8bc7786b191b4f1b1f161de983db71643bf9f51e/opentelemetry-sdk/src/opentelemetry/sdk/resources/__init__.py#L41-L43

So each access is here is copying the dict:

If the goal of Resource doing this is to achieve immutability, maybe it can be updated upstream to use a FrozenDict/ImmutableDict library?

Need to update code introduced in #83 when these issues are closed:
googleinterns/cloud-operations-api-mock#58
googleinterns/cloud-operations-api-mock#56

There are some fields of the cloud trace API that can be set, but we ignore. For example same_process_as_parent_span and child_span_count. We'll need to look into how difficult it is to extract this info from OpenTelemetry and then work to send it over in the exporter

These breaking changes:

• open-telemetry/opentelemetry-python#1082
• open-telemetry/opentelemetry-python#1085
• open-telemetry/opentelemetry-python#596 - label_keys param remove from meter.create_metric(), need to update examples

These packages should have some version specifier to prevent any issues with breaking changes.

See #17 for discussion.

#17 has tox install opentelemetry-api/sdk packages to run tests against from a pinned commit. This keeps us up developing closely with opentelemetry-python repo before GA. After GA where there are fewer breaking changes, we should switch to actually testing against released versions of those packages.

OpenTelemetry span status needs to be mapped correctly to GCT's span status (google.rpc.Code). Currently we just copy the int value over:

But OTel has OK = 1 but google.rpc.Code has OK = 0

Hi,

I'm on the Cloud Client Libraries team. Just wanted to give you folks a heads up that we're in the process of making major version bumps to the Python libraries with breaking changes. I'm about to cut a release for google-cloud-trace and monitoring will follow in a week or two.

Please pin the dependencies on these libraries to less than the next major to avoid breaking users.

google-cloud-monitoring<2.0.0
google-cloud-trace>=0.24.0, <1.0.0

Thanks!

3.4 was dropped from the python SDK open-telemetry/opentelemetry-python#1099

3.5 is no longer supported upstream opentelemetry-python, so needs to be dropped here

Cloud Trace Client API specifies some limits on # of attributes, byte size max of attributes, # of events, etc. Need to look into how the API deals with these if the limits are exceeded.

I am thinking to just publish them on readthedocs.io but will explore other options as well. We are planning to leave a reference to these new docs in the official opentelemetry-python docs, but remove everything else.

Following the GCP spec, we should have the following layout
docs
examples
exporters

 metric
 trace

tools

 propagator
 resource_detector (pending final decision in open-telemetry/oteps#111)

with tools in different packages then the exporters.

Currently if we try to publish new metric data more than once every 10 seconds, we just ignore the publish attempt until the time has been greater than 10 seconds. However, when metrics are stateless, this will lead to data loss as the aggregation is thrown out on the exporter side but also reset by the batcher.

The solution to this is to store the checkpoints from the records of the previous publish attempt and merge those checkpoints into the records of the next export attempt.

Hi there, I see https://pypi.org/project/opentelemetry-exporter-google-cloud/ is still on 0.16b1. Is there any chance of a new release?

I'd find it really useful to have 0.17b0 there.

I'm intending to add a conda-forge feedstock for this package, but other packages like https://anaconda.org/conda-forge/opentelemetry-api are already at opentelemetry version 0.17b0, skipping 0.16b1.

BatchSpanProcessor is (effectively) a requirement in production, to avoid making tons of rpc requests. How it works is it spins up a worker thread that stores spans until a certain threshold, at which point it deals with all of them. We need to look into the interaction between the new thread and grpc calls and make sure it works fine.

The current plan is to export only the Last value as a Gauge, ignoring Min, Max, Sum, Count.

https://docs.github.com/en/github/administering-a-repository/enabling-required-reviews-for-pull-requests

• Copy over exporter code
• Basic tox set up for running tests
• Circle CI to run tox set up
• Move documentation?

open-telemetry/opentelemetry-python#1591 moves propagators to opentelemetry.propagate

When running opentelemetry cloud-exporter on GKE a default agent is set to opencensus when opentelemetry is the framework being used.

Potentially may be a default collector set up by GCP.

Opentelemetry Version: v0.9.0
Cloud Exporter Version: 0.10.dev0

I'm sending traces from an asgi app that is not hosted on google.
while in the trace ui I can see the opentelemetry label http.method and it's correct, GET POST etc, it seems the corresponding google span is not populated.

snippet-bot scan result

Life is too short to manually check unmatched region tags.
Here is the result:
Great job! No unmatching region tags found!

Report generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.

We should grab info about environment and what Google tools we are using, if any, from metrics.

it seems currently valueRecoder is not supported. Any plan for it?

This is implemented in the Go exporter already, see here and here. Note this is separate from the g.co/agent span label set here

The Cloud Trace Propagator uses the X-Cloud-Trace-Context header key, which has upper case letters. When using it with the gRPC instrumentation, it sets this as gRPC metadata which will throw a validation error and abort the request because of the upper case letters.

Please make the key all lower case, I tested it on my own services on Cloud Run using a Flask app (http from Google Cloud's Load Balancer which sets the x-cloud-trace-context header initially), and is passed on to my backend service using gRPC and it works with the metadata key set to the same, and appears that the AppServer component on Cloud Run in-between the two services also picks up the trace ID with the lowercase metadata header, and everything links up correctly as expected so I can now see my full end-to-end request on Google Trace correctly.

Run mypy in dev/CI. Would be nice to have a config that works cleanly with vscode.

Some things we'd like to test are:

• Tests for the various features that the Exporter supports. (events, links, status codes, etc.)
• Test that contains an invalid span and assert failure
• Export multiple spans

https://github.com/GoogleCloudPlatform/opentelemetry-operations-python/pull/101/files/688b4bf84ad632635dc91a86628c624f69c89d86#diff-6e7e9495c01b3a91f2e2d9cdb033d1f0c9e032ed3b7b7f7aefbf4f2731d02bb2R416

This can be removed after open-telemetry/opentelemetry-python#1573 is closed.

Internal Google spec for exporters was recently created. We currently are not following the correct package / folder structure. There are also some functions that are similar across the two exporters and can be generalized.

It looks like a dependent module, google-cloud-trace, released a new major version yesterday, with breaking package changes, breaking our deployment and bringing down our entire service: https://github.com/googleapis/python-trace/blob/master/CHANGELOG.md

Please can you update this module to use the latest release of google-cloud-trace, or lock the version that works with the current module (0.24.0). If you update the code to work with the new module, please lock the version that works so this doesn't happen again (or only support minor version updates).

I fixed it by adding this line to the end of my requirements.txt file to override the automatically installed version installed when adding this module:

google-cloud-trace==0.24.0

Delete https://pypi.org/project/opentelemetry-exporter-cloud-trace/ and https://pypi.org/project/opentelemetry-exporter-cloud-monitoring/ from PyPI in a while after we know it is unused. Charts of usage here:

• https://pepy.tech/project/opentelemetry-exporter-cloud-trace
• https://pepy.tech/project/opentelemetry-exporter-cloud-monitoring

As of 0.18b1, opentelemetry-instrumentation-grpc doesn't have the feature to exclude trace propagation to the specific endpoints. Because of this, gRPC servers always send health check request traces to Cloud Traces via the exporter as you see in the screenshot below:

It would be nice if the exporter support filter option to avoid sending unnecessary traces to the Cloud Trace. Otherwise, the upstream (opentelemetry-instrumentation-grpc) should have something similar to OTEL_PYTHON_FLASK_EXCLUDED_URLS in opentelemetry-instrumentation-flask

We currently only support detection of GCE resources. Cloud Monitoring supports these resources https://cloud.google.com/monitoring/custom-metrics/creating-metrics#custom-metric-resources so we should seek to match that support.

When we instrument requests + an exporter that uses http requests and grpc (ex. cloud trace exporter) we get an infinite loop. The flow is:

Span A (manually started by user)
When we export span A, the exporter makes a request, that request is wrapped and instrumented in Span B.
When we export span B, the exporter makes a request, that request is wrapped and instrumented in Span C.
When we export span C, the exporter makes a request, that request is wrapped and instrumented in Span D.
...
forever.

How this issue is usually resolved is by adding the flag "suppress_instrumentation" to contextvars before exporting, so that requests are NOT wrapped and instrumented anymore. Cloud Trace exporter uses grpc, which spins up a new thread, where contextvars are ignored, that makes a request which, since contextvars got ignored, continues to infinite loop.

The current solution is simply to blacklist and always suppress the url that grpc makes the request to. This is obviously not the best way to deal with the issue for a variety of reasons (e.g. what is the url changes in the future? What is grpc changes and makes more requests? What if the user wants to legitimately instrument a request to this url?).

I've talked with the grpc team and they have said there is currently no way of passing contextvars to grpc threads, but it is something they are working on. Another possible solution is changing the oauth threadpool to someone have this flag in its contextvars.

Several places we are using dictionaries and then either letting the library convert or doing SomeProto(**d). It would be better to use protos directly

Waiting for #64

Adding more examples of what we can export, stuff like links, attributes and events. Also a FAQ / troubleshooting section with common problems setting up.