OpenTelemetry also allows to report metrics:

• https://github.com/open-telemetry/opentelemetry-java/blob/master/QUICKSTART.md#metrics
• https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/metrics

zio-telemetry doesn't support these, but could potentially

Currently the opentracing module will report 2 traces for a single (1) request:

Here, the 'ROOT' span belonging only to 'zio-backend' was merely created when instantiating the module, and is probably of no interest to the user. The only correct one is the second reported trace (having 2 spans) it seems?

When implementing #119, the same behaviour was encountered at first. It was solved by instantiating the module with a noop, default span.

There exists also a noop span for OpenTracing, and the same approach as OpenTelemetry will probably resolve the issue. Note that it will also solve the burden of choice the user has right now when choosing a rootspan when instantiating the module. (Even though a default name is provided, naming of the root span is probably not something that should be of relevance when instantiating the layer)

based on the instruction, when I'm running (the same happening for the BackendServer in the case of opentelemetry):
sbt "opentelemetryExample/runMain zio.telemetry.opentelemetry.example.ProxyServer"

I'm getting following output:

[info] welcome to sbt 1.5.5 (AdoptOpenJDK Java 11.0.11)
[info] loading settings for project zio-telemetry-build-build-build from metals.sbt ...
[info] loading project definition from /Users/htaheri/PlayArea/zio-telemetry/project/project/project
[info] loading settings for project zio-telemetry-build-build from metals.sbt ...
[info] loading project definition from /Users/htaheri/PlayArea/zio-telemetry/project/project
[success] Generated .bloop/zio-telemetry-build-build.json
[success] Total time: 1 s, completed 23 Jul 2021, 16:03:09
[info] loading settings for project zio-telemetry-build from metals.sbt,plugins.sbt ...
[info] loading project definition from /Users/htaheri/PlayArea/zio-telemetry/project
[success] Generated .bloop/zio-telemetry-build.json
[success] Total time: 2 s, completed 23 Jul 2021, 16:03:11
[info] loading settings for project root from build.sbt ...
[info] set current project to root (in build file:/Users/htaheri/PlayArea/zio-telemetry/)
[info] running zio.telemetry.opentelemetry.example.ProxyServer
[error] Nonzero exit code: 1
[error] (opentelemetryExample / Compile / runMain) Nonzero exit code: 1
[error] Total time: 4 s, completed 23 Jul 2021, 16:03:19

I had to check out to the branch to v0.7.2 to make it work, so seems some updates from that time till today causing some problem.

Due to misconfigured cross-build, release 0.0.1 hasn't been published for Scala 2.13.

Taken from #444:

Unless the operation contains an error, the StatusCode should remain "unset" and attempts to set value Unset will be ignored. A more sensible default would be to always set an Error StatusCode for any defect or failure and allow the caller to change if needed.

OpenCensus documentation is missing examples how to create Tracer and properly wire things to export into eg. Jaeger or Datadog.

There are no tests for this module now, it would be good to have some to be sure that nothing breaks when making code changes.

The idea behind defining a service there can exist multiple implementations like opentracing, opentelemetry, honeycomb etcetera. Now we've defined per tracing solution a seperate module, but I think it's a great idea to unify this like github.com/tpolecat/natchez/ does

WDYT?

It is impossible to implement a proper span for a ZStream returning method.

It requires a version of Tracing::span for Zstream instead of ZIO.

For more details, see the spec: https://opentelemetry.io/docs/instrumentation/java/manual/#record-exceptions-in-spans

Currently all APIs accept a toErrorStatus: PartialFunction[E, StatusCode] and then set the Span's Status like this:

The default is Map.empty, meaning that for any defect or failure, we set the Span's StatusCode to UNSET, together with a Description. IMO there a a few problems:

• It is a bit confusing: According to the specification, Description is optional (for example: Don't set the span status description if the reason can be inferred from http.status_code) and MUST only be used with the Error StatusCode value.
• Unless the operation contains an error, the StatusCode should remain "unset" and attempts to set value Unset will be ignored. A more sensible default would be to always set an Error StatusCode for any defect or failure and allow the caller to change if needed.
• Sometimes, for a ZIO[R, E, A], we also want to set StatusCode based on A. For example, the OpenTelemetry's semantic conventions for HTTP recommends to set StatusCode Error for HTTP status codes in the 4xx and 5xx ranges. So in this case, we need to look into a "successful" response to know which StatusCode to set.

For example, if we change the default, we can use something like toErrorStatus: Either[E, A] => UIO[Any] with a default that set StatusCode to Error, with a Description, for any defect or failure.

WDYT?

My company uses mostly a Java stack and uses opentelemetry auto instrumentation for the majority of its services however I work on a ZIO service. We would also like to leverage auto instrumentation. We are able to get some connected spans through auto instrumentation but we have noticed when the root span is created with zio-telemetry and any downstream span is created with auto instrumentation, the spans will not be connected.

I see #207 solves this issue when the auto instrumented spans are from non-zio code but when the auto instrumented spans are from ZIO code this solution no longer works. As an example, lets say I have a non-zio database library that is wrapped in a ZIO library, so all my calls to the database are ZIOs. If I create a new span and call into this ZIO database wrapper library, the spans in jaeger will be separate.

What is interesting however is that when my service receives a grpc call that downstream calls a ZIO wrapped database library, all of the auto instrumented spans are connected. However in this case zio-telemetry is never used.

My guess on what is going on is that the active span set in the ZIO environment does not translate to the underlying Context for auto instrumentation.

For more details, see the spec: https://opentelemetry.io/docs/instrumentation/java/manual/#create-spans-with-links

See specs: https://opentelemetry.io/docs/reference/specification/baggage/api/

All methods in the companion object should be covered by tests. span, root and injecting and extracting are covered already. Test for tagging and logging as well as baggage items should be added as well.

• tag
• log
• baggage

Currently we provide an example of Jaeger tracer usage in opentelemetry-example. There is a demand to add the example of DataDog tracer usage too. It would be good to make both opentelemetry-example and opentelemetry-instrumentation-example work with DataDog.

Related: #686

There is a feature request in the Discord channel (https://discord.com/channels/629491597070827530/639825316021272602/1113090265041862697) to add an ability to make a "scoped span".

So, we need to add a new method similar to the existing Tracing.span. It must not take a zio effect as an argument but instead return scoped ZIO value, like:

  def span(
    spanName: String,
    spanKind: SpanKind = SpanKind.INTERNAL,
    attributes: Attributes = Attributes.empty(),
    links: Seq[SpanContext] = Seq.empty
  )(implicit trace: Trace): ZIO[Scope, Nothing, Unit]

There is no need to add an aspect method since it doesn't wrap up zio effect.

Usage examples:

def makeStream(): ZStream[Any, Nothing, Int] =
  ZStream.scoped(tracing.spanScoped("stream span")) *> ZStream.repeat(1).take(10)

val workflow: ZIO[Scope, Nothing, String] = for {
  _ <- tracing.spanScoped("parent span name")
  _ <- tracing.addEvent("parent event")
  _ <- tracing.spanScoped("child span name") // It is worth checking if this "parent/child" semantics makes sense for spanScoped
  _ <- tracing.addEvent("child event")
  a <- ???
} yield a

val workflow1 = ZIO.scoped(ZIO.debug("workflow 1") *> workflow)
val workflow2 = ZIO.scoped(ZIO.debug("workflow 2") *> workflow)

ZIO 2.0 is at Milestone 4, with an RC expected in the next few weeks.
https://github.com/zio/zio/releases/tag/v2.0.0-M4

The API is nearly stable at this point, so any early migration work against this version should pay off towards the official 2.0 release.

The progress is being tracked here:
zio/zio#5470

The Stream Encoding work in progress is the only area where the API might still change before the RC.

We are actively working on a ScalaFix rule that will cover the bulk of the simple API changes:
https://github.com/zio/zio/blob/series/2.x/scalafix/rules/src/main/scala/fix/Zio2Upgrade.scala
We highly recommend starting with that, and then working through any remaining compilation errors :)

To assist with the rest of the migration, we have created this guide:
https://zio.dev/howto/migrate/zio-2.x-migration-guide/

If you would like assistance with the migration from myself or other ZIO contributors, please let us know!

Add these two plugins with corresponding check commands integrated into CI workflow.

With zio-config, zio-json and zio-http out there, it makes sense to rewrite the OpenTracing and OpenTelemetry examples using them.

For the sake of simplicity, the implementer should preserve the existing scenario, i.e., server and client, tracing the whole request-response cycle. Once the examples are ported, we can think about making additional scenarios.

It would be nice if the functions exposed currently with an implicit class were also exposed in the service, with the ZIO as parameter. I have some use case where I don't want to have OpenTracing in the environment and I have an OpenTracing.Service object, but I'm forced to use provideSomeLayer to eliminate the dependency since the logic isn't exposed directly on the service.

As of v0.1.0, OpenTelemetry abstractions are temporary dropped to simplify migration to new ZIO version. All future versions should support both OpenTracing and OpenTelemetry, preferrably as separate modules.

The implementer of this ticket should design an OpenTelemetry module, around the existing JVM client.

Apologies in advance if this doesn't make sense, I'm still making sense of the opentelemetry world...

I've been playing with the opentelemetry auto instrumentation and zio-telemetry and have both of them submitting spans to jaeger very nicely. In case where requests come in through akka http and play, the auto-instrumentation creates the root span and my zio runtime picks it up successfully. However, my zio code then has to call through into non-zio code (slick, in this case) to access the database. The auto-instrumentation instruments jdbc, which works, but the spans for those sql calls are not connected to the zio telemetry spans.

I believe the issue here is that Tracing.live picks up the default current context, but zio doesn't push a scope onto the thread local context prior to executing the "legacy" code. I'm still trying to figure out how to make this connection...

From what I can tell so far, there a few general approaches we could take to this:

1. Expose the span and eventually document how to propagate the context by hand if desired. This is the simplest for zio-telemetry in that it just means making currentSpan public.

2. Set the thread-local context prior to executing spanned effects. I've been experimenting with this in combination with (1) but don't quite have it working yet. Notionally, it looks something like this:

for {
  currentSpan <- Tracing.currentSpan
  result <- ZIO.effect { 
     val scope = Tracer.withSpan(currentSpan)
     try { 
     .... do stuff
     } finally {
        scope.finish()
     }
  }
} yield result

Assuming this can be made to work, it seems like it ought to be possible to wire into Tracing.span() and avoid the need for exposing currentSpan as public.

3. Integrate directly with the various future/callback context propagation systems in play (otel akka, otel java-concurrent, opentracing scala-concurrent). This seems like it might be the cleaner way to avoid any propagation issues with respect to fibers vs thread pools and such but doesn't necessarily cover the case of directly executed code. Unless there's an issue with using the thread-local context as the conduit, that seems like the obvious solution in that the various concurrency handlers below are designed to pick up their context from the calling thread.

4. Something else entirely?

I believe we need to add an Unsafe suffix to:

• Tracing.getCurrentContext
• Tracing.getCurrentSpan
• Tracing.getCurrentSpanContext

They are not supposed to be used by most users in most scenarios.

I am trying to use zio-opencensus using the following dependency

"dev.zio" %% "zio-opencensus" % "0.7.0"

but get the following error when running sbt update

[error] sbt.librarymanagement.ResolveException: Error downloading dev.zio:zio-opencensus_2.13:0.7.0
[error]   Not found
[error]   Not found
[error]   not found: /home/petter/.ivy2/local/dev.zio/zio-opencensus_2.13/0.7.0/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/dev/zio/zio-opencensus_2.13/0.7.0/zio-opencensus_2.13-0.7.0.pom
...

It looks like zio-opencensus is not published at all in the sbt ci-release command, but I am not really sure where and how that is configured. Anyway, I would appreciate if zio-opencensus could get published.

The example used to demonstrate library capabilities needs to be simplified and better explained.

https://repo1.maven.org/maven2/dev/zio/zio-opentelemetry_3/

I only see 2.0.0-RC3

By the way it's probably time to cut a 2.0.0 final (#548)?

There have been a few feature PRs merged since the last release.

Would it be possible to have a new minor version? I know y'all are working hard on upgrading to zio 2, but if it's not too much trouble we'd appreciate it :)

It would be nice to have some functions to manipulate the OpenTelemetry Baggage.

There are have been some breaking changes in the upstream api for opentelemetry tracing. Most notably, BREAKING CHANGE: The only way to specify span parenting is via a parent Context from https://github.com/open-telemetry/opentelemetry-java/releases/tag/v0.9.1

Has anyone given any thought as to how to adjust zio-telemetry to match yet?

I think there left a few things to improve to fully migrate to module pattern 2.0:

• make in the next version all accessors deprecated in all modules
• migrate opencensus and opentracing modules
• find another way to encode Syntax without requiring to provide Tracing in R.
• update docs

Taken from #444:

Sometimes, for a ZIO[R, E, A], we also want to set StatusCode based on A. For example, the OpenTelemetry's semantic conventions for HTTP recommends to set StatusCode Error for HTTP status codes in the 4xx and 5xx ranges. So in this case, we need to look into a "successful" response to know which StatusCode to set.

The question was asked in the Discord channel (https://discord.com/channels/629491597070827530/639825316021272602/1082410092852416642):

Hey all! I wanted to ask about the status of ZIO 2.x w/ zio-telemetry & feeding tracing info into Datadog. Most of the previous conversation in this channel revolves around ZIO 1.x. It's also my understanding that automatic instrumentation does not yet work with DataDog/dd-trace-java for ZIO 2.x (though there is a PR open here: DataDog/dd-trace-java#4848)

Is manual instrumentation possible here, or is the reliance of DD's tracer on ThreadLocal for scope state breaking? If it is possible, does anyone have a preexisting example? I've only been able to locate the Jaeger example so far.

Taken from #444:

It is a bit confusing: According to the specification, Description is optional (for example: Don't set the span status description if the reason can be inferred from http.status_code) and MUST only be used with the Error StatusCode value.

We are starting the context from a graphQL API ( or even earlier in case of distributed tracing, from the client). I didn't know how to pass the context to the zio managed one via the FiberRef other than writing the below class in our project so I can access the package protected setters.

Is there a way to already to this ? else I think something should be available to pass an existing context inside the zio managed one

package zio.telemetry.opentelemetry

import io.opentelemetry.context.Context
import zio.ZIO

object ZioTelemetryAccessor {

  def setCurrent(ctx: Context): ZIO[Tracing, Nothing, Unit] = for {
    fib <- ZIO.access[Tracing](_.get.currentContext)
    _   <- fib.set(ctx)
  } yield ()

}

The goal of ZIO Telemetry should be to seamlessly add OpenTelemetry support via ZIO's own built-in logging and metrics, adding integration points as necessary and useful.

TODO

• How to map ZIO logging + annotations + other context to OpenTelemetry traces
• How to map ZIO logging + annotations + other context to OpenTelemetry logs
• How to map ZIO metrics to OpenTelemetry metrics
• How to map ZIO context + annotations to OpenTelemetry baggage

Document usage of example application in Example usage section in README.md.

We need to add Scala 3 cross-compilation

When doing this

other
        .send(request.headers(headers.toMap))
        .span(show"${request.method.method} ${request.uri.path.mkString("/")}")
        .tag("span.kind", "client")
        .tag("http.method", request.method.method)
        .tag("http.url", request.uri.toString())
        .tag("type", "ext")
        .tag("subtype", "http")
        .flatMap(res => OpenTracing.tag("http.status_code", res.code.code) *> ZIO.succeed(res))

The tag are not set on the span of send but on it's parent.

Scala 3.1.1 with -Yexplicit-nulls
zio-opentelemetry 2.0.0-RC1

when upgrading from zio 2.0.0-RC2 to 2.0.0-RC3 we have the compiler throwing

java.lang.AssertionError: assertion failed while compiling /foobar/tests/src/main/scala/tracing.scala
[error] ## Exception when compiling 1 sources to /foobar/tests/target/scala-3.1.1/classes
[error] java.lang.AssertionError: assertion failed
[error] scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:11)
[error] dotty.tools.dotc.core.Types$MethodType.<init>(Types.scala:3697)
[error] dotty.tools.dotc.core.Types$CachedMethodType.<init>(Types.scala:3717)
[error] dotty.tools.dotc.core.Types$MethodTypeCompanion.apply(Types.scala:3788)
[error] dotty.tools.dotc.core.Types$MethodTypeCompanion.apply(Types.scala:3787)
[error] dotty.tools.dotc.core.Types$LambdaType.newLikeThis(Types.scala:3470)
[error] dotty.tools.dotc.core.Types$LambdaType.newLikeThis$(Types.scala:3403)
[error] dotty.tools.dotc.core.Types$MethodOrPoly.newLikeThis(Types.scala:3483)
...

Single file with single line

src/main/scala/tracing.scala

val y = zio.telemetry.opentelemetry.Tracing.live

build.sbt

scalaVersion := "3.1.1"

scalacOptions ++= Seq(
  "-unchecked",
  "-deprecation",
  "-feature",
  "-Yexplicit-nulls",
  "-source",
  "future"
)

libraryDependencies ++= Seq(
  "dev.zio" %% "zio"               % "2.0.0-RC3",
  "dev.zio" %% "zio-opentelemetry" % "2.0.0-RC1"
)

Document the current functionality in README.md

We should illustrate use of zio-telemetry using an example app. It should consist of multiple HTTP services depending on each other. The first service should inject the trace context to all requests it makes. The other services extract and propagate that information. While a first version could just log tracing information, it would be nice to spin up a container running an actual opentracing collector, such as Jaeger to collect data.

• Build at least two services
• Inject / Extract tracing headers
• Log traces
• Collect traces in Jaeger

• opentelemetry-example
• opentelemetry-instrumentation-example
• opentracing-example

In this version, no accessors will be required, just services that provide methods to perform these functions.

Is a Scala 3 release planned?
What are the roadblocks?

Initially, work was done in the PR: #580. Now it is time to make a backport for zio1.

Reported in Discord channel: https://discord.com/channels/629491597070827530/639825316021272602/1120329881318862848

I am currently trying to integrate zio-http with zio-opentelemetry. While we use relatively old versions of both, I am unable to e.g. create a span for each request I receive that ends only when the response body stream completes. With several hacks I do manage to create a child span for the response stream, but the parent span ends where it begins.
I come here to ask if there is an already premade solution to this, even if it necessitates an upgrade of the libraries.

The bug was reported in the Discord channel:
https://discord.com/channels/629491597070827530/639825316021272602/1116089112529748000

I am using auto-instrumentation for my grpc service that also is a Kafka consumer. When I receieve a request or Kafka message I want to manually set custom attributes. I either do this by calling a java library to set custom attributes:
ZIO.serviceWithZIO[Tracing](_.scopedEffect(library.setAttributes(...))
Or I use the zio library to set custom attributes:
ZIO.serviceWithZIO[Tracing](_.setAttribute("foo", "bar")
I am seeing the autoinstrumented traces for both grpc and Kafka messages but I am not seeing any custom attributes for the traces when I look in jaeger.

Here is my Tracing layer:

    (ZLayer.fromZIO(
      ZIO.attempt(GlobalOpenTelemetry.getTracer("io.opentelemetry.api"))) ++
      ContextStorage.openTelemetryContext) >>> OtelTracing.live
  }

And I am using v3.0.0-RC14. Previosuly when I was on v2.0.3 the custom attribiutes worked for grpc but not for Kafka.
Any reason these attributes arent showing up when I view them in jaeger?

It is quite annoying that we need to pay the price of typing useless type annotations each time we call zio @@ span[Throwable, Unit]("...") or similar aspect methods when initially the goal of introducing them was to have better API ergonomics.
We ended up in a situation where some of the methods require typing 3 type parameters after #702 gets merged.