We often get asked how to copy http request/response headers into OpenTelemetry span attributes. The java instrumentation agent does not do this by default, primarily because it risks leaking sensitive data. Fortunately, there is a way to instruct the java agent to collect http headers and attach them to your spans.

This project is a quick demonstrative example of capturing http headers as span attributes.

The main application is HttpHeaderAttributesMain. This application spins up a small HTTP server on port 8182, and a client that sends a request to the server once a second.

This is an unusual ab/use of HTTP and is only intended to demonstrate how to tell the java instrumentation to copy headers into attributes.

The client sends a POST request with a demeanor HTTP header and the name of a person in the request body.

POST /greeting HTTP/1.1
Content-Length: 8
Host: localhost:8182
User-Agent: Java-http-client/11.0.9.1
Content-Type: text/plain
demeanor: TIRED

Benjamin

The server responds with an originator header whose value is the uppercase version of the person's name. The body contains the person's name and an emoji corresponding to their demeanor.

HTTP/1.1 200 OK
Date: Thu, 15 Dec 2022 21:39:27 GMT
Content-Type: text/plain
originator: BENJAMIN
Transfer-Encoding: chunked
Server: Jetty(9.4.48.v20220622)

Benjamin => 🥱

To build and run the example, simply run the gradle build target run. You must have java 11+ installed and an otel collector running on localhost.

./gradlew run

Configuration is provided to the java agent via system properties passed on the commandline. No manual code changes required!

The upstream community documentation for this feature is here.

If you look in the build.gradle.kts file, you will notice several JVM commandline args. These are used to wire up the java agent, and provide our custom configuration:

-javaagent:splunk-otel-javaagent-1.18.0.jar
-Dotel.javaagent.debug=true
-Dotel.resource.attributes=deployment.environment=http-testenv
-Dotel.service.name=HttpHeaderAttributes
-Dotel.instrumentation.http.capture-headers.client.request=demeanor
-Dotel.instrumentation.http.capture-headers.client.response=originator
-Dotel.instrumentation.http.capture-headers.server.request=demeanor,user-agent
-Dotel.instrumentation.http.capture-headers.server.response=originator

The last 4 are the interesting ones. These tell the agent which headers to capture. Specifically, we are asking the java agent to capture these 4 headers:

• client instrumentation - outgoing request - demeanor
• client instrumentation - incoming response - originator
• server instrumentation - incoming request demeanor and user-agent
• server instrumentation - outgoing response - originator

If your application is purely a client or purely a server, you would probably only need half of this, but we've provided both directions on both sides for demonstration.

In the APM trace view, we have generated a simple trace with one client span and one server span:

If we expand the topmost client span, we can view the relevant attributes:

We see that the span contains two new attributes that would not have been there otherwise: http.request.header.demeanor and http.response.header.originator.

If we now expand the server span, we can see the new attributes there as well:

The three additional attributes are http.request.header.demeanor, http.request.header.user_agent, and http.response.header.originator.

HTTP headers that are turned into attributes are namespaced/prefixed with one of:

• http.request.header.
• http.response.header.

In addition to the prefix being added header names will have hyphens converted to underscores (see User-Agent becoming user_agent in the above example).

Furthermore, because HTTP headers may be repeated, the values are treated as a list. The resulting attribute value is surrounded with square brackets and the values are concatenated with commas.