Instead of hardcoding the fields offsets (e.g. HTTP request path), get them at runtime from DWARF information.

Technically it is possible. E.g if you run:

objdump -Wi executable

Then you will find something like:

 <1><9bf94>: Abbrev Number: 39 (DW_TAG_structure_type)
    <9bf95>   DW_AT_name        : net/http.Request
    <9bfa6>   DW_AT_byte_size   : 248
    <9bfa8>   Unknown AT value: 2900: 25
    <9bfa9>   Unknown AT value: 2904: 0xac940
 <2><9bfb1>: Abbrev Number: 24 (DW_TAG_member)
    <9bfb2>   DW_AT_name        : Method
    <9bfb9>   DW_AT_data_member_location: 0
    <9bfba>   DW_AT_type        : <0x65741>
    <9bfbe>   Unknown AT value: 2903: 0
 <2><9bfbf>: Abbrev Number: 24 (DW_TAG_member)
    <9bfc0>   DW_AT_name        : URL
    <9bfc4>   DW_AT_data_member_location: 16
    <9bfc5>   DW_AT_type        : <0x8a4bb>
    <9bfc9>   Unknown AT value: 2903: 0
 <2><9bfca>: Abbrev Number: 24 (DW_TAG_member)
    <9bfcb>   DW_AT_name        : Proto
    <9bfd1>   DW_AT_data_member_location: 24
    <9bfd2>   DW_AT_type        : <0x65741>
    <9bfd6>   Unknown AT value: 2903: 0
 <2><9bfd7>: Abbrev Number: 24 (DW_TAG_member)
    <9bfd8>   DW_AT_name        : ProtoMajor
    <9bfe3>   DW_AT_data_member_location: 40
    <9bfe4>   DW_AT_type        : <0x65731>
    <9bfe8>   Unknown AT value: 2903: 0
 <2><9bfe9>: Abbrev Number: 24 (DW_TAG_member)
    <9bfea>   DW_AT_name        : ProtoMinor
    <9bff5>   DW_AT_data_member_location: 48
    <9bff6>   DW_AT_type        : <0x65731>
    <9bffa>   Unknown AT value: 2903: 0
 <2><9bffb>: Abbrev Number: 24 (DW_TAG_member)
    <9bffc>   DW_AT_name        : Header
    <9c003>   DW_AT_data_member_location: 56
    <9c004>   DW_AT_type        : <0x98c7d>
    <9c008>   Unknown AT value: 2903: 0
 <2><9c009>: Abbrev Number: 24 (DW_TAG_member)
    <9c00a>   DW_AT_name        : Body
    <9c00f>   DW_AT_data_member_location: 64
    <9c010>   DW_AT_type        : <0x95a2d>
    <9c014>   Unknown AT value: 2903: 0
 <2><9c015>: Abbrev Number: 24 (DW_TAG_member)
    <9c016>   DW_AT_name        : GetBody
    <9c01e>   DW_AT_data_member_location: 80
    <9c01f>   DW_AT_type        : <0x9c158>
    <9c023>   Unknown AT value: 2903: 0
 <2><9c024>: Abbrev Number: 24 (DW_TAG_member)
    <9c025>   DW_AT_name        : ContentLength
    <9c033>   DW_AT_data_member_location: 88
    <9c034>   DW_AT_type        : <0x6606b>
    <9c038>   Unknown AT value: 2903: 0
 <2><9c039>: Abbrev Number: 24 (DW_TAG_member)
    <9c03a>   DW_AT_name        : TransferEncoding
    <9c04b>   DW_AT_data_member_location: 96
    <9c04c>   DW_AT_type        : <0x69294>
    <9c050>   Unknown AT value: 2903: 0
 <2><9c051>: Abbrev Number: 24 (DW_TAG_member)
    <9c052>   DW_AT_name        : Close
    <9c058>   DW_AT_data_member_location: 120
    <9c059>   DW_AT_type        : <0x65091>
    <9c05d>   Unknown AT value: 2903: 0
 <2><9c05e>: Abbrev Number: 24 (DW_TAG_member)
    <9c05f>   DW_AT_name        : Host
    <9c064>   DW_AT_data_member_location: 128
    <9c066>   DW_AT_type        : <0x65741>
    <9c06a>   Unknown AT value: 2903: 0
 <2><9c06b>: Abbrev Number: 24 (DW_TAG_member)
    <9c06c>   DW_AT_name        : Form
    <9c071>   DW_AT_data_member_location: 144
    <9c073>   DW_AT_type        : <0x8a5a5>
    <9c077>   Unknown AT value: 2903: 0
 <2><9c078>: Abbrev Number: 24 (DW_TAG_member)
    <9c079>   DW_AT_name        : PostForm
    <9c082>   DW_AT_data_member_location: 152
    <9c084>   DW_AT_type        : <0x8a5a5>
    <9c088>   Unknown AT value: 2903: 0
 <2><9c089>: Abbrev Number: 24 (DW_TAG_member)
    <9c08a>   DW_AT_name        : MultipartForm
    <9c098>   DW_AT_data_member_location: 160
    <9c09a>   DW_AT_type        : <0x94646>
    <9c09e>   Unknown AT value: 2903: 0
 <2><9c09f>: Abbrev Number: 24 (DW_TAG_member)
    <9c0a0>   DW_AT_name        : Trailer
    <9c0a8>   DW_AT_data_member_location: 168
    <9c0aa>   DW_AT_type        : <0x98c7d>
    <9c0ae>   Unknown AT value: 2903: 0
 <2><9c0af>: Abbrev Number: 24 (DW_TAG_member)
    <9c0b0>   DW_AT_name        : RemoteAddr
    <9c0bb>   DW_AT_data_member_location: 176
    <9c0bd>   DW_AT_type        : <0x65741>
    <9c0c1>   Unknown AT value: 2903: 0
 <2><9c0c2>: Abbrev Number: 24 (DW_TAG_member)
    <9c0c3>   DW_AT_name        : RequestURI
    <9c0ce>   DW_AT_data_member_location: 192
    <9c0d0>   DW_AT_type        : <0x65741>
    <9c0d4>   Unknown AT value: 2903: 0
 <2><9c0d5>: Abbrev Number: 24 (DW_TAG_member)
    <9c0d6>   DW_AT_name        : TLS
    <9c0da>   DW_AT_data_member_location: 208
    <9c0dc>   DW_AT_type        : <0x8e6f5>
    <9c0e0>   Unknown AT value: 2903: 0
 <2><9c0e1>: Abbrev Number: 24 (DW_TAG_member)
    <9c0e2>   DW_AT_name        : Cancel
    <9c0e9>   DW_AT_data_member_location: 216
    <9c0eb>   DW_AT_type        : <0x7d89a>
    <9c0ef>   Unknown AT value: 2903: 0
 <2><9c0f0>: Abbrev Number: 24 (DW_TAG_member)
    <9c0f1>   DW_AT_name        : Response
    <9c0fa>   DW_AT_data_member_location: 224
    <9c0fc>   DW_AT_type        : <0x9c198>
    <9c100>   Unknown AT value: 2903: 0
 <2><9c101>: Abbrev Number: 24 (DW_TAG_member)
    <9c102>   DW_AT_name        : ctx
    <9c106>   DW_AT_data_member_location: 232
    <9c108>   DW_AT_type        : <0x7d950>
    <9c10c>   Unknown AT value: 2903: 0

Right now we don't fully capture the time it takes from request start for Go. I have some usecases where the Go instrumentation reports similar timings to manual instrumentation, whereas the kprobes socket filter does capture the full time. I think with enabling separate type of kprobes for Golang instrumentation we'll be able to get similar timings.

Making our HTTP client for integration tests run without keepalive, which is default for Go, it causes failure in few of the Traces tests.

To make this happen I ran the test suite with the following addition to the Transport configuration for the HTTP client:

DisableKeepAlives: true,

Current C code has some hacks that only work for x86. E.g. assuming little endian or using specific CPU registers. Move all this stuff to another .h file and prepare it for supporting other architectures e.g. ARM64

Provide a Prometheus endpoint to scape internal metrics, such as processed messages.

Currently, a single agent instance only instruments a single executable.

Instead of providing a single executable name, provide a list of regexpes that could match multiple processes in the system, and instrument all of them from the same instrumenter instance.

During the aggregation of HTTP call spans, the eBPF code stores ongoing spans by using the request context pointer as key (private field ctx in type http.Request).

While this is performant (only have to store a pointer as map key), this is an implementation detail that might be broken if the http.Request implementation changes.

We could consider using something more stable, such as the public field http.Request.RemoteAddr. However this might involve some extra processing because this field is a variable-length string.

An extra attribute should be added, such as side={client,server}

Java Integration tests are failing in ARM64 hosts when trying to search by executable name, as, instead of greeting, the process ins named something like: {greeting} /usr/bin/qemu-x86_64 /greeting /greeting.

Current testing image is personal, so we should create a grafana/ebpf-autoinstrumenter-tests Hub entry with multi-arch images.

For containerized demos, we are currently using my personal Docker account (mariomac/ebpf-autoinstrument:latest), which is updated and pushed manually.

We need to create a GitHub action that pushes a new image to the Grafana Docker organization on each merge into main branch.

Basically, investigate and document which capabilities are needed to run the executable without requiring full privileged mode. Provide some examples for e.g. Docker and Kubernetes deployments.

The way the instrumenter currently works, it searches either for net/http.HandlerFunc.ServeHTTP or github.com/gin-gonic/gin.(*Engine).ServeHTTP, and then adds an instrumentation point to the latest found function, but not to both.

This could lead to not being able to instrument customers that are progresssively migrating away from Gin to another framework.

Timeouts aren't properly reported. Probably because the ServeHTTP handler does not end by the usual mean and then the trace is not completed.

From other services in the App O11y plugin, I see that they report many resource attributes that we don't, and they could be useful e.g. to differentiate between instances:

• host.name
• os.description
• os.type
• process.command_args
• process.executable.name
• process.executable.path
• process.owner
• process.pid
• process.runtime.description (e.g. go version go1.19.2 linux/amd64)
• process.runtime.name
• service.instance.id
• service.namespace
• telemetry.sdk.language
• telemetry.sdk.name
• telemetry.sdk.version

If we detect that the process runs in a container:

• container.id

If we detect that the process runs in K8s:

• k8s.node.name
• k8s.pod.name
• k8s.namespace.name

For Kubernetes, the easiest way would be to just allow setting these values via env vars, then use the valueFrom clause in the deployment descriptors.

For service calls shorter than 1ms, the traces show no duration data.

Build sample gRPC client and server programs in go to be used for testing the gRPC testing.

Relates to #20

To minimize traffic, allow the customers to report only routes following a given pattern.

This task can reuse the pattern definitions from issue #6

Including a quickstart tutorial.

Placeholder task. To be completed with subtasks when we design it.

Currently, metrics/traces reports the http.target attribute as something like:

/users/123/article/456

We should let users defining regexpes so they can provide http.route according to the semantic conversions document

E.g.

target: /users/\d+/article/\d+
route: /users/{:userId}/article/{:articleId}

In e.g. a docker-compose file, deploy:

• a simple http service (e.g. ping server)
• http-autoinstrumenter
• grafana-agent
• open source tempo/mimir containers
  (optionally replace grafana agent+tempo+mimir by an OTEL traces+metrics collector)

On each testing scenario, play with the test service and verify that the data queried from tempo/mimir contains the generated traces/metrics.

We might need to be able to debug the eBPF C code in case we hit an issue when the instrumenter is deployed.

We should allow the LogLevel configuration option to affect the eBPF C code as well.

Placeholder task. To be completed with subtasks when we design it.

Placeholder task. To be completed with subtasks when we design it.

Placeholder task. To be completed with subtasks when we design it.

The make update-offsets task should be run each time Go version or any of the libraries we eventually instrument are updated.

We should create a periodic GitHub action that runs make update-offsets and submits a pull request each time there are new changes in the destination file.

Placeholder task. To be completed with subtasks when we design it.

In projects previously using a similar architecture, 200K events/second consumed 0.5 CPUs. The impact was caused mostly because of the userspace having to wakeup on each message in the ringbuffer. In the case of the linked project, moving from a ringbuffer to a hashmap allowed aggregating millions of events/second with 0.1% CPU.

In the case of HTTP/GRPC servers, it seems unlikely that a single host will need to process so many events/second. So our current implementation should be fine. However, we could create performance tests to see how many resources we are consuming on a typical high-load scenarios.

If we decide to optimize the performance, I would recommend to address the userspace wakeups on each HTTP request message.

We can configure the ringbuffer to accumulate messages and send them to the user space in batches of X (user configurable). If, after a user-configured timeout, the batch size didn't reach X, we will anyway send them. This will decrease by orders of magnitude the number of userspace wakeups.

For standard Go library, we are overriding the following function:

net/http.(*ServeMux).ServeHTTP

However, other programs might use a function with the same signature:

github.com/mariomac/goblog/src/assets.(*CachedHandler).ServeHTTP

https://github.com/mariomac/goblog/blob/main/src/assets/handler.go#L101

Just overriding the name should work.

From time to time, integration tests fail with the following message:

?   	github.com/grafana/ebpf-autoinstrument/test/integration/components/testserver/std	[no test files]
2023/04/19 12:37:40 INFO Getting feature for point lat=409146138 long=-746188906
2023/04/19 12:[37](https://github.com/grafana/ebpf-autoinstrument/actions/runs/4743350995/jobs/8422787779#step:4:38):40 name:"Berkshire Valley Management Area Trail, Jefferson, NJ, USA" location:{latitude:409146138 longitude:-746188906}
2023/04/19 12:37:40 INFO Getting feature for point lat=409146138 long=-746188906
2023/04/19 12:37:40 name:"Berkshire Valley Management Area Trail, Jefferson, NJ, USA" location:{latitude:409146138 longitude:-746188906}
2023/04/19 12:37:40 INFO Getting feature for point lat=4091461[38](https://github.com/grafana/ebpf-autoinstrument/actions/runs/4743350995/jobs/8422787779#step:4:39) long=-746188906
2023/04/19 12:37:40 name:"Berkshire Valley Management Area Trail, Jefferson, NJ, USA" location:{latitude:409146138 longitude:-746188906}
2023/04/19 12:38:[40](https://github.com/grafana/ebpf-autoinstrument/actions/runs/4743350995/jobs/8422787779#step:4:41) INFO Getting feature for point lat=409146138 long=-746188906
2023/04/19 12:38:40 name:"Berkshire Valley Management Area Trail, Jefferson, NJ, USA" location:{latitude:409146138 longitude:-746188906}
2023/04/19 12:38:40 INFO Getting feature for point lat=409146138 long=-746188906
2023/04/19 12:38:40 name:"Berkshire Valley Management Area Trail, Jefferson, NJ, USA" location:{latitude:409146138 longitude:-746188906}
2023/04/19 12:38:40 INFO Getting feature for point lat=409146138 long=-746188906
2023/04/19 12:38:40 name:"Berkshire Valley Management Area Trail, Jefferson, NJ, USA" location:{latitude:409146138 longitude:-746188906}
--- FAIL: TestSuite_NoDebugInfo (59.90s)
    --- FAIL: TestSuite_NoDebugInfo/GRPC_RED_metrics (5.01s)
        eventually.go:72: 
            	Error Trace:	/home/runner/work/ebpf-autoinstrument/ebpf-autoinstrument/test/integration/red_test.go:188
            	            				/home/runner/work/ebpf-autoinstrument/ebpf-autoinstrument/vendor/github.com/mariomac/guara/pkg/test/eventually.go:[51](https://github.com/grafana/ebpf-autoinstrument/actions/runs/4743350995/jobs/8422787779#step:4:52)
            	            				/opt/hostedtoolcache/go/1.20.3/x64/src/runtime/asm_amd64.s:1598
            	Error:      	"[]" should have 1 item(s), but has 0

Attaching test logs:

Test Logs.zip

It seems that integration test coverage files aren't always generated:

This leads to some PRs decreasing the coverage by 30%, and others restoring it.

Preparing a demo with a Grafana service, I realized that at least some grafana containers (e.g. Loki) do not include executables with DWARF info.

We will need to integrate an offsets tracker like the one from OTEL https://github.com/open-telemetry/opentelemetry-go-instrumentation/tree/main/offsets-tracker

When a handler invokes another handler (e.g. a security filter), we might report twice the same transaction. We'd need to add some kind of "cookie" that discards an HTTP handling if the entry is already in the map. Same for return handler.

Expose a prometheus HTTP endpoint for scraping RED metrics

In metrics that are generated by SpanMetrics from our Spans, the status code remains unset.
You can see here how status is correctly reported in normal metrics (left) but not in span metrics (right):

Below is the content of a trace information, as reported. It seems like they are expecting a status.code attribute (we use http.status_code):

Currently, you can select which executable to instrument by its name. This could lead to some issues in the Kubernetes scenario where deploying as a daemonset is not an option, and you need to deploy as a sidecar. For example, when you have 100 nodes and 10 instances of the service.

In that case, to avoid that a sidecar instruments processes from other pods in the same node, we could configure the instrumenter to listen to e.g. "process listening on port 8080". Since port space is internal to the pod, you can be sure you are instrumenting only the right executable.

Placeholder task. To be completed with subtasks when we design it.

In the App O11y Grafana plugin, there is a placeholder for Go runtime information but we aren't providing data:

We should look into their code to see which metric names and format use each section, to be compliant with them.

It would be similar to the image that is locally created to generate the eBPF code. It will be integrated within our CI to allow:

• Check drone drift (requires adding drone to the image)
• Check that ebpf code is generated
• Run tests from drone before creating/pushing the image

Probably via github actions

Currently we track GRPC, Goroutines and HTTP in a single file. This will be a bit unmaintainable as long as we keep growing in languages and libraries.

Currently, the instrumenter looks for the offsets of all the instrumentable functions (GRPC, goroutine queuing, HTTP...) and returns a map with all the found offsets, which are passed to a single eBPF loader.

I'd suggest to do the following architectural change:

The instrumenter should look for all the instrumentable functions, but should return multiple maps that would trigger different parts of the code selectively (for example, if only net/http Serve functions are found, load the BPF code of the HTTP tracer and the BPF code for the Goroutine tracer).

This will require:

• splitting the go_nethttp.c code into go_grpc.c, go_server_common.c etc.... This will require multiple ebpf/... subpackages.
• Optionally, sharing ringbuffers when HTTP and GRPC are traced at the same time.
• Modifying the executable offsets inspector.

Enable prometheus metrics export as an alternative to OTEL. Given an endpoint, the auto-instrumenter will push the metrics there.

When enabling the OS-level HTTP tracer, some kernel versions aren't loading with this message:

integration-javaautoinstrumenter-1  | time=2023-06-06T15:36:29.797Z level=ERROR msg="can't instantiate instrumentation pipeline" err="instantiating start instance \"ebpf\": loading and assigning BPF objects: field KprobeSysExit: program kprobe_sys_exit: apply CO-RE relocations: load kernel spec: no BTF found for kernel version 5.15.49-linuxkit: not supported"

Might it be that the function is defined as int BPF_KPROBE(kprobe_sys_exit, int status) { ?

Most Grafana services use that library. We should support it for dogfeeding.

Will go into another repo.

Placeholder task. To be completed with subtasks when we design it.

Current test coverage metrics are much lower than required, as they only include unit tests.

Go 1.20 allow exporting coverage from standalone executables: https://go.dev/blog/integration-test-coverage