Following https://keepachangelog.com/en/1.0.0/

There are too many spurious changes in PR from times I forgot to run the formatter. We want CI to enforce gofmt -s to avoid issues like this.

The PR adding the step will obviously also need to run the formatter on the whole project.

The purpose of this argument is to disable documentation generation by the generator. This would allow the VS Code extension to become solely responsible for generating documentation tooltips.

This is a blocker for finalizing autometrics-dev/vscode-autometrics#28

Feel free to suggest a better name for the argument 😅

The --version bit should use linker flags in a build script to be relevant

This is a stepping stone to deal more largely with #44 later.

The use-case for contexts

Currently the runtime context of autometrics embeds a context.Context just because it was needed when we added OpenTelemetry support. Now there's a specific need for contexts:

• We want to be able to pass data between function calls (to transmit the traceId for example)
• We want to be able to pass arbitrary data (it is the only way to have autometrics work with any instrumented code)

Therefore, instead of always initializing the context to a default one, we want to add the options and ability to create an autometrics.Context:

• with a context.Context parent, and/or
• with specific key-value pairs embedded (a TraceId and a SpanId)

End goal

Creating a context like that will allow autometrics to fetch any data using its embedded context, therefore enabling autometrics to attach any runtime data to the generated metrics

Support for popular libraries

Doing this will allow, in a second time, to use the function signatures of instrumented functions to detect what popular framework they use. If a function has func hello(ctx *gin.Context) as its signature, we could detect the gin context and generate code that will extract and add this information directly in autometrics. At that point, having tracing info in the metrics would only be one middleware away (the one that calls ctx.Set(autometrics.TraceIdKey, "traceId"))

Being able to do this needs a big refactoring of the internal.generate package though, since it kind of assumes there's only 1 way to generate the //autometrics:defer statement

This very, very likely implies writing a wrapper function to be used as middleware just like the TS version does. It should use the facilities from #47 to inject the traceID and spanID in the context, and wrap the response writer in order to read the status code and automatically mark the handler as successful or not.

This also calls for an addition to the autometrics:inst directive, so that each function could specify the range of status codes that should be considered an error from the point of view of autometrics. It will default to anything >400 and could optionally take (multiple) --http-code-ok-range 100,499 --http-code-error 404 arguments to override the range that's considered OK (in a "last rule wins" manner)

If not already possible, i would like for the metrics to be namespaced if a namespace option was specified

autometrics.Init(
    autometrics.WithNamespace("imaginary_application"),
)

function_calls_duration_seconds_count would become imaginary_application_function_calls_duration_seconds_count

if this is not already possible then i would be happy to raise a PR to add this

Pending #47 merge.

Once the PR is merged, the ability to inject a new trace ID/span ID into a Gin context will only happen through a middleware library, to properly fill the autometrics.MiddlewareTraceIDKey and SpanIDKey fields with hex-encoded byte strings of the relevant IDs

Tracking issue for autometrics-dev/autometrics-shared#21

This needs the change in autometrics-shared (for the alerting rules and the dashboard) before it can be meaningfully acted on here

To be useful in a go generate context, autometrics need to be able to know the information about the function it "decorates".

To have the lightest API possible (i.e. not needing to specify the function name as a go generate argument) instead we want to parse the file, find the function declaration that follows the comment, and then extract from that:

• the function name
• the name of the named return values (we're interested in the error type)

Hints:

• go/parser.ParseFile
• go/ast.File
• go/ast.FuncDecl
• go/ast.Field

See autometrics-dev/autometrics-shared#61

The main metrics names are

const COUNTER_NAME_PROMETHEUS: &str = "function_calls_count";
const HISTOGRAM_BUCKET_NAME_PROMETHEUS: &str = "function_calls_duration_bucket";
const GAUGE_NAME_PROMETHEUS: &str = "function_calls_concurrent";

and should be created in an autometrics.Init function that could be called in the main package of the autometrics users.

All metrics MUST have function (for the function name) and module (for the qualified package name) in the labels

See autometrics-dev/autometrics-shared#74

Pending #47 merge.

Once the PR is merged, the ability to inject a new trace ID/span ID into an Echo context will only happen through a middleware library, to properly fill the autometrics.MiddlewareTraceIDKey and SpanIDKey fields with hex-encoded byte strings of the relevant IDs

Having the binary for the generator in the assets of a release will make it clearer that there's supposed to be a binary working with the library, cf. #29

Adding support for OpenTelemetry on top of Prometheus should be done in a few steps:

• Add an argument to the go generator to choose between Prometheus and OpenTelemetry
• Add support for using OpenTelemetry or Prometheus in the autometrics.Instrument method (as argument?)
  • Do we even need one? It seems that for tracing applications you need to do a lot of wrapping anyways around your whole service and not just a few functions.
• Add a comment generator for the OpenTelemetry visualizers
• Add an environment variable to read the base URL when it's OpenTelemetry (or rename the existing Prometheus one into something more generic)
• Add an example in the repo that uses OpenTelemetry instead of Prometheus
• Think about how alerts can be handled (so how to put labels on metrics when relevant)

This could be used to speed up the process of instrumenting a code base if you know you want to instrument everything in a given file.

• //go:generate autometrics --inst-all (remember in already accepts --no-doc argument) and //go:generate autometrics --doc-all`
• Also, to be reentrant, the generator MUST always walk all functions in the file, removing //autometrics:defer statements before choosing whether to process them
• //go:generate autometrics --rm-all could also be added to clean everything up in one go generate call for offboarding

Ref: autometrics-dev/autometrics-rs#70

I think that Prometheus may have some subtle differences in how it handles counters and gauges, especially around server restarts. As a result, I think it's better for all the implementations to use monotonic counters for the function calls.

https://github.com/autometrics-dev/autometrics-shared

Hello, and thanks for the interesting project. I actually heard about it from Evan at Berline.rs meetup.

I'm trying to add this to my project, but it looks like i need the 'autometrics' binary, but neither readme specify how to actually install it, nor releases contain any binaries.

Can you please explain/link - how to get the binary the go generate command runs?

❯ go generate ./...                       
server/api/v1/server_api_v1.go:3: running "autometrics": exec: "autometrics": executable file not found in $PATH

❯ go version
go version go1.19.2 linux/amd64

See autometrics-dev/autometrics-shared#28

Giving out the context created by autometrics.PreInstrument is the best (only?) way to enable library users to pass the context down to the function's callees if they wish to.

Having autometrics context through a complete call tree will allow to have cleaner traces support, and better caller information for the call graph feature.

We could also reuse part of the middleware code, to shadow the function argument context with the autometrics one if we are able to make a type-compatible variant.

This is useful for the middleware usages, where the instrumentation is actually written by the user, but would still want the doc comments to contain links to prometheus and stuff.

Mostly reading information there and making sure to capture it if present https://www.w3.org/TR/trace-context/

The function will create a <function_name>_autometrics function that will use a named return value err to decide whether or not function_name call is successful or not.

The goal is that the code modification to make autometrics work is

+ //go:generate go-autometrics
- func My_handler(args: interface{}) error {
+ func My_handler(args: interface{}) err error {
+        defer My_handler_autometrics()
         // Do stuff
         return nil
}

Not entirely sure what the support is like upstream: open-telemetry/opentelemetry-go#559

Panic Stack:

runtime error: slice bounds out of range [99:78]
/usr/local/go/src/runtime/panic.go:154 (0x44453b)
	goPanicSliceB: panic(boundsError{x: int64(x), signed: true, y: y, code: boundsSliceB})
/root/go/pkg/mod/github.com/autometrics-dev/[email protected]/pkg/autometrics/instrument.go:69 (0x1b444ce)
	callerInfo: callInfo.Parent.Function = functionName[index+1:]
/root/go/pkg/mod/github.com/autometrics-dev/[email protected]/pkg/autometrics/ctx.go:262 (0x1b42104)
	FillTracingAndCallerInfo: callInfo := callerInfo(ctx)
/root/go/pkg/mod/github.com/autometrics-dev/[email protected]/prometheus/autometrics/instrument.go:130 (0x1b48bb1)
	PreInstrument: ctx = am.FillTracingAndCallerInfo(ctx)
/root/InnerGitlab/wms-query/pkg/api/query_svc_map.go:150 (0x2590f19)
	(*Handler).QueryServiceMapSingleConnection: amCtx := autometrics.PreInstrument(autometrics.NewContext(

After debug, I found that in this code line

it still uses functionName(which is github.com/autometrics-dev/autometrics-go/prometheus/autometrics.PreInstrument in my case), but the parent function name is gitlab.ubiservices.ubi.com/zhuyanxi1998/wave/services/monitoring/component/query-service/pkg/api.(*Handler).QueryServiceMapSingleConnection, which is longer than functionName, so it out of range.

I think in Line69, the variable functionName might be changed to parentFrameFunctionName.

There is a fair amount of content in the readme. Someone that was integrating the Rust library commented that it would be useful to have a clear quickstart guide for anyone that doesn't need to be convinced about the project but just wants to follow a set of steps to add it. This is what we added for the Rust library

The new function should be named <function_name>_register_metrics or something. We can figure out where and when to call it later.

My code:

package handler

import (
	"gmod/pkg/version"
	"net/http"

	"github.com/autometrics-dev/autometrics-go/prometheus/autometrics"
	"github.com/gin-gonic/gin"
)

//go:generate autometrics
//autometrics:inst
func (h *Handler) VersionHandler(ctx *gin.Context) {
	defer autometrics.Instrument(autometrics.PreInstrument(autometrics.NewContext(
		nil,
		autometrics.WithConcurrentCalls(true),
		autometrics.WithCallerName(true),
	)), nil) //autometrics:defer

	ctx.JSON(http.StatusOK, gin.H{
		"version": version.Version,
	})
}

First run of go generate ./... runs smooth, properly generates docs and instruments code. Executing it second time causes panic:

❯ go generate ./...
panic: runtime error: slice bounds out of range [:-1]

goroutine 1 [running]:
github.com/autometrics-dev/autometrics-go/internal/generate.cleanUpAutometricsComments({{{0x911459, 0x3}, {0x0, 0x0}, {0x0, 0x0}, 0x1, 0x1, 0x0}, {0x0, ...}, ...}, ...)
        /home/krzwiatrzyk/go/pkg/mod/github.com/autometrics-dev/[email protected]/internal/generate/documentation.go:43 +0x691
github.com/autometrics-dev/autometrics-go/internal/generate.walkFuncDeclaration(0xc000166000, 0xc000188000, {0xc0000322da?, 0x8cd560?})
        /home/krzwiatrzyk/go/pkg/mod/github.com/autometrics-dev/[email protected]/internal/generate/generate.go:145 +0x29b
github.com/autometrics-dev/autometrics-go/internal/generate.GenerateDocumentationAndInstrumentation.func1({0x9c6da0?, 0xc000188000?})
        /home/krzwiatrzyk/go/pkg/mod/github.com/autometrics-dev/[email protected]/internal/generate/generate.go:100 +0x4b
github.com/dave/dst.inspector.Visit(0xc000182240, {0x9c6da0?, 0xc000188000?})
        /home/krzwiatrzyk/go/pkg/mod/github.com/dave/[email protected]/walk.go:341 +0x31
github.com/dave/dst.Walk({0x9c82c0?, 0xc000182240?}, {0x9c6da0?, 0xc000188000?})
        /home/krzwiatrzyk/go/pkg/mod/github.com/dave/[email protected]/walk.go:52 +0x68
github.com/dave/dst.walkDeclList({0x9c82c0, 0xc000182240}, {0xc000132e60?, 0x2, 0x50?})
        /home/krzwiatrzyk/go/pkg/mod/github.com/dave/[email protected]/walk.go:38 +0x69
github.com/dave/dst.Walk({0x9c82c0?, 0xc000182240?}, {0x9c6d60?, 0xc00011ef70?})
        /home/krzwiatrzyk/go/pkg/mod/github.com/dave/[email protected]/walk.go:321 +0x173e
github.com/dave/dst.Inspect(...)
        /home/krzwiatrzyk/go/pkg/mod/github.com/dave/[email protected]/walk.go:353
github.com/autometrics-dev/autometrics-go/internal/generate.GenerateDocumentationAndInstrumentation({{{0x911459, 0x3}, {0x0, 0x0}, {0x0, 0x0}, 0x1, 0x1, 0x0}, {0x0, ...}, ...}, ...)
        /home/krzwiatrzyk/go/pkg/mod/github.com/autometrics-dev/[email protected]/internal/generate/generate.go:110 +0x2c5
github.com/autometrics-dev/autometrics-go/internal/generate.TransformFile({{{0x911459, 0x3}, {0x0, 0x0}, {0x0, 0x0}, 0x1, 0x1, 0x0}, {0x0, ...}, ...}, ...)
        /home/krzwiatrzyk/go/pkg/mod/github.com/autometrics-dev/[email protected]/internal/generate/generate.go:54 +0x2e8
main.main()
        /home/krzwiatrzyk/go/pkg/mod/github.com/autometrics-dev/[email protected]/cmd/autometrics/main.go:77 +0x1c5
pkg/handler/version.go:36: running "autometrics": exit status 2

A "supported" latency is a latency target that exists in the buckets of the function_call_duration histogram. If the latency used in the autometrics directive is not exactly one of the bucket values of the histogram, then the latency alert will never trigger.

In the same vein, a "supported" objective target is a percentage that exists in the bundled configs/autometrics.rules.yaml file. Support to create custom objective targets might be added later (using Sloth binary and types to generate a matching rules file with custom objectives, like the Rust version does), but for the time being, if the objective latency percentile/success rate is not one of the precompile rules file, then the latency/success rate alert will never trigger.

The autometricsGeneratorContext should be augmented with the supported value sets for each arguments, so that it can use the Validate method to bail out and error if users try to use the generator with unsupported latencies/objectives

This setup will be useful to spin up an integration test of the package

Getting Prometheus alerts generation

As a reminder, the rust implementation uses this kind of syntax to trigger the generation of alerts for a single function:

#[autometrics(alerts(success_rate = 99.9%, latency(99% <= 200ms)))]
pub async fn handle_http_requests(req: Request) -> Result<Response, Error> {
  // ...
}

We want to provide a similar experience with the Go version, by exploiting the //autometrics:doc directive currently used per-function. This issue proposes a design for the feature as well as a few technical solutions to create the feature.

Reusing Sloth

The rust implementation is relying on Sloth, there's no reason to avoid it here. If we're lucky enough, as it's written in Go, we might be able to reuse the types from its library to serialize alerting rules (We can just build the relevant Go structure and Marshal those)

New argument to autometrics directive

We probably want to specify that the generator should both create documentation and alerts, so a good matching syntax would be

//autometrics:doc,alerts --success-rate 99.9 --latency-perc 99 --latency-threshold 200ms
func handleHttpRequests() (err error) {
        // ...
        return nil
}

That would allow to just split the directive arguments by ,, and then use something as shlex to parse the alerts parts just like CLI flags.

New argument to the go:generate directive

The //go:generate autometrics directive will need to take an extra argument to point to the global location of the Sloth alerts file, otherwise we won't know where to write. The syntax could be

//go:generate autometrics --alerts-file ../../autometrics.sloth.yaml

Having a CLI flag instead of a positional argument (i.e. //go:generate autometrics ../../autometrics.sloth.yaml) helps with backwards compatibility later.

Data races

We want the calls to the generator to concatenate all the Sloth rules to a unique file, so that sloth (the binary) can generate the prometheus rules we want.

autometrics (the go-generator) is only called once per file, so we can safely generate a Sloth fragment per single call to autometrics. Once that's done, we can use a small file-lock library like fslock to query a global, out-of-process, lock on the resulting global file in order to safely concatenate the Sloth fragments in the same file

We would still need to add a step at the end to generate the Prometheus rules. Maybe each autometrics call could use their lock to also keep it while exec-ing sloth to regenerate the Prometheus rules. That would mean that the last call to get the lock will be the one producing the final file. As long as we can guarantee that the last process that has the lock, also is the process with the last version of the sloth rules, we're fine.

It is important to help potential users assess the performance impact of the library. I suppose we could add an extra script to the example app that also measures the runtime impact of autometrics?

That means we need to find the currently used Golang profiling tools to plug it to the demo application.

Question

How do we make the documentation of a function contain generated links to queries to Prometheus?

Issues

• Go does not want you to get smart and access the documentation of functions. You MUST have the documentation just above the function declaration
• Go generate does not work well with modifying the code in place. All the arguments you get from environment variables when calling go generate would be out of date as soon as you start editing the file in place. So that's a huge race condition waiting to happen

Possible solutions

Generate an extra package

The proposed solution would be to generate a single file autometrics package, with doc_gen.go containing a huge documentation string for the package, with a heading per decorated function.

The advantage here is that we can use file-locking on the doc_gen.go to actually fix the race condition issue. But the disadvantage is that you have to explore the huge documentation of the generated autometrics package to find the correct
documentation

Two-pass design

The proposed solution would be to have autometrics-go actually be 2 go:generate binaries, to run in 2 passes on the code:

A single top level generate call per file would exist:

package main

//go:generate autometrics


// indexHandler is a cool function.
//
// It is used to handle the `/` route
//
//autometrics:doc
func indexHandler(args interface{}) error {
        return nil
}

First pass: the comment pass

That pass would detect all the //autometrics comment and:

• replace the comment in place with extra comments about the generated prometheus queries with links (we
  want to have that code marked or under a heading so it's easy to replace in case someone wants to reset those)
• find the function that's being commented in the AST and mark it (add function_name, module pair to a list) for the code generation pass.
• ensures that the error is a named returned value

Second pass: the code generation pass

We can now use the list of (function_name, module) collected from the first pass to actually generate a <package>_autometrics_gen.go file to deal with the creation of defer-able functions that will be actually used as wrappers inside the client code (which would be defer <function_name>_autometrics())

Decision ?