autometrics-dev / diff-metrics Goto Github PK

A github action that comments on PR to describe impact on metrics

TypeScript 99.51% JavaScript 0.49%

diff-metrics's Introduction

Autometrics Report

This GitHub action will comment on Pull Requests to tell you how metrics are going to be affected.

The report tells you immediately if your new feature is well instrumented, and shows a useful summary of the metrics that will be reported without needing to go through the diff.

Inputs

gh-token: a github token that gives access to
- the PR
- the repo
- read/write access to comments on issues/PR
The built-in ${{ secrets.GITHUB_TOKEN }} will work, you do not need to create a new one. To make the built-in token work, the job must be given a specific set of permissions. The permissions added in the "Example Usage" section show the minimal set of permissions needed.
rs-roots: a list of project roots for rust projects, one root per line. The values are given relative to the root of the repository, and should point to the directory containing the Cargo.toml file.
ts-roots: a list of project roots for typescript projects, one root per line. The values are given relative to the root of the repository, and should point to the directory containing the package.json file.
go-roots: a list of project roots for golang projects, one root per line. The values are given relative to the root of the repository, and should point to the directory containing the go.mod file.
py-roots: a list of project roots for python projects, one root per line. The values are given relative to the root of the repository, and should point to the directory containing the actual module source code. If you have a monorepo, you need to specify the root of each submodule.
retention-days: the number of days to keep the list of functions as workflow artifacts. By default it will use the same retention settings as the settings in the repository (by setting retention-days to 0)
am-version: a string that allows to choose the version of am_list to download/use. You can skip the patch (e.g. 0.2) or the minor (e.g. 0) to tell the action to download the latest version that falls within the bound. Defaults to an empty string (""), which means "download the latest version, semver-wise"

Argument	Mandatory
gh-token	yes
rs-roots	no
ts-roots	no
go-roots	no
py-roots	no
retention-days	no
am-version	no

Outputs

The action does 2 things:

it writes comments to pull request giving the monitoring impact of the Pull Request.
it saves the data used to compute this report as workflow artifacts. Workflow artifacts stay private to the repository that created them, but this allows for further processing if need be.

Example Usage

The job must only contain the checkout step and the diff-metrics step, the steps that follow within the job would act on an older version of the repository.

name: Metrics report

on: [pull_request]

jobs:
  build:
    # The task only runs on linux x64 machines.
    runs-on: ubuntu-latest

    # Permissions are necessary to be able to edit and write comments on the PR
    permissions:
      issues: write
      pull-requests: write
      repository-projects: read
      contents: read

    steps:
      - uses: actions/checkout@v3
      - uses: autometrics-dev/diff-metrics@v1
        with:
          gh-token: ${{ secrets.GITHUB_TOKEN }}
          rs-roots: |
            .

Mono repo example

In the case of a mono repo that would look like

.
├── project-a
│  ├── README.md
│  │ ...
│  └── Cargo.toml
├── project-b
│  ├── README.md
│  │ ...
│  └── Cargo.toml
├── project-c
│  ├── README.md
│  │ ...
│  └── Cargo.toml
├── project-d
│  ├── README.md
│  │ ...
│  └── go.mod
├── project-ts
│  ├── README.md
│  │ ...
│  └── package.json
└── README.md

The step using diff-metrics would look like this:

uses: autometrics-dev/diff-metrics@v1
with:
  gh-token: ${{ secrets.GITHUB_TOKEN }}
  rs-roots: |
    project-a
    project-b
    project-c
  go-roots: |
    project-d
  ts-roots: |
    project-ts

Language support

Look at the issues in the repository to see the advancement of language support. All languages in the table will be eventually supported.

Language	Support
Rust	✅
Typescript	✅
Go	⚠️¹
Python	✅
C#	❌

Golang's version detects functions decorated with //autometrics directives, but does not deal with net/http middleware wrapper yet. ↩

diff-metrics's People

Contributors

Stargazers

Watchers

diff-metrics's Issues

Add Python support to the action

A py-roots input should be added to the action, and allow to track changes within a Python project

Blocked by autometrics-dev/am_list#7

Follow breaking change of `am_list` 0.3

am_list 0.3 has a flurry of breaking changes, but the easiest way to deal with those is to just go fetch the module/function names at their new locations, and leave all the rest as-is.

Add CONTRIBUTING guide to manage releases

Github Actions release management model relies on moving tags (a moving v1 tag that follows the latest v1.X tag that also moves and follows the last v1.X.Y tag).

If we want to manage multiple major releases, we will need to add and protect separate branches for each major release, and maintainers will need to be able to delete protected tags on the repo (to be able to move them)

The CONTRIBUTING guide should also mention using sample repositories to test the action integrated in a real repo that should show changes.

Report the percentage of annotated functions in the new/deleted functions in PR

When a PR adds or remove functions (but mostly when adding), we want to show in the summary the percentage of affected functions that are annotated for autometrics. The goal is to provide a good view about the future metrics coverage of new code that gets added in a PR.

This needs support from am_list to also report the total list of functions in a code base, on top of reporting the list of annotated functions.

Show inclusion to/removal from SLOs for annotated functions in the comment

Blocked by autometrics-dev/am_list#13

Once am_list returns the objectives an annotated function is part of, then we can also add that information in the generated comment (to tell when a function is added/removed from an objective)

Release v2 with Python support once octokit issue resolved

Blocked by #68

Breaking change release because rust-roots got renamed to rs-roots

Add invocation information in the datasets that get saved as artifacts

For multi-language projects, or even for single-language ones, it's important that the artifact produced has all the necessary information about the invocation that produced it, mainly:

the version of am_list used, and
the language for which it was generated (the -l option)

Always fetch latest version of `am_list`

Once am_list is stable ( autometrics-dev/am_list#15 ), we can always fetch the latest version instead of having to download a hardcoded version (either fixed in the code of the action like in the PoC phase, or as an action input with a default value).

This makes it easier for all users to just get the best version possible of the annotated functions in the repository

Add Golang support to the action

A go-roots input should be added to the action, and allow to track all the changes in Go projects.

The preliminary support for it has been added in am_list already.

Rename repository

Now that the action is out with a proper slug (autometrics-report), the repo name makes little sense.

Maybe renaming the repo to report-action would be better? Either way, we'll need to just double check that everything will keep working and that the README is updated if we do the change.

Create an artifact with the dataset computed on push events

The artifact should contain the datasets computed from the after ref of the push, as well as the datasets computed from the before ref of the push.

Keeping both sets will allow to easily compute the differences linked to a specific push event, as well as compute the differences between separate push events that are far apart.

The retention duration of the artifact should be a parameter of the action