cirruslabs / cirrus-ci-annotations Goto Github PK

Since we've open sourced the agent where the annotations are used we should just move code from this repository to the agent one: https://github.com/cirruslabs/cirrus-ci-agent. So there is no need for an additional release in the agent repo.

First, thanks for the convenient annotation support! <3

I'm using deadnix and found out its report format isn't (yet) supported by Cirrus.
As I'm not fluent in Go, I wrote a quick adapter in jq instead:

# Explode { results: [a, b, c], ...foo } into a+foo, b+foo, c+foo
del(.results) as $self | .results[] | . + $self |

# Map each object to the schema expected by Cirrus CI
{
 "path": .file,
 "level": "warning",
 "message": .message,
 "start_line": .line, "end_line": .line,
 "start_column": .column,
 "end_column": .endColumn
}

You can see it in action on here for instance

eg https://cirrus-ci.com/task/6526868808794112

Helpful for Ruby Code Style.

https://docs.rubocop.org/en/stable/

Example:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="0" file="tests.py" name="tests.Tests-20190419163018" skipped="0" tests="7" time="2.447" timestamp="2019-04-19T16:30:21">
	<testcase classname="tests.Tests" file="tests.py" line="129" name="test_deps" time="1.968" timestamp="2019-04-19T16:30:20">
		<system-out>
<![CDATA[]]>		</system-out>
		<system-err>
<![CDATA[]]>		</system-err>
	</testcase>
	<testcase classname="tests.Tests" file="tests.py" line="42" name="test_dividers" time="0.025" timestamp="2019-04-19T16:30:20">
		<system-out>
<![CDATA[]]>		</system-out>
		<system-err>
<![CDATA[]]>		</system-err>
	</testcase>
	<testcase classname="tests.Tests" file="tests.py" line="95" name="test_info" time="0.002" timestamp="2019-04-19T16:30:20">
		<system-out>
<![CDATA[]]>		</system-out>
		<system-err>
<![CDATA[]]>		</system-err>
	</testcase>
	<testcase classname="tests.Tests" file="tests.py" line="137" name="test_make_div" time="0.001" timestamp="2019-04-19T16:30:20">
		<system-out>
<![CDATA[]]>		</system-out>
		<system-err>
<![CDATA[]]>		</system-err>
	</testcase>
	<testcase classname="tests.Tests" file="tests.py" line="118" name="test_restructuredtext" time="0.445" timestamp="2019-04-19T16:30:21">
		<system-out>
<![CDATA[]]>		</system-out>
		<system-err>
<![CDATA[]]>		</system-err>
	</testcase>
	<testcase classname="tests.Tests" file="tests.py" line="62" name="test_splitter" time="0.001" timestamp="2019-04-19T16:30:21">
		<system-out>
<![CDATA[]]>		</system-out>
		<system-err>
<![CDATA[]]>		</system-err>
	</testcase>
	<testcase classname="tests.Tests" file="tests.py" line="70" name="test_utilities" time="0.006" timestamp="2019-04-19T16:30:21">
		<system-out>
<![CDATA[]]>		</system-out>
		<system-err>
<![CDATA[]]>		</system-err>
	</testcase>
</testsuite>

taken from https://cirrus-ci.com/task/5434806341992448

go test -json ./... can produce a nice stream of json events for tests that can be converted to annotations. The only caveat is that it doesn't seem possible to have both json output to a file and a normal output to console at the same time: one for realtime log streaming and the other for generating annotations.

FYI @edigaryev

JetBrains recently released first EAP of Qodana: a CLI to run IntelliJ inspections nicely in CI. I've attached report example. Looks like it has all the data we need.

result-allProblems.json.zip

ESLint is able to generate reports in json format, e.g.:

yarn run eslint -f json -o reports/eslint.json app/javascript/controllers/index.js

Here is an example eslint.json from the reports folder.

The example file to test is app/javascript/controllers/index.js:

// This file is auto-generated by ./bin/rails stimulus:manifest:update
// Run that command whenever you add a new controller

import {application} from "./application";

import SearchableController from "./searchable_controller";

import BackSearchableController from "./back_searchable_controller";

import SortableController from "./sortable_controller";

import TestController from "./test_controller";

application.register("searchable", SearchableController);
application.register("back-searchable", BackSearchableController);
application.register("sortable", SortableController);
application.register("test", TestController);

Its corresponding plain text report looks like this:

  4:9  error  Replace `application` with `·application·`  prettier/prettier

✖ 1 problem (1 error, 0 warnings)
  1 error and 0 warnings potentially fixable with the `--fix` option.

Right now RawDetails of the annotation is empty. We can parse it from entries like:

{"testID":4,"error":"Expected: <0>\n  Actual: <1>\n","stackTrace":"package:test_api            expect\ntest/counter_test.dart 7:7  main.<fn>.<fn>\n","isFailure":true,"type":"error","time":3350}

Helpful for Ruby specs (tests)

https://rspec.info/

go vet produces an output easy to parse:

# github.com/cirruslabs/cirrus-ci-annotations/util
util/location.go:28:15: err declared and not used

Similar to an already existing model we're already using to parse other formats into, so that other tools aware of the Cirrus could generate a JSON compatible with it.

Hello.

I see annotations like this:

But I don't see cop names there.

For example, regular CLI output:

> toys rubocop
[2020-09-08 20:45:56   ANY]  ["rubocop", "rubocop"]
Inspecting 13 files
....C...W....

Offenses:

exe/compile/_base_command.rb:22:10: C: Style/SoleNestedConditional: Consider merging nested conditions into outer unless conditions.
			abort unless system 'toys config check'
         ^^^^^^
exe/compile/pages_command.rb:89:4: W: Lint/UselessMethodDefinition: Useless method definition detected.
			def render_partial(file_name, base_dir: PAGES_TEMPLATES_DIR, **options) ...
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

13 files inspected, 2 offenses detected, 2 offenses auto-correctable

So, there are Style/SoleNestedConditional and Lint/UselessMethodDefinition. And they're important: for example, to check changelog for them. Or I can have a PR with additional gem update, rubocop-performance, and it has its own namespace, Performance/, and I want to know if I have errors from this add-on or there are errors from a new rubocop version (or errors with CI strategy, like it was actually).

So, please, output somewhere a cop name, for example before offense text, like it's in CLI.

I've checked, there are cop names in JSON output:

{
	"metadata": {
		"rubocop_version": "0.90.0",
		"ruby_engine": "ruby",
		"ruby_version": "2.7.1",
		"ruby_patchlevel": "83",
		"ruby_platform": "x86_64-linux"
	},
	"files": [{
		"path": ".toys.rb",
		"offenses": []
	}, {
		"path": "Gemfile",
		"offenses": []
	}, {
		"path": "exe/_bundler_setup.rb",
		"offenses": []
	}, {
		"path": "exe/compile.rb",
		"offenses": []
	}, {
		"path": "exe/compile/_base_command.rb",
		"offenses": [{
			"severity": "convention",
			"message": "Style/SoleNestedConditional: Consider merging nested conditions into outer `unless` conditions.",
			"cop_name": "Style/SoleNestedConditional",
			"corrected": false,
			"correctable": true,
			"location": {
				"start_line": 22,
				"start_column": 10,
				"last_line": 22,
				"last_column": 15,
				"length": 6,
				"line": 22,
				"column": 10
			}
		}]
	}, {
		"path": "exe/compile/all_command.rb",
		"offenses": []
	}, {
		"path": "exe/compile/assets_command.rb",
		"offenses": []
	}, {
		"path": "exe/compile/data.rb",
		"offenses": []
	}, {
		"path": "exe/compile/pages_command.rb",
		"offenses": [{
			"severity": "warning",
			"message": "Lint/UselessMethodDefinition: Useless method definition detected.",
			"cop_name": "Lint/UselessMethodDefinition",
			"corrected": false,
			"correctable": true,
			"location": {
				"start_line": 89,
				"start_column": 4,
				"last_line": 91,
				"last_column": 6,
				"length": 88,
				"line": 89,
				"column": 4
			}
		}]
	}, {
		"path": "exe/compile/pdf_command.rb",
		"offenses": []
	}, {
		"path": "exe/compile/view_object.rb",
		"offenses": []
	}, {
		"path": "exe/deploy.rb",
		"offenses": []
	}, {
		"path": "extra/deploy_api.rb",
		"offenses": []
	}],
	"summary": {
		"offense_count": 2,
		"target_file_count": 13,
		"inspected_file_count": 13
	}
}

Hello.

It'd be nice to support remark.

It has CLI, which based on https://github.com/unifiedjs/unified-args, which supports reporters: https://github.com/unifiedjs/unified-args#--report-reporter

Including JSON: https://github.com/vfile/vfile-reporter-json

Hello.

I have such config:

# ...

rspec_task:
  container:
    matrix:
      image: ruby:2.4
      image: ruby:2.5
      image: ruby:2.6
      image: ruby:2.7
  <<: *bundle_cache

  test_script: toys rspec --format=json --out=rspec.json

  always:
    rspec_artifacts:
      path: rspec.json
      type: text/json
      format: rspec

And annotations are duplicated from every image:

It seems logically, but maybe we can collapse, or hide them, or don't report duplicates from matrix?

I'm not sure, if you think that's correct — OK.

See TAP -- it's pretty easy to parse, and a lot of test harnesses can produce TAP

See https://github.com/TOPdesk/dart-junitreport/blob/master/tool/example.jsonl as a test report example.

https://github.com/spotify/XCLogParser can produce a json output with error lines. Worth investigating how to support it.

See https://github.com/jstemmer/go-junit-report