CI test output `results.json` contains a lot of duplicate entries about julia HOT 6 CLOSED

We do have @testset for already, so maybe that can somehow combine them? Or just automatically match up all tests with the same line number, and combine them into a single report? I don't know if that needs to be done before generating, or if the later UI tools are capable of merging those already.

from julia.

It has entries for all non-Pass results, and some tests do @test_broken in loops.

This isn't a bug, but arguably it could de-duplicate test failures in the reporting itself.

from julia.

This is a bit of a tangent, but for cases where there's a test run in a loop, I wonder if we could somehow do better than

for i in [...]
     @test f(i)
end

with something like

@bulktest for i in [...]
    @test f(i)
end # Combine into a single test result somehow

or similar

from julia.

This isn't a bug, but arguably it could de-duplicate test failures in the reporting itself.

Yea, I think that would be good. I'll see about implementing that.

from julia.

@tecosaur on slack I proposed that we just make the results.json contain testset objects.
If the testset has any failures, then group them into the testset somehow.

Recording Results objects seems too tricky because Test.jl doesn't retain Pass info for memory reasons (that seem hard to avoid) and recording testsets are good because they always exist, pass or fail or otherwise, which seems better for the way buildkite analytics is set up.

from julia.

Ah right, I see the sense in that. From a "tracking test results over time" perspective I still think there's value to be had in test-level granularity, but am not sure what's the best balance of concerns.

One PR I'm most of the way through spinning up is just merging duplicate entries, which seems like a completely clear improvement.

from julia.