Combinators to assert performance metrics about tasty-bench HOT 7 CLOSED

@alexbiehl-gc could you possibly give a try to a01a3ff? I've added a basic support of baselines. It works as follows:

1. Dump benchmark results in CSV format:

$ cabal bench --benchmark-options '--csv fibo.csv'
All
  fibonacci numbers
    fifth:     OK (1.32s)
      155 ns ±  13 ns
    tenth:     OK (1.89s)
      1.8 μs ± 104 ns
    twentieth: OK (1.94s)
      235 μs ±  20 μs
    from file: OK (2.00s)
      1.9 μs ± 133 ns

Now fibo.csv contains

Name,Mean (ps),2*Stdev (ps)
All.fibonacci numbers.fifth,154592,13086
All.fibonacci numbers.tenth,1787280,104364
All.fibonacci numbers.twentieth,235140039,19850780
All.fibonacci numbers.from file,1892341,132588

2. Change implementation as desired.
3. Rerun benchmarks, providing previous report as a baseline:

$ cabal bench --benchmark-options '--baseline fibo.csv'
All
  fibonacci numbers
    fifth:     OK (1.46s)
      172 ns ±  15 ns
      11.3% slower than baseline
    tenth:     OK (2.08s)
      2.0 μs ± 174 ns
      11.8% slower than baseline
    twentieth: OK (4.23s)
      258 μs ±  11 μs
      9.6% slower than baseline
    from file: OK (2.18s)
      2.1 μs ± 119 ns
      9.8% slower than baseline

How does it look?

from tasty-bench.

Sweet! I think this is a great first step! I am unsure how an assertion over these would look like. As for the assertion one possible idea could be to redefine Benchmarkable as

data Benchmarkable = Benchmarkable { 
  benchmark         :: Int64 -> IO (),
  baselineDeviation :: Maybe Double
}

and have another combinator to annotate Benchmarkables. Now, if a baseline deviation and --baseline is present, the results are being compared and the test succeeds or fails accordingly.

Another possibility would be to use Tasty's own localOption this would allow application of baseline deviations for single tests and groups alike.

from tasty-bench.

b0448b4 adds --fail-if-slower PERCENT command-line option and a corresponding tasty Option for fine-grained control:

$ cabal bench --benchmark-options '--baseline fibo.csv --fail-if-slower 10'
All
  fibonacci numbers
    fifth:      FAIL (1.32s)
      156 ns ±  13 ns, 24% slower than baseline
    ten.th:     OK (1.92s)
      1.8 μs ± 128 ns
    twen,tieth: OK (3.84s)
      236 μs ± 9.6 μs
    from file:  OK (2.02s)
      1.9 μs ± 177 ns

1 out of 4 tests failed (9.15s)

Passing --fail-if-slower 30 still reports about 24% slowdown, but does not fail the benchmark.

If it looks good, I'll proceed with documentation.

from tasty-bench.

This is good stuff. Wanted to have this for some time at work but never got to it. Thank you for your efforts!

One last nit: Personally I’d prefer a dedicated combinator instead of plain localOption.

from tasty-bench.

One thing: It might be good to have another, stricter version of the combinator that fails if too slow or too fast. It would help always be on top and aware of changes that cause baseline adjustment.

from tasty-bench.

One last nit: Personally I’d prefer a dedicated combinator instead of plain localOption.

Probably not at this stage: I'd like to see this feature getting significant traction first.

It might be good to have another, stricter version of the combinator that fails if too slow or too fast.

Good idea. Added --fail-if-faster.

I'd appreciate if you give this functionality a try, before I release it officially.

from tasty-bench.

https://hackage.haskell.org/package/tasty-bench-0.2.1 has been released, including this feature.

from tasty-bench.