CSV output appears to be incompatible with criterion-compare about tasty-bench HOT 4 CLOSED

I'll give https://github.com/Bodigrim/tasty-bench#comparison-against-baseline a try…

from tasty-bench.

criterion-compare expects CSV contain exactly 7 columns (https://github.com/AndreasPK/criterion-compare/blob/13ec87146ab48f706a75476b8e952ae979afd0b8/src/CsvParse.hs#L19-L30), while tasty-bench generates only three, corresponding to the first (getBenchName), second (statsMean) and fifth (doubled statsStd) columns in criterion output.

I cannot fix this on tasty-bench side simply because I do not have missing data (which is confidence intervals for mean and stdev). It could be probably worked around on criterion-compare side, but after all it is for criterion.

The simplest workaround is awk 'BEGIN {FS=","; OFS=","} ; {print $1,$2,$2,$2,$3/2,$3/2,$3/2}'.

from tasty-bench.

Actually, I can add a command-line switch to fake criterion compatible output.

from tasty-bench.

After more consideration, I'm reluctant to fix this. In general it's wrong to compare reports produced by different benchmark frameworks, especially by faking non-existing data.

To be honest, I do not feel much need in a dedicated package of this sort, because any tool from Excel to gnuplot is sufficient to plot data from two CSVs. I cannot even build criterion-compare with GHC 8.10: several of its dependencies restrict base < 4.14 or worse. And when I throw allow-newer, build fails with can't load framework: Cocoa (not found).

from tasty-bench.