Noise in Sandmark about sandmark HOT 4 OPEN

Now with ASLR turned off

Still the noise is around 2%.

from sandmark.

It looks like most of the benchmarks are not actually very noisy (the noise observed is well below 1%), and a smaller group of benchmarks are noisier. This suggests that you could track per benchmark how many iterations to run and what's the expected noise level, and try to have more stable benchmarks and fewer unstable benchmarks to keep the total running time in check. In particular, most of the long-running benchmarks are not noisy in this test, so maybe they could be run a single time. knucleotide is the only >=10s noisy benchmark.

Regarding error bars: instead of error bars specific to a run (this requires several iterations to show error bars), you could store for each benchmark its typical "noise range" (the largest observed difference in past noise-detecting runs), and display those as error bars for all future runs. This gives good visual feedback when looking at benchmark graphs, without requiring several iterations.

from sandmark.

Noise has been reported for the soli benchmark for 5.1.0+trunk.
Reference: ocaml/ocaml#11102 (comment)

from sandmark.

Noise has been reported for the soli benchmark for 5.1.0+trunk.
Reference: ocaml/ocaml#11102 (comment)

As mentioned in the linked comment, soli running time is too small. Either we make it run longer or remove the macro benchmark tag. It was a mistake to have tagged it as a macro benchmark in the first place.

See #348. I believe I've fixed a number of these to run longer.

from sandmark.