Benchmark with @benchmark only one sample? about benchmarktools.jl HOT 4 CLOSED

It is. Before running the actual performance experiment, BenchmarkTools runs a "tuning" process in which the provided kernel is executed several times in order to gauge how many kernel evaluations per sample is needed to minimize error due to timer precision/accuracy.

See https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#introduction

from benchmarktools.jl.

Would you please keep this discussion open for a while?

I don't think it is reasonable for the benchmark to conclude that only a single sample is sufficient. Clearly even when the tuning decides that the execution time for one sample is reasonably long, there is still scatter. In the referenced example, the scatterer is actually between something like 9.6 and 10.1 over five runs. In my opinion that is significant.

from benchmarktools.jl.

Would you please keep this discussion open for a while?

People can still comment on a closed issue, so feel free to discuss. But this is intended behavior, so there's no action item, and hence I closed the issue.

I don't think it is reasonable for the benchmark to conclude that only a single sample is sufficient.

BenchmarkTools doesn't ever make any statistical decisions about how many samples to take. It takes at least one sample, and then as many additional samples as possible within the user-provided time and sample budget. As described in the docs, the default time budget is 5 seconds, so BenchmarkTools took as many samples as it could (which, of course, is only one).

from benchmarktools.jl.

I see. I missed the part about being able to control the number of samples. Thanks a bunch.

from benchmarktools.jl.