Add a switch to measure wall clock time about tasty-bench HOT 4 CLOSED

Yes, instance IsOption Mode. This allows to specify mode as a command line argument, but more importantly supports also fine-grained tuning via setOption: you likely want to measure wall time only for benchmarks of subprocesses, but not for anything else.

from tasty-bench.

I would be interested in adding a way to do this:

• At which level should the flag exist? Cabal flag or a new field in Benchmarkable or sth else?
• How to measure (monotonic) time? getMonotonicTimeNSec (only available since GHC 8.4), a new external library like clock or criterion-measurement, or custom FFI?

from tasty-bench.

This should be customizable on per-benchmark level, something akin to

data Mode = CpuTime | WallTime 
instance Options Mode where ...

For time measurement I'd use https://hackage.haskell.org/package/tasty-1.4.2.1/docs/Test-Tasty-Runners.html#v:getTime. Alternatively getMonotonicTimeNSec is probably also fine.

What's your motivation to work on this? Do you have a use case, where measuring CPU time is not sufficient?

from tasty-bench.

So make Mode a tasty option via IsOption? And thanks for the pointer to getTime in tasty, completely forgot that tasty also has to measure monotonic time.

Do you have a use case, where measuring CPU time is not sufficient?

Yes, I played around with sth where I have to call out to subprocesses (in my case, z3), and I want that to be part of the benchmarked time.

from tasty-bench.