[RFC] Classifying benchmarks based on running time about sandmark HOT 5 CLOSED

Thanks @singhalshubh.

from sandmark.

Having things classified such that people can select what they run classified by runtime is reasonable, often for the initial run you just want to look at a collection of the macro benchmarks.

We have had success at pin-pointing issues with some of the micro-benchmarks (e.g. C call overhead, cost of local roots in the hash function, read-barrier overheads, stack switching costs), so I wouldn't argue that they are without use. However they are probably only useful when you want to look at something specific where a micro-benchmark makes sense.

If we classify by runtime, then it is possible to have a script which can automatically tag them into the runtime buckets.

from sandmark.

We have had success at pin-pointing issues with some of the micro-benchmarks (e.g. C call overhead, cost of local roots in the hash function, read-barrier overheads, stack switching costs), so I wouldn't argue that they are without use.

That makes sense. It would be useful to associate with the microbenchmarks what they're meant to be measuring in comments.

If we classify by runtime, then it is possible to have a script which can automatically tag them into the runtime buckets.

Yes, but I suspect that since we need to do this just once, manually tagging them might be faster. :-)

from sandmark.

Can we add a couple of more tags for this classification on the running time?

• Hours: The sample benchmark tests forCoq run for few hours, and it will be good to see how the the code performs over time.
• Days: Database performance benchmarks with varying loads are run for at least 24-48 hours as longevity tests to see their behavior. Since, these systems are meant to be run without downtime, it will be good collect data for such longer periods, and see how the system behaves. Irmin benchmarks could possibly be added.

from sandmark.

Lets close this issue. Its resolved in #188

from sandmark.