Benchmarks do not run for long enough; too biased by non-reading code performance about biofast HOT 2 CLOSED

Not everyone is working with human data. The test data file gives >100-fold coverage for a bacterial genome. For real datasets at this scale, startup time matters. Long startup time is a weakness and should get penalized.

In addition, I don't have the bandwidth to run these tools on much larger datasets. Even the fastest implementation would take several hours with hyperfine.

from biofast.

@lh3

In addition, I don't have the bandwidth to run these tools on much larger datasets. Even the fastest implementation would take several hours with hyperfine.

I am sure folks can help. For example, I could spin up an ec2 instance each week to update the benchmarks on a big dataset for those implementations that have been added or changed.

At least add a disclaimer in the README, since this benchmark will likely get treated as gospel since folks value your opinion highly. They don't consider the nuances or limitations of the benchmark (aka. times where startup time doesn't matter like the 100gb fastq benchmark you argued for elsewhere!)

from biofast.