Help wanted: Benchmarks in xray_core about xray HOT 9 OPEN

That just laid the framework. I think we could probably use more benchmarks though of core functionality.

from xray.

@anderoonies I honestly haven't developed strong opinions on this. My initial thought would be that we'd want to focus on situations involving more complexity. If we optimize those, presumably we'd do well in simpler cases, and conversely, if we're fast on a line without edits and slow on a line with lots of edits it still seems like we'd be too slow. That said, I think each scenario might be different. I have a lot of experience optimizing, but not much experience designing a long-lived benchmark suite. Would be happy to hear perspectives on the best design considerations.

from xray.

If anyone wants to get started on this, this is a config for running bench on nightly without preventing compiles on stable: max-sixty@bc950a8

With commands

cd xray_core/
cargo +nightly bench --features "dev"

...or reply here if there are other ways

from xray.

Another option for micro benchmarks that is stable/nightly friendly is criterion: https://github.com/japaric/criterion.rs.

from xray.

You can put the benchmarks in a benches directory in the crate root. They'll still run when you call cargo bench, but then you don't have to put them behind a feature flag.

from xray.

Can this be closed now that #62 is merged?

from xray.

i'm taking a look at this but had a question about how atomic the benchmarks should be.
right now benchmarks are done for individual functions in one scenario. should the same functions be tested in multiple scenarios—e.g. selecting to the end of a line without any edits, selecting to the end of a line that has multiple edits, etc. how should benchmarks for the same functions—but with different setups—be organized?

thanks!

from xray.

i'm curious to hear others' experience and input as well, being new to writing benchmarks myself.
the existing benchmarks @rleungx added establish a pattern of testing individual functions of the editor API under single, pretty "intense" scenarios. i'm happy to extend that to benchmark the rest of the core API.
@nathansobo, as someone very familiar with the underlying implementations, are there any behaviors of the editor you feel should be focused in benchmarking? in the rgasplit paper, briot et al use "randomly generated traces" for performance evaluations, but i'm not sure the consensus on randomness in benchmarking

from xray.

I'm not sure random edits are as important as sequential edits that simulate what a human would do. But testing against documents containing lots of edits will likely be important.

from xray.