Refactoring & Optional data concurrency about counter-rs HOT 4 OPEN

Sounds good. With regard to benches, I'd recommend criterion instead of nightly benches. I want to expose the entire surface of the library, including benchmarking, to end-users without requiring a nightly toolchain.

from counter-rs.

Refactoring the code is fine, to an extent. This is very much a style thing, and I don't want to pre-approve anything that I'll regret later on. That said, I agree that 2k lines is too big for a well-factored source file.

I'm willing to look at a PR adding Rayon, but that PR should include some benchmarks showing at what data magnitude the feature is justified, and documentation exposing that information to end users. Rayon is sometimes a game-changer, but you can't just naively throw it at problems and expect to see an improvement.

FWIW: In the context of

    let mut items = self
        .map
        .iter()
        .map(|(key, count)| (key.clone(), count.clone()))
        .collect::<Vec<_>>();

I would be extremely surprised to discover that par_iter performed better under any circumstance. That said, I'm willing to be convinced by a well-crafted benchmark.

Either way, a refactor is a wholly separate concern from adding feature-gated Rayon support, so these should be two distinct PRs.

from counter-rs.

I agree with your points.
Here is my plan

1. prepare a separate PR for refactoring. A lot of it will be stylistic choices, and finding an acceptable solution will help me understand what you envision for the repo.
2. when 1. is done, add benches against the current version of the code. It is likely that this will be a separate PR since there are some opinionated choices to be made such as nightly #[bench] vs criterion. The data i am planning to handle are in million/billion scale (LLM dataset deduplication/counting) I'll likely add benches that go that high.
3. Prepare a rayon PR that includes rayon integration and documentation, with further benches as identified as necessary.

from counter-rs.

1. is started in #36

from counter-rs.