Making benchmarks better about hashbrown HOT 5 OPEN

Also each benchmarked hashmap should be using the same hashing algorithm as otherwise you are comparing apples and oranges

from hashbrown.

Current discussion on r/cpp about benchmark many hashmaps: https://www.reddit.com/r/cpp/comments/auwbmg/hashmap_benchmarks_what_should_i_add/

It may give ideas about new benchmarks and new hashmaps to consider.

Also, it's notable that Abseil's Swiss Table seem to struggle in benchmarks featuring insert/erase. In the benchmarks in the README we see hashbrown lagging 10% behind FxHashMap in grow_by_insertion, and otherwise always being faster, so it's unclear if hashbrown also suffers from this.

from hashbrown.

@matthieu-m Are the benchmark results available somewhere?

from hashbrown.

Not that I know of, yet, short of running them yourself. At the moment the OP is gathering maps and benchmarks ideas, I am hoping that once everything is setup he'll publish some results.

from hashbrown.

For benchmarks you should replicate these. They are really good. They test:

1. LookupHit
2. LookupMiss
3. Insert
4. InsertErase (repeatedly insert and erase)
5. Iteration
6. Clear+InsertOrdered (ordered = same order as iteration order of the hashtable)
7. Clear+InsertUnordered (unordered = random order)

The tests are parametrized to run across the cross product of these dimensions:

1. payload sizes (4, 8, 16, 32, 64)
2. hot and cold tables (hot=in cache, cold=not in cache)
3. high and low density tables (high=max load factor, low=min load factor)

There is a good discussion in the forum about the design of these benchmarks and how they are useful. The most notable thing we found about benchmarks when implementing SwissTable is that benchmarks that predict production performance are very hard to write. In the end we wrote benchmarks to act as tools of understanding how a table behaves under certain loads and conditions: lookup/insert/miss, hot/cold, dense/sparse, etc. The only benchmarks we found useful for the hashtable as a whole was running large production workloads against them. In your case you have rustc-perf.

from hashbrown.