When testers find a significant difference, it is nice if they can profile the benchmark code, first with the baseline version and then with the comparison version, in order to determine which Underscore functions have slowed down (or sped up) most. For this to produce useful numbers, it is important that the benchmark runs with only one version at a time; otherwise, the profiler is likely to just average the results.

As discussed in #1 it is required that running this benchmark on Windows is also possible (excluding Windows would cut off a lot of testing targets).

We specifically need to find a way to replace the pull.sh script with a cross-platform solution, either by using git-submodule or reimplementing the behavior as a Node.js script.

The benchmark package does (at least) two questionable things:

1. In order to decide whether the performance difference is significant, it uses the Mann-Whitney U test, which is meant for ordinal scale data, instead of Student's t test, which is more appropriate for ratio scale data such as the performance measurements we're making here.
2. It recompiles JavaScript code from a string on every test run in order to prevent engine optimizations. Usually we're interested in performance with engine optimizations, since this is how code tends to run in the real world. Besides that, since most of the benchmark code is probably out of reach for this compilation trick, optimization is only partly disabled, producing inconsistent optimization characteristics. Maybe this behavior can be disabled; this is worth investigating.

I can think of a couple of options (from least to most effort):

• Accept the quirks and do nothing.
• Switch to an alternative benchmark framework that doesn't have these quirks. I'm not (yet) aware of an alternative.
• Forego a convenient benchmark framework. Instead, repeat the benchmark code a fixed number of times (say, 10) for each version of Underscore and report each individual result, so that people can compute their own statistics.
• Fork the benchmark package, fix the issues, submit a PR. Use our own version regardless of whether the PR is accepted. If not accepted, publish as a separate package on NPM.

I see that generate-fixtures.js spends some effort to ensure that one third of events is mobile and a quarter has a referrer. Is this based on observations from real-world data?

Otherwise, the randomized values seem to be drawn from a flat distribution. All events are of type PAGEVIEW. Are these choices approximately realistic? For example, are other event types rare?

I'm asking because the distribution of values may affect the proportions of time spent in alternative branches of the code, which in turn may affect the running time.

If we end up concluding that the data are not very representative, one possible solution may be to take real data and replace each unique string by another, non-identifying string (using wholesale search/replace) in order to arrive at a single, censored-but-representative dataset.