Comments (13)
Well, I tried...
main...mindplay-dk:iso-bench:collect-test-results
I tested this in the context of the benchmark, and it did work - but it takes way longer than the specified 2 seconds to actually measure 2 seconds worth of runs.
So this is somehow adding a lot of runtime overhead. I'm not sure why.
But I honestly don't fully understand this code. 🤔
from iso-bench.
@mindplay-dk is because you changed the minMs
for that, but you need to change the ms
property.
The minMs
property is the amount of milliseconds a single sample needs to run to take into account its output to the total ms
, so if you set minMs
as 2 seconds, it is going to re-run the samples, increasing the amount of cycles in each sample, until one single sample reaches the 2 seconds and is not discarded.
What you want is to run multiple small samples and get the average, so if you want 2 seconds, just set ms
to 2000ms. By default minMs
is set to 100ms, so it means that is going to gather a minimum of 100ms on each sample until it reaches 2000ms, which is about 20 samples.
In short, ms
is the verry same as maxTime
in benny, and minMs
is the very same as minTime
in benny:
/**
* The maximum time a benchmark is allowed to run before finishing (secs).
*
* Note: Cycle delays aren't counted toward the maximum time.
*
* @default 5
*/
maxTime: number
/**
* The time needed to reduce the percent uncertainty of measurement to 1% (secs).
*
* @default 0
*/
minTime: number
benny will increase the cycles of each sample until they reach the minTime
, discarding all other samples, and will run samples until they reach the maxTime
. benny also has the minSamples
option that iso-bench doesn't have. The same for benchmark.js (also named minTime and maxTime. I guess I have to rethink my option names xD). They are common options in benchmark suites.
PS: I'm gonna clean the code. It was made yesterday afternoon in a couple hours just to see that everything works as expected.
from iso-bench.
I see.
Shouldn't minMs
just be an implementation detail with a useful default maybe?
When is it useful to have control of this value?
from iso-bench.
On that note, I was confused by the "samples" value output in the console as well - why would you need to know how many cycles the framework used internally to collect the results?
I've been trying some different math on the collected values - not sure if it makes sense to include any or some of this in the library itself? You can see an example here:
For a benchmark like the one I'm trying to fix, probably the most reliable and meaningful value is the min
, which also happens to be the simplest to calculate.
The mean
is also simple enough, but quite unreliable, as there are always outliers skewing the result.
The typical
value is a little "invention" of my own - I tried a whole bunch of different algorithms to see which ones give the most stable results between runs. It's fairly complex and a bit magical. It might make more sense to include a simple distance-weighted mean, which works almost as well.
Of course, the point of exposing the individual values was so you didn't need to own this code, but it might be nice if something a bit more immediately useful came out of it.
What do you think?
from iso-bench.
I see.
Shouldn't
minMs
just be an implementation detail with a useful default maybe?When is it useful to have control of this value?
It had a 100ms default value, which worked well for all my tests. The best way to run IsoBench is to run it with default values.
Anyway, I published a new update with different options, with default values that return more consistent results.
On that note, I was confused by the "samples" value output in the console as well - why would you need to know how many cycles the framework used internally to collect the results?
I don't know. benchmark.js shows it so I also done so xD
I've been trying some different math on the collected values - not sure if it makes sense to include any or some of this in the library itself? You can see an example here:
For a benchmark like the one I'm trying to fix, probably the most reliable and meaningful value is the
min
, which also happens to be the simplest to calculate.The
mean
is also simple enough, but quite unreliable, as there are always outliers skewing the result.The
typical
value is a little "invention" of my own - I tried a whole bunch of different algorithms to see which ones give the most stable results between runs. It's fairly complex and a bit magical. It might make more sense to include a simple distance-weighted mean, which works almost as well.Of course, the point of exposing the individual values was so you didn't need to own this code, but it might be nice if something a bit more immediately useful came out of it.
What do you think?
I'm not sure what those results mean :-(
min
is the sample with less opeations per second?
I'm not an algorithms guy, so I went with the mean which is the one I can implement easily, and I think that is the one that benny and benchmark.js use. The thing is that they clean outputs depending on the percentaje difference between samples and IsoBench still doesn't. That's something I will add in the future because I think that this logic that they explain in the post is the one that can give the more stable results.
The new release also returns the test array in the run()
method, but right now is kinda basic (just the information it uses to calculate the mean, although the last release by default runs only 1 sample). I'm going to update it over time to add more useful information.
EDIT: Cleaned the code and added a Processor system so benchmark output is easier to manage. This is in preparation to add events during the benchmarking process, and also to be able to output charts, tables, HTML and so on.
from iso-bench.
I'm not sure what those results mean :-(
ha, no, of course not, I didn't actually post the code! sorry 😅
well, the changes I had locally were only working with my branch that exposes the raw numbers, and you current version still doesn't do that.
but I can dump my functions here, in case any of it is useful to you.
here's what I was printing out:
for (const { name, values } of tests) {
console.log({
name,
min: minOf(values),
mean: meanOf(values),
typical: weightedMean(values, 10)
})
}
and here's all the math behind that:
function weightedMean(values: number[], factor = 10) {
const buckets = sortIntoBuckets(values, 100)
.filter((bucket) => bucket.values.length > 0)
.map((bucket) => ({
...bucket,
weightedAverage: distanceWeightedMean(bucket.values, factor)
}))
let totalWeight = 0
let weightedSum = 0
for (const bucket of buckets) {
const weight = Math.pow(bucket.values.length, factor)
totalWeight += weight
weightedSum += weight * bucket.weightedAverage
}
return weightedSum / totalWeight
}
function meanOf(values: number[]): number {
let sum = 0
for (const value of values) {
sum += value
}
return sum / values.length
}
function minOf(values: number[]): number {
let min = Number.POSITIVE_INFINITY
for (const value of values) {
if (value < min) {
min = value
}
}
return min
}
function maxOf(values: number[]): number {
let max = 0
for (const value of values) {
if (value > max) {
max = value
}
}
return max
}
type Bucket = { min: number; max: number; values: number[] }
function sortIntoBuckets(values: number[], numBuckets: number): Bucket[] {
const min = minOf(values)
const max = maxOf(values)
const range = max - min
const buckets: Bucket[] = []
for (let i = 0; i < numBuckets; i++) {
buckets.push({
min: min + (range * i) / numBuckets,
max: min + (range * (i + 1)) / numBuckets,
values: []
})
}
for (const value of values) {
const index = Math.min(
Math.floor(((value - min) / range) * numBuckets),
numBuckets - 1 // note: max of bucket range is inclusive *only* for the last bucket
)
buckets[index].values.push(value)
}
return buckets
}
function distanceWeightedMean(values: number[], factor = 2): number {
if (values.length === 0) {
throw new Error('cannot computed weighted average of an empty list')
}
if (values.length === 1) {
return values[0]
}
if (values.length === 2) {
return (values[0] + values[1]) * 0.5
}
let sum = 0
for (const value of values) {
sum += value
}
const avg = sum / values.length
const error = (value: number) => Math.abs(avg - value)
let deviation = 0
for (const value of values) {
deviation = Math.max(error(value), deviation)
}
if (deviation === 0) {
return values[0]
}
let totalWeight = 0
let weightedSum = 0
for (const value of values) {
const weight = Math.pow(1 - error(value) / deviation, factor)
totalWeight += weight
weightedSum += weight * value
}
return weightedSum / totalWeight
}
as said, the min
value really is the most stable and reliable, as it comes the closest to ignoring "less than ideal" conditions for every run.
but I guess, technically, that could depend on what you're benchmarking... which is why I was trying a bunch of different things.
the distanceWeightedMean
is fairly simple and popular for a lot of things - Amazon uses it to average product ratings.
my weightedMean
implementation isn't strictly only a weighted mean - it's actually sorting the results into "buckets", and then calculating a weighted mean, with weights according to the number of values that fall into those buckets.
I have no idea if this is better or worse than what benchmark.js is doing - it looks like hard core statistical stuff, so I'm out of my league... personally, I would favor something non-scientists have a shot and understanding... or if I had to use whatever that statistical algorithm is, I would find a third party package with a well tested implementation. I wouldn't suggest copy/pasting stuff from benchmark.js unless you really understand what you're copying - even if it works well, because you're going to own it then. 😅
but you do you of course. 👍
I'm having a bit of trouble keeping up with all the changes, so I will probably set this project on the back burner until you think you're finished. 🙂
from iso-bench.
oh, one little thing, since you can't merge my changes now - I do think you need to make a small change here:
Lines 155 to 161 in 1194a7a
Something like this instead:
private _runTest(test:Test, cycles:number): number[] {
const values: number[] = [];
while(cycles-- > 0) {
const startTS = process.hrtime.bigint();
test.callback();
values.push(Number(process.hrtime.bigint() - startTS) / 1000000);
}
return values;
}
You're measuring the time it takes for the loop itself to execute - while this may seem insignificant, the faster/smaller the function you're measuring, the more it'll skew the results.
from iso-bench.
I'm having a bit of trouble keeping up with all the changes, so I will probably set this project on the back burner until you think you're finished. 🙂
Yeah sorry 😅 it is now getting closer to what I want to achieve. Just pushed a new update with all the new Processor logic, so it will be easier to implement new Processors or custom ones.
Thank you for all the code. When I finish with the Processor thingy I'm going to analyze all these algorithms, and maybe even add an option to calculate different ones (min, mean, "typical"), because what each person is trying to achieve in each case is different.
You're measuring the time it takes for the loop itself to execute - while this may seem insignificant, the faster/smaller the function you're measuring, the more it'll skew the results.
Good catch but this is intentional because of exactly that reason. For fast methods (for example a buffer.readuintUint8
) only measuring each run adds a lot of skew because when you call process.hrtime.bigint()
, NodeJS goes inside the native layer to gather the data, and that's more expensive than just the while(cycles-- > 0) {
line. When the data is gathered from the native layer you already have your time and you can waste as much CPU cycles as you want, but while the call is "travelling" to the native layer the time is still ticking because the native layer still doesn't know that you asked for the time, and that "travel" is VERY expensive. Also, not every system can achieve nanosecond resolution, so in those systems the skew is going to be astronomical as it may return "this lasted 0 milliseconds, now 1 millisecond, now 0 milliseconds again...". By adding them up to a single loop that must last a minimum amount of milliseconds, you also reduce that skew in those systems.
You can test it by running this simple script:
const ITERATIONS = 10_000_000;
const values = [];
function expensiveFunction() {
/asdasd/.test("dfghsdfghdfgjasdasddftghsdfhg");
}
function testEachRun() {
let cycles = ITERATIONS;
while(cycles-- > 0) {
const startTS = process.hrtime.bigint();
expensiveFunction();
values.push(Number(process.hrtime.bigint() - startTS) / 1000000);
}
console.log("testEachRun", values.reduce((total, value) => total + value, 0));
}
function testAllRuns() {
let cycles = ITERATIONS;
const startTS = process.hrtime.bigint();
while(cycles-- > 0) {
expensiveFunction();
}
console.log("testAllRuns", Number(process.hrtime.bigint() - startTS) / 1000000);
}
testAllRuns();
testEachRun();
As you can see, if you measure each run, the total time is almost triplicated for the very same amount of iterations, and that's because all the extra time it takes to call the process.hrtime.bigint()
method. If you only call it one time instead, you reduce the skew.
By the way, the skew is not important as long as all the tests have the very same skew. These benchmark libraries are created to compare different code between them and see which one is faster, not to calculate the exact CPU cycles that an operation takes, as there's always a skew because at some point you need to access a clock, a counter or whatever thing to calculate the differences.
from iso-bench.
Hmm, have you tried using the platform standard performance.now()
instead?
https://developer.mozilla.org/en-US/docs/Web/API/Performance/now
from iso-bench.
Hmm, have you tried using the platform standard
performance.now()
instead?https://developer.mozilla.org/en-US/docs/Web/API/Performance/now
In Node.js that's just an alias for process.hrtime
divided by 1000000. Just exactly as I'm already doing xD
https://github.com/nodejs/node/blob/484ad833580c978d2530d2257af95a34970db454/lib/internal/perf/utils.js#L17
What I don't remember is why I decided to go with bigint to later convert it to a Number. I think that because internally nodejs works with bigint when using process.hrtime
? Going to check on that. Although I don't think I can't avoid much more skewing anyway.
from iso-bench.
This is nuts.
How did they manage to create a function just for accessing a micro timer that is this slow. 😐
from iso-bench.
How did they manage to create a function just for accessing a micro timer that is this slow. 😐
Is not that. Is just that simply calling a native function has its overhead. JavaScript drawbacks. It was not designed for performance, although V8 guys are doing whatever they can 😅
from iso-bench.
Calling any native function causes this much overhead?
I mean, they do say "javascript is slow", but... yikes. 😅
from iso-bench.
Related Issues (5)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from iso-bench.