12. Results aggregation about bacalhau HOT 3 CLOSED

We're going to compare traces rather than outputs for evaluation.

from bacalhau.

We'll implement a basic trace comparison function used for confidence results in future milestones.

NOTE: this is a rabbithole and a big amount of data science work to get it robust.

Web interface might be useful to visualize the running system...

from bacalhau.

Notes on trace comparison...

The simplest time-series similarity measurement is the cross-correlation coefficient. Basically that’s where you place the signals on top of each other, then multiply each point, sum them all up. You can normalise if you have a frame of reference. Higher values mean the signals are more similar.
But it depends very much on the use case. For eg. how do you define similarity? If it “looks” similar? Or “sounds” similar. That’s one common interpretation between normal cross correlation and cross spectral density.
Outliers cause issues too if the process is noisy.
If the data can we “warped” i.e. squashed like doppler, then there’s methods to unwarp the signal.
And then there’s just good old mean squared error. If two signals are the same, then when subtracting one from the other (and squaring and summing), you should be left with zero. Anything non zero means the signals are not the same.
Lots of different ways!

Also, if you have more data, more signals, then there may be a more abstract space that gives a better measure of similarity. E.g. euclidian distance in many dimensions (for eg. cpu, ram. temp, gpu, hash of data, etc. etc.).

from bacalhau.