How to optimize config for a huge movie collection track ? about olaf HOT 1 OPEN

Interesting to hear your experiences. I have indeed tested Olaf with 800GB of music and have not experimented with larger datasets or non-music (less information rich) signals.

Some insights/pointers/possible optimisations:

• Depending on how similar the reference and query audio is and how long a query is allowed to be the configuration can be optimised: if you want to support short (1s) noisy queries you need a lot of fingerprints, if you can deal with long (20s+), clear queries, the system can be configured to use much less fingerprints.
• To speed up db creation you might want to look at lmdb bulk import which expects sorted keys. The idea here is to extract fingerprints, sort them externally and do a single bulk import. This is not supported by olaf by default but is a potential performance improvement.
• There will be a lot of duplicate fingerprints in your dataset. I suspect the slowdown and size increase to be related to these hash matches. A B+tree essentially becomes an inefficient list if there are many hash collisions.
• To further reduce fingerprints you might want to look at a silence treshold. Now Olaf also extracts fingerprints from quiet parts which perhaps can be skipped for your use case.
• Olaf can be distributed: perhaps you can maintain 10 seperate instances and put an api in front of Olaf to distribute queries and return results.

Good luck with your project

from olaf.