Minhash LSH Forest Top-K results differs from MinHash LSH about datasketch HOT 4 CLOSED

On the different top-k query results: this is expected. Because the methods are all approximate rather than exact, results may not be 100% accurate. The common work flow of LSH Forest for top-k is to first retrieve "top-r" results from the index, sort the r number of results by exact similarity computation (or using MinHash signatures), and then output the top-k. r is bigger than k (e.g. 10x bigger).

For MinHashLSH, you cannot directly control the number of results returned. When your result sizes are large you may want to use a higher threshold (e.g. 0.9). Have multiple LSH indexes optimized for a series of descending thresholds can help too.

from datasketch.

Thanks for your suggestions! You mentioned "Have multiple LSH indexes optimized for a series of descending thresholds can help too". Can you give me some more guidance on how can I achieve this? Thanks!

from datasketch.

Nothing fancy. What I meant is just some version of the following:

lshes = {0.9 : MinHashLSH(threshold=0.9),
              0.8: MinHashLSH(threshold=0.8),
              0.7: MinHashLSH(threshold=0.7),
              0.6: MinHashLSH(threshold=0.6),
              0.5: MinHashLSH(threshold=0.5),}

So your query processing can start with the highest threshold, try the lshes in descending order of thresholds, until you get enough query results.

from datasketch.

OK, I understand. Thanks!

from datasketch.