Question: Setting Expiry Time for Entries in Redis LSH? about datasketch HOT 7 CLOSED

Setting allkeys-lru or expiration times on top-level Redis keys could lead to inconsistency.

from datasketch.

This is not possible with the current implementation. In general, this would be challenging because it is not possible to set expiry on Redis hash child keys (according to SO at least - https://stackoverflow.com/questions/16545321/how-to-expire-the-hset-child-key-in-redis).

Check out https://github.com/ekzhu/datasketch/blob/master/datasketch/storage.py#L360 to see where the problems lie.

from datasketch.

@ae-foster Thanks for the information.

By inconsistencies in setting an allkeys-lru, do you mean that we would lose the hashes present in the top level Redis key? Or would end up with any other issues?

I am trying to keep my system resilient in case we do end up hitting max memory limits. I am not too concerned if we lose hashes we are trying to find similar matches with, in case we run into such an issue. But of course, I would definitely be concerned if the system itself stops running properly.

from datasketch.

Some of the values in Redis are other Redis keys. You could end up with a case where you look up the contents of a key (having read it as a value, e.g. by SMEMBERS), and find it (erroneously) empty because that key was deleted. Could cause unexpected behaviour!

from datasketch.

Thanks for the info.

from datasketch.

@SpandanThakur Maybe you can consider using the MongoDB storage @aastafiev is working on at #53? This should be able to utilize disk instead of keeping all data in memory.

from datasketch.

Sure @ekzhu , I will check out the pull.

from datasketch.