Ability to insert items with non-integer keys about annoy HOT 12 CLOSED

No hesitations but it's probably not trivial. You would have to store a mapping that works well with mmap meaning it has to be packed in contiguous memory. Strings can have different length etc so I'm not sure how to do that easily.

You could probably prepend the index with something like <id1><string1>NULL<id2><string2>NULL... and binary search in that index... I don't know

from annoy.

the other solution is just put it in a separate file and use some existing hashmap format like bdb, TC,

from annoy.

Or just store an indirect pointer alongside the item vectors, and have it point to a separate variable-sized chunk of memory after the vectors but before the tree nodes.

from annoy.

yes but that only gives you mapping in one direction, what about the other direction? you need some kind of binary search or hashing to make that work

from annoy.

I guess instead of using a contiguous sequence of nodes you can treat the whole annoy index as a hashmap and use something like cuckoo hashing? that way you just need to store the mapping from nodes to keys, not the other way around.

from annoy.

Oh, I see, Chris is asking about general keys. Yeah, then it needs to be pretty much like a sparkey hash.

from annoy.

Given that Sparkey already supports this with mmapping how would you feel about solving it with an annoy dependence on Sparkey?

from annoy.

I don't think Sparkey is available in PyPI so not sure if we want to depend on it

from annoy.

So maybe we can hold off on implementing for python annoy at the moment, but should be fine for the java version. In the meantime I can also try and get Sparkey submitted to PyPI. If we get Sparkey in PyPI then does this seem like a clean enough solution?

from annoy.

Sure. If you rely on Sparkey you could actually ignore the C++ part and implement this as a pure Python solution

from annoy.

Just realized that there is a C library for Sparkey so should be possible to do this outside of Python altogether. Would be nice to manage the dependencies through Travis but not sure how that would work with the travis.yml being language: python. Any ideas?

from annoy.

I don't know either maybe a git submodule?

It sounds like a lot of work to get this working compared to the benefit tbh... it's not super hard to just store a hash table separately. I think this is also how the other ANN libraries work (flann, sklearn, panns, nearpy)

from annoy.