Comments (7)
I don't think so; and any workaround would likely be too slow (swapping during indexing/search). Annoy targets the scenario where the whole index fits in main memory.
A scalable solution here would be distributing the index across multiple nodes, in a cluster. But that's not on the roadmap for Annoy AFAIK. Interesting and not that difficult algorithmically, but it would bring a lot of devops complexity.
from annoy.
I haven't tried it, but in theory Annoy works with out of core data. If you put it on an SSD drive then it probably could perform OK. I think you would experience a slowdown of say 10-100x.
One issue that comes to mind is that Annoy builds up the index in RAM and then writes it to disk – it would be more efficient to build it up directly to disk. That should actually be pretty easy to support.
Spinning disk would be ridiculously slow – probably 1000x slower at least.
More harder stuff you can do to optimize for out of core:
- Partition the index (like @piskvorky said)
- Don't store the vectors in the index, just the search structure
- Support axis-aligned splits instead of arbitrary vectors – this way the splits will only take a few bytes rather than 4kB per split (with (1000 dim))
I also encourage you to do some sort of dimensionality reduction before putting the data into Annoy – even a simple SVD down to 100D would probably help tremendously.
from annoy.
See https://github.com/spotify/annoy/blob/master/src/annoylib.h#L399 for how memory is used while adding items. It isn't until you call save
that the index is written to disk. Then later if you call load
it will perform an mmap. It would make more sense to support mmap during insertion too. Problem is afaik mmap doesn't support resizing, but you could probably just write to the file pointer, flush, then munmap/mmap again (not sure how fast it would be). This way you would always use the file system for persistance, and the kernel will use the page cache to fit as much as possible in RAM. With an SSD this would be pretty reasonable
from annoy.
@piskvorky @erikbern Thanks for your advice. I'm going to try partition the index first, and I also add two functions for get_nns_by_item
and get_nns_by_vector
to support returning both ids and distances, which are get_nnsd_by_item
and get_nnsd_by_vector
respectively.
from annoy.
@erikbern Hi, I am using annoy tree to get most accurate match of product names. My question is regarding size of tree. When I saved the tree on disk its size ig 2.3GB. But while creating the tree I can see it is using 3 times more memory. Is this okay?
from annoy.
Not clear why, how are you measuring it? Measuring memory consumption of a process is notoriously unreliable
from annoy.
from annoy.
Related Issues (20)
- How many trees should I use? HOT 2
- Memory Leak in Annoy (get_nns_by_vector)? HOT 8
- Annoy Object Not Pickle'able HOT 1
- Add sample weights to distance metric? HOT 3
- Source distribution not availabe for 1.17.2 version HOT 2
- Unable to inherit the AnnoyIndex class HOT 2
- doesn't work correctly if torch tensor is input. But also doesn't throw error. Pls add an assertion that this only takes np arrays not torch tensors HOT 2
- _Vector should use position-only parameter for the index HOT 3
- How do you reduce a vector to 2 coordinates HOT 1
- [Distance] What did I do wrong?
- [MSVC] Annoy failed to run test on Windows HOT 1
- Some segment faults HOT 1
- Regarding updating an existing ANNOY model HOT 2
- Anyone tried storing trees and nodes in DynamoDB? HOT 1
- Is there any workaround to be able to use the Chebyshev distance with this library? HOT 1
- from annoy import AnnoyIndex
- Annoy build failed in MSVC x86 mode
- Using a built Annoy tree in a different device HOT 1
- ? HOT 1
- problem building with python3.12 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from annoy.