Run index operations in the background about llama_index HOT 2 CLOSED

I think you'll need to spin up separate worker servers and queue tasks, as you said. I don't think this is specific to the gpt_index project, and I wouldn't expect built-in support.

However, if you go ahead and rewrite gpt_index in something lower level like C or Rust, then that's something that could potentially be beneficial in making runtime indexing more plausible. I looked into Cython to maybe do this myself, but that might only make a marginal difference since some native Python methods are already implemented with CPython and not really the underlying bottleneck.

The bottleneck was the embedding step using Ada embeddings in my use case. This is due to roundtrips from OpenAI servers to generate the embeddings, so I guess threading could be a solution, however, you'll just run into other issues like throttling or 429s which is why I think built-in gpt_index improvements shouldn't be expected.

from llama_index.

@teoh we are actively improving the runtime of index building and querying, so hopefully this will be less of an issue in the near future!

As @aliyeysides noted, the feature you are describing might be best handled in the application layer (i.e. outside of GPT Index). We are moving more and more underlying logic to be executed async, but the top level API would remain blocking for ease of use.

from llama_index.