Indexing of long text documents are tricky about annlite HOT 6 CLOSED

We are working on a feature that will allow the user to have multiple indices and sub_indices around the same DocArray API, I think this could be useful for you?

from annlite.

We are working on a feature that will allow the user to have multiple indices and sub_indices around the same DocArray API, I think this could be useful for you?

I don't know yet, how it will look like.
But Document's nested structure (chunks are senteces from long text) are suitable for this case, just annlite indexer doesn't allow to index (just store) documents without embeddings.

from annlite.

in this case you would need to have your own version of AnnLiteIndexer indexing different parts in different DocArrays, but yes current implementation does not work

from annlite.

in this case you would need to have your own version of AnnLiteIndexer indexing different parts in different DocArrays, but yes current implementation does not work

could you please explain how would sub_indices work.
When do you plan to implement it ?

Thanks

from annlite.

@tommykoctur The subindex has been released. https://docarray.jina.ai/fundamentals/documentarray/subindex/

from annlite.

Thank you, but I don't think that this would help me. I would probably add another LMDB to store root doc information to save some space.

from annlite.