Optimization task panicked after a collection is recovered, causing search API timeout about qdrant HOT 5 OPEN

I do indeed expect a corrupted snapshot state somehow.

We did just merge #4132 which likely fixes this panic. It'll be part of our next release.

In 1.9 we also improved snapshots, also improving data consistency, which would likely solve this from happening in the future. Here's the related PR if you're interested: #3420

from qdrant.

Hey @no7dw, thanks for reporting this.

Could you please provide a bit more details on how the snapshot was created (on which version of the engine?)

from qdrant.

the current version is v1.9 while the backup snapshot is created by v1.8.3

we used the following command just follow the docs:

backup:
curl -s -X POST "$QDRANT_PROD_URL/collections/$COLLECTION_NAME/snapshots"

restore:
curl -X POST "$QDRANT_DEV_URL/collections/$COLLECTION_NAME/snapshots/upload" -H 'Content-Type:multipart/form-data' -F "snapshot=@/data/qdrant/snapshots/$COLLECTION_NAME/$SNAPSHOTS_NAME"

additional details:
we're investing (not sure about this yet) the operation cause, one of is which is: our team once deleted some of the unused vectors(multiple model but only one is used in prod), detail was disclosed in this issue

from qdrant.

Does the very same happen repeatedly? Or was this a one time occurrence? That isn't entirely clear to me from the issue description.

from qdrant.

happen repeatedly , once I use a new vm, after restore one of the collection (new_text) will soom become
"error": "Service internal error

from qdrant.