[Question] When does Qdrant remove deleted vectors? about qdrant HOT 7 CLOSED

Hey @retinio,

there are optimization parameters:

                "deleted_threshold": 0.2,
                "vacuum_min_vector_number": 1000,

which define condition per segment when deleted vectors should trigger the optimizer.

Please note, that once deleted, vectors are not affecting search results in any way

from qdrant.

@generall
If my collection has deleted vectors more then points count Qdrant's optimization never run?

from qdrant.

As Andrey mentioned, those parameters are per segment. You have quite a bit of segments (48), so with this number of points it makes sense it has not been run yet. I wouldn't worry about it.

It is done this way because actually removing the vectors from disk immediately is more expensive than keeping them until enough have been deleted.

Also note that the point/vector counts are approximate and should not be relied upon. That is described here.

from qdrant.

@timvisee Thanks!
I have got one more question.
I have tried to reduce count of segments by setting in the config

storage:
    optimizers:

      # If the number of segments exceeds this value, the optimizer will merge the smallest segments.
      max_segment_number: 5     

Do I understand correctly when optimizer will be run my bit of segments (48) will be merged?

from qdrant.

Yes. According to your collection info you shared above you have "default_segment_number": 0, which means its chosen automatically. It defaults to the number of CPUs you have, which is likely why it is 48. Changing the above value like you suggested should reduce it further.

When changing this, you do need to trigger the optimizers at least once. I have drafted a documentation page on how you could do that, you can see a preview of it here. Sending another update operation, such as upsertion a point, is fine as well.

Note that a small number of segments is fine as long as you have a low number of points. If you plan to scale your setup, you likely want to stick to the default of 48 (your number of CPUs).

from qdrant.

thank you for answering @timvisee

from qdrant.

It defaults to the number of CPUs you have, which is likely why it is 48.

More precisely, it is number of sahrds times number of CPUs

from qdrant.