[Bug]: v2.4.0 datanode 内存使用过高 about milvus HOT 11 OPEN

The title and description of this issue contains Chinese. Please use English to describe your issue.

from milvus.

Referring to the Sizing Tools, allocate Data Nodes with 2 cores of 8 GB x 2 pods.
However, during actual operation, the Data Nodes was an OOM, and after expansion, the memory usage reached 40G.

from milvus.

datanode log:

datanode.log

from milvus.

@yesyue please share more info about how you using milvus, e.g. what kinds of requests did you call to milvus, how many, and how frequency of them? also please help all the milvus pods logs for invesgitaion.

/assign @yesyue
/unassign

from milvus.

100 Million/day entites write to milvus

from milvus.

100 Million/day entites write to milvus

after I inserted 10M entites total, then milvus docker stop and crash. I use IVF_SQ8 index, installed milvus with gpu.
I use batch insert 10000 (only insert if enough 10000 entities.

after crash I can't connect to connection again and can't use anything. Any solution?

from milvus.

1. seems that flush can not catch up the read.
2. how many partitions do you have? if you have many partitions or collections, the flush and memory consumption will be larger than estimation.
3. there is bunch of configs to tune, like concurrent flush number -> dataNode.dataSync.maxParallelSyncMgrTasks (for 2.4)
   memory used for growing segment

from milvus.

100 Million/day entites write to milvus

after I inserted 10M entites total, then milvus docker stop and crash. I use IVF_SQ8 index, installed milvus with gpu. I use batch insert 10000 (only insert if enough 10000 entities.

after crash I can't connect to connection again and can't use anything. Any solution?

how much gpu memory do you have?
please open another issue with detailed logs so we can help

from milvus.

querynode (3).log

from milvus.

querynode (3).log

1.could you offer log for datanode?
2. it would be great if you have a datanode pprof, so you know which part takes of your memory. Most likely it's insert buffer takes the memory and you can tune the flush parameter

from milvus.

I saw you in many issues and we'd like to offer help.
feel free to contact me at [email protected] if necessary

from milvus.