[Bug]: sysbench bulk_insert 1000threads 98% memory usage and not released about matrixone HOT 20 OPEN

profile:
mo_profile0329.tar.gz

from matrixone.

第三次测试：(打开gc日志)
mo sysbench bulk_insert 1000threads 测试运行大概到380s时被oom kill，执行tps结果和第一次基本差不多
运行到380s时内存情况：

mo log:
mo_log2.tar.gz

profile：
mo_profile_bulk.tar.gz

from matrixone.

last gc log：
gc 58 @523.221s 2%: 0.25+4656+0.31 ms clock, 16+235/74447/169465+20 ms cpu, 232580->236203->147538 MB, 247313 MB goal, 18 MB stacks, 0 MB globals, 64 P

from matrixone.

处理其他 s-1

from matrixone.

处理其他 s-1

from matrixone.

没有进展，在做其他issue

from matrixone.

与 tn 相关占用巨大内存的地方有两个：

1. dequeuePreparing -> MutIndex 建立内存索引
2. HandleRowsInsert

初步看来主要原因是由于 flush 不及时导致，没有及时清理内存中的数据。

增快 flush 频率，可以避免 OOM

from matrixone.

在做其他 issue

from matrixone.

##15580 merge 后，oom 应该可以避免 @heni02
后面会进一步减少 tn 的内存消耗

from matrixone.

自测中

from matrixone.

SQL statistics:
    queries performed:
        read:                            0
        write:                           28595
        other:                           0
        total:                           28595
    transactions:                        917734260 (981824.22 per sec.)
    queries:                             28595  (30.59 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          934.7219s
    total number of events:              917734260

Latency (ms):
         min:                                    0.00
         avg:                                    1.00
         max:                                55095.98
         95th percentile:                        0.00
         sum:                            913347614.78

Threads fairness:
    events (avg/stddev):           917734.2600/34714.55
    execution time (avg/stddev):   913.3476/9.71

自测 15分钟已经没有再出现 OOM.

@heni02

from matrixone.

今晚回归测试验证

from matrixone.

main commit:2eeef9246a17f1dfe7265f911976caa68626defc
job：https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9019462365/job/24803638728

昨晚回归测试该流程时段内存基本占满了，但没有oom

三分钟后内存释放完

三分钟期间，连接mo失败报错

mo log and profile plz contact me

from matrixone.

在看其他issue

from matrixone.

处理其他 issue

from matrixone.

处理其他 issue

from matrixone.

@w-zr please continue to investigate this issue

from matrixone.

分布式环境也存在这个问题：

job url：https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9211125528/job/25354293997
commit：e4c504a606585cb373a5bb7c452e804469716610

from matrixone.

date5.26回归，bulk_insert导致dn oom
job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9244059586/job/25434572970

https://grafana.ci.matrixorigin.cn/d/cluster-detail-namespaced/cluster-detail-namespaced?orgId=1&var-namespace=mo-nightly-regression-20240526&var-account=All&var-interval=$__auto_interval_interval&var-cluster=.%2A&var-loki=loki&from=1716734826666&to=1716775847600
2次oom时间

profile：
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22CU2%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240526%5C%22,pod%3D%5C%22nightly-regression-dis-dn-0%5C%22%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22process_cpu:cpu:nanoseconds:cpu:nanoseconds%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221716771420000%22,%22to%22:%221716771480000%22%7D%7D%7D&schemaVersion=1&orgId=1

from matrixone.

两次OOM前的内存情况。

from matrixone.