[BACKPORT][v1.6.2][IMPROVEMENT] Investigate performance bottleneck in v1 data path about longhorn HOT 4 CLOSED

Testing inside virtual machine might be ok too. But looks like the seq write inside virtual machine is capped at 10k somehow. Is it the max that VM can achieve natively?

I believe it may be limited by my laptop's capabilities. I set up the cluster using Multipass, and there wasn't a way to configure that.

from longhorn.

Ref PR: longhorn/longhorn-engine#1104

from longhorn.

Verified on v1.6.x-head 20240508

• longhorn v1.6.x-head 46706be
• longhorn-engine v1.6.x-head longhorn/longhorn-engine@4f53092

The test steps
#8436 (comment)

Result Passed

1. In the AWS EC2 t2.xlarge environment, I did not observe significant differences in IOPS.
   Therefore, I have decided to build up a cluster in a local virtual machine to conduct this test.
   The test result is as follows.

v1.6.1

=========================
FIO Benchmark Summary
For: test_device
CPU Idleness Profiling: disabled
Size: 30G
Quick Mode: disabled
=========================
IOPS (Read/Write)
        Random:           10,329 / 4,684
    Sequential:          17,677 / 10,204

Bandwidth in KiB/sec (Read/Write)
        Random:        413,252 / 141,958
    Sequential:        461,723 / 170,358
                                        

Latency in ns (Read/Write)
        Random:        431,773 / 553,568
    Sequential:        357,682 / 547,949

v1.6.x-head

=========================
FIO Benchmark Summary
For: test_device
CPU Idleness Profiling: disabled
Size: 30G
Quick Mode: disabled
=========================
IOPS (Read/Write)
        Random:           14,329 / 6,573
    Sequential:          22,396 / 10,423

Bandwidth in KiB/sec (Read/Write)
        Random:        513,981 / 153,945
    Sequential:        536,158 / 169,366
                                        

Latency in ns (Read/Write)
        Random:        392,363 / 527,421
    Sequential:        343,545 / 515,456

Here is the summary table

Hi @PhanLe1010
I think we can close this ticket. If there are any concerns regarding the test results, please let me know.

from longhorn.

Hi @roger-ryao

Thank you for testing!

In the AWS EC2 t2.xlarge environment, I did not observe significant differences in IOPS.

t2.xlarge is using EBS with burst performance to 3,000 IOPS. This is too slow and will not expose the improvement. Please use the ec2 with instance store (attached nvme) like c5d.2xlarge

Therefore, I have decided to build up a cluster in a local virtual machine to conduct this test.

Testing inside virtual machine might be ok too. But looks like the seq write inside virtual machine is capped at 10k somehow. Is it the max that VM can achieve natively?

from longhorn.