breuner / elbencho Goto Github PK

A distributed storage benchmark for file systems, object stores & block devices with support for GPUs

License: GNU General Public License v3.0

Makefile 2.53% Shell 10.87% C++ 86.32% C 0.28%

benchmark storage nvme deep-learning live-stats distributed parallel file-systems block-storage ior mdtest fio gpu s3 linux windows hdfs

elbencho's Introduction

elbencho

A distributed storage benchmark for files, objects & blocks with support for GPUs

elbencho was inspired by traditional storage benchmark tools like fio, mdtest and ior, but was written from scratch to replace them with a modern and easy to use unified tool for file systems, object stores & block devices.

(click to expand)

Features
Usage
Build Prerequisites
- Dependencies for Debian/Ubuntu
- Dependencies for RHEL/CentOS
Build & Install
Now what?
- Results & Charts
- Questions & Comments
Optional Build Features

Features

Unified latency, throughput, IOPS benchmark for file, object & block storage
Supports local and shared storage through distributed service mode
For modern NVMe storage or classic spinning disk storage
GPU storage access performance testing through Nvidia CUDA or GPUDirect Storage (GDS)
Live statistics show how the system behaves under load
Multi-threaded and async I/O support through libaio
Results by first and by last finished thread
CSV file output to easily create graphs in spreadsheet apps or via elbencho-chart tool
Data integrity verification option

Usage

The built-in help (elbencho --help) provides simple examples to get started.

You can get elbencho pre-built for Linux & Windows from the Releases section and from Docker Hub.

Build Prerequisites

Building elbencho requires a C++17 compatible compiler, such as gcc version 7.x or higher.

Dependencies for Debian/Ubuntu

sudo apt install build-essential debhelper devscripts fakeroot git libaio-dev libboost-filesystem-dev libboost-program-options-dev libboost-thread-dev libncurses-dev libnuma-dev lintian libssl-dev

Dependencies for RHEL/CentOS

sudo yum install boost-devel gcc-c++ git libaio-devel make ncurses-devel numactl-devel openssl-devel rpm-build

On RHEL / CentOS 7.x: Prepare Environment with newer gcc Version

Skip these steps on RHEL / CentOS 8.0 or newer.

sudo yum install centos-release-scl # for CentOS
# ...or alternatively for RHEL: yum-config-manager --enable rhel-server-rhscl-7-rpms
sudo yum install devtoolset-8
scl enable devtoolset-8 bash # alternatively: source /opt/rh/devtoolset-8/enable

The scl enable command enters a shell in which the environment variables are pointing to a newer gcc version. (The standard gcc version of the system remains unchanged.) Use this shell to run make later. The resulting executable can run outside of this shell.

Build & Install

Start by cloning the main repository:

git clone https://github.com/breuner/elbencho.git
cd elbencho

make help will show you all build & install options.

(Note that S3 support is not enabled by default due to longer build time, but can easily be enabled. See the additional build info below.)

This is the standard build command:

make -j $(nproc)

You can run elbencho directly from the bin subdir (bin/elbencho), but you probably want to run make rpm or make deb now to build a package and install it. On Ubuntu, run this:

make deb
sudo apt install ./packaging/elbencho*.deb

There you go. Happy benchmarking!

Now what?

Now comes the fun part: It's time to find out what your system can deliver.

The built-in help (elbencho --help) provides many usage examples. You will be interested in throughput and IOPS, typically for a single client and also for multiple clients. For the latter, see --hosts & --service.

If your cluster is using Slurm to allocate nodes, you can find examples here.

If GPU data transfer performance is critical for you, e.g. because you are running DeepLearning applications, you will also want to include GPUs in your read/write benchmarks (--gpuids).

Instead of giving benchmark parameters on the command line, you can also use a config file (--configfile myconfig.conf). You can find an example here.

Results & Charts

Elbencho presents two result columns: One showing the aggregate status at the point in time when the fastest I/O thread finished its work ("First Done") and one for the aggregate end result at the point in time when the slowest thread finished its work ("Last Done"). Ideally, both are close together.

To generate charts from your result series, e.g. based on different block sizes or different thread counts, use elbencho's csv file option (--csvfile) and the elbencho-chart tool to easily generate a chart from your csv file. Alternatively, spreadsheet tools like Microsoft Excel or Google Sheets can be used for this.

Questions & Comments

In case of questions, comments, if something is missing to make elbencho more useful or if you would just like to share your thoughts, feel free to contact me: sven.breuner[at]gmail.com

Optional Build Features

elbencho --version shows which optional features are included in an executable.

Nvidia CUDA Support

CUDA support for GPU data transfers will automatically be enabled when CUDA development files (cuda_runtime.h and libcudart.so) are installed on the build system. Alternatively, elbencho CUDA support can be manually enabled or disabled. See make help for details.

Nvidia GPUDirect Storage (GDS) Support

GPUDirect Storage (GDS) support through the cuFile API will automatically be enabled when GDS development files (cufile.h and libcufile.so) are installed on the build system. Alternatively, elbencho GDS support can be manually enabled or disabled. See make help for details.

S3 Object Storage Support

Enabling S3 Object Storage support will automatically download a AWS SDK git repository of over 1GB size and increases build time from a few seconds to a few minutes. Thus, S3 support is not enabled by default, but it can easily be enabled as described below.

S3 Dependencies for RHEL/CentOS 8.0 or newer

sudo yum install cmake libarchive libcurl-devel libuuid-devel zlib zlib-devel

S3 Dependencies for Ubuntu 20.04 or newer

sudo apt install cmake libcurl4-openssl-dev uuid-dev zlib1g-dev

Build elbencho with S3 Support

The static Linux executable in the Releases section includes S3 support, in case you prefer to use this instead of building your own version.

To build elbencho with S3 support, just add the S3_SUPPORT=1 parameter to the make command. (If you previously built elbencho without S3 support, then run make clean-all before this.)

The S3 support of elbencho is based on Amazon's AWS SDK CPP. Thus, you either need to provide it or elbencho needs to download and build it.

Option A: If you prefer to provide your own version of the AWS SDK CPP instead of having elbencho download it, then here is an example of a cmake command to generate an SDK that is compatible with elbencho and the corresponding elbencho build command:

# In the AWS SDK build dir:
cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/usr/local/ -DCMAKE_INSTALL_PREFIX=/usr/local/ -DBUILD_ONLY="s3;transfer" -DAUTORUN_UNIT_TESTS=OFF -DENABLE_TESTING=OFF -DBYO_CRYPTO=ON -DBUILD_SHARED_LIBS=OFF

make -j $(nproc)
sudo make install

# In the elbencho git clone top level dir:
make -j $(nproc) S3_SUPPORT=1 AWS_INCLUDE_DIR=/usr/local/include/ AWS_LIB_DIR=/usr/local/lib64/

Option B: If you are not sure whether you can provide a compatible build of the SDK or if you just generally prefer to have elbencho take care of the AWS SDK CPP build then use this command:

make S3_SUPPORT=1 -j $(nproc)

elbencho's People

Contributors

Stargazers

Watchers

elbencho's Issues

Continue on when encountering s3 errors

We're using elbencho to stress/load-test local s3 server. Elbencho quits as soon as it encounters a single error.

17:21:37.689 ERROR: Object download failed. Endpoint: xxxxx; Bucket: elbencho-job; Object: biggie/file40; Range: bytes=56699904-56700927; Exception: ; Message: No response body.; HTTP Error Code: 502

It would be great if there was an option to continue on with the test and just log/ignore the number of errors encountered

Building on RHEL 8?

The instructions for RHEL 8 don't work. (tried RHEL 8.4 and 8.5)

These packages don't exist: boost-devel libaio-devel numactl-devel

I just spent an hour trying to get boost/boost-devel installed and it is a rat's nest of package dependencies and mess.

Or is their a linux version that this will install cleanly on? Or can you disable boost/numa. We don't need those features. We only need S3.

Update: Also tried building this on ubuntu 22.04:

<SNIP>
In file included from /elbencho/external/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/crypto/openssl/CryptoImpl.h:14,
                 from /elbencho/external/aws-sdk-cpp/aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp:9,
                 from /elbencho/external/aws-sdk-cpp/aws-cpp-sdk-core/ub_core.cpp:26:
/usr/include/openssl/hmac.h:34:27: note: declared here
   34 | OSSL_DEPRECATEDIN_3_0 int HMAC_CTX_reset(HMAC_CTX *ctx);
      |                           ^~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make[3]: *** [aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/build.make:917: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/ub_core.cpp.o] Error 1
make[3]: Leaving directory '/elbencho/external/aws-sdk-cpp'
make[2]: *** [CMakeFiles/Makefile2:758: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/all] Error 2
make[2]: Leaving directory '/elbencho/external/aws-sdk-cpp'
make[1]: *** [Makefile:136: all] Error 2
make[1]: Leaving directory '/elbencho/external/aws-sdk-cpp'
make: *** [Makefile:205: externals] Error 1

Enhancement: Create large directory hash

Greetings, new to your tool, your friends at Vast and I have a long history at Yahoo, and now I'm at Intel.

What if I want to stress out a more complex MD testing strategy like I can with a commercial tool, like Virtana Workload Wisdom?

Where I can define
Number of dirs
Number of subdirs in each dir
Number depth of tree
Number of files per directory

And perform this across each of any # of exported POSIX filesystems available to a cluster of compute hosts...and/or define a # of threads to traverse the hashed tree randomly or ordered (north to south, west to east, etc)

I need to work elbencho against storage for AI workloads primarily, but there is a need for the same vendor solution(s) for general storage, including HFC HPC and EDA workloads.

I look forward to your thoughts.

-JeffM. (Tell AP and JD I said hello!)

Add millisecond resolution in reporting timestamps

The current timestamps used in the result statistics, such as the CSV file, only contain second-level resolution, eg:

2024-04-16T23:28:59+0000

If two operations (like SYNC and DROPCACHE) take <1 second they can show up as the same timestamp, thereby relying only on the order in the CSV file as to the order they occurred. Without millisecond resolution it's challenging to correlate elbencho results with a testing framework's logs when the framework uses millisecond resolution but elbencho does not -- does elbencho's results with timestamp 2024-04-16T23:28:59+0000 come before or after 2024-04-16T23:28:59.143+0000?

As ISO 8601 includes millisecond support this wouldn't fundamentally alter the nature of the value, just improve its resoultion.

Value in latency histogram is larger than latency max value, wrong computation or wrong interpretation?

For some latency values, the max value is lower than the maximum value in the histogram:

File latency = 5.30 ms
max value in Files lat hist = 5.792 ms

The same is valid for the IO latency values:
max IO latency = 196 us
max value in IO lat hist = 216 us

Do I misinterpret the values, or is there something wrong with the computation of the values?

      Files latency    : [ min=405us avg=2.10ms max=5.30ms ]
      Files lat % us   : [ 1%<=430 50%<=862 75%<=4096 99%<=5792 ]
      Files lat hist   : [ 430: 1, 512: 1, 608: 1, 724: 1, 862: 1, 1024: 1, 4096: 2, 4870: 1, 5792: 1 ]
      IO latency       : [ min=6us avg=24us max=196us ]
      IO lat % us      : [ 1%<=6.8 50%<=12 75%<=22 99%<=216 ]
      IO lat hist      : [ 6.8: 9, 8.0: 21, 9.6: 14, 12: 18, 14: 5, 16: 5, 20: 2, 22: 3, 26: 6, 46: 1, 64: 2, 76: 6, 90: 1, 108: 4, 128: 1, 216: 2 ]

Assuming that the history values are really measured values, this would mean that some or all max values are wrong and that design decisions based on the max value would not be correct, since latency based long-distance architectures could be statistically significantly wrong, even if measured with a very high number of measurements. In the IO values (with a very small lot), the measurement error is ~ 10%

csvfile - In progress results over time, similar to FIO?

Greetings.

In testing elbencho and using the csvfile option I get a start and end summary that can be graphed. I am looking for graphing the performance over time, identifying flushes, page cache, storage device issues, etc.

Is there a way to execute elbencho where results are recorded to csvfile at an interval during the benchmark, say every 500ms to 1s?

Thanks

crash with free(): invalid next size (normal): 0x00002aaad8002310

I seem to be reproducibly crashing elbencho and getting the following error:

WRITE: 0 files/s; 65 MiB/s; 0 files; 2978 MiB; 16 threads; 1% CPU; 46s*** Error in `/global/u2/g/glock/src/elbencho/bin.cgpu/elbencho': free(): invalid next size (normal): 0x00002aaad8002310 ***
FAULT HANDLER (PID 66952 / TID 66986): Abnormal termination
terminate called after throwing an instance of 'ProgException'
  what():  Backtrace:
1: /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho(_ZN8SignalTk12logBacktraceB5cxx11Ev+0x169) [0x4cb579]
2: /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho(_ZN8SignalTk18faultSignalHandlerEi+0xa8) [0x4cc158]
3: /lib64/libc.so.6(+0x361e0) [0x2aaaac8f71e0]
4: /lib64/libc.so.6(gsignal+0x110) [0x2aaaac8f7160]
5: /lib64/libc.so.6(abort+0x151) [0x2aaaac8f8741]
6: /lib64/libc.so.6(+0x794a7) [0x2aaaac93a4a7]
7: /lib64/libc.so.6(+0x7fcc3) [0x2aaaac940cc3]
8: /lib64/libc.so.6(+0x815c9) [0x2aaaac9425c9]
9: /usr/lib64/libstdc++.so.6(_ZNSt13basic_filebufIcSt11char_traitsIcEE26_M_destroy_internal_bufferEv+0x1e) [0x2aaaabebc64e]
10: /usr/lib64/libstdc++.so.6(_ZNSt13basic_filebufIcSt11char_traitsIcEE5closeEv+0x4b) [0x2aaaabebf9eb]
11: /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho(_ZN7CPUUtil6updateEv+0x262) [0x525ff2]
12: /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho(_ZN17WorkersSharedData25incNumWorkersDoneUnlockedEv+0x5c) [0x4e17cc]
13: /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho(_ZN6Worker17incNumWorkersDoneEv+0x4b) [0x4f478b]
14: /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho(_ZN11LocalWorker3runEv+0x12e0) [0x4f26e0]
15: /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho(_ZN6Worker11threadStartEPS_+0x12) [0x4f43f2]
16: /usr/lib64/libstdc++.so.6(+0xc338f) [0x2aaaabe8c38f]
17: /lib64/libpthread.so.0(+0x7569) [0x2aaaac6aa569]
18: /lib64/libc.so.6(clone+0x3f) [0x2aaaac9b9a2f]

Error: HTTP client error on benchmark interruption: Connection refused. Service: cgpu08:1611
HTTP client error in benchmark phase: Connection refused. Service: cgpu08:1611
srun: error: cgpu08: task 0: Aborted
srun: launch/slurm: _step_signal: Terminating StepId=2794884.0

This appears to happen once my job reaches the --timelimit under specific circumstances. I run my job using the following:

srun -N 1 -n 1 --nodelist cgpu08 /global/u2/g/glock/src/elbencho/bin.cgpu/elbencho --service --foreground &

/global/u2/g/glock/src/elbencho/bin.cgpu/elbencho --hosts cgpu08 --timelimit 45 --blockvaralgo fast --blockvarpct 25 --resfile xyz --csvfile xyz.out.csv --threads 16 --files 1 --dirsharing --mkdirs --sync --direct --write --size 1t --block 4k --rand /vast/glock

This is using elbencho from commit c997820 using gcc (SUSE Linux) 7.5.0 and boost-1.74.0.

I think I've been sporadically hitting the same error under different levels of concurrency but for some reason the stack trace doesn't make it out to the stderr of my batch job and I had to run elbencho interactively to get the above error. For example, the following configuration failed with a similar "HTTP client error on benchmark interruption" error:

/global/u2/g/glock/src/elbencho/bin.cgpu/elbencho --hosts cgpu05,cgpu06,cgpu07,cgpu12,cgpu13,cgpu14 --timelimit 45 --resfile secondary_quiet.2p-6s.2780749.out.results --csvfile secondary_quiet.2p-6s.2780749.out.csv --livecsv secondary_quiet.2p-6s.2780749.out.live.csv --threads 16 --files 1 --dirsharing --mkdirs --sync --direct --write --size 1t --block 4k --rand /vast/glock

Can not run configuration file

When I run elbencho --configfile tools/example_configuration/randomWrite.elbencho /data, there is an error:
ERROR: Cannot open config file: tools/example_configuration/randomWrite.elbencho
How can I deal with this?

Output fields list and descriptions

Is there a document somewhere that explains and/or describes the output fields, either live or output files?

For example, what does "Entries" correlate to?
is Files total, just a total number of files that were accessed during the test?
what about "Files latency" vs IO Latency, what is the correlation between the two?

A document with the field descriptions would be very helpful :)

Question about --timelimit

Hi,

If I use in one execution -w and -r and the --timelimit, then the timelimit duration is for the total execution and not for each case (write and read). Could be considered such option if it does not exist already?

regards,
George

Can we Test Sequential Reads and Writes using elbencho

Is there any param for sequential IO like we have for Random --rand ?

How do you run this without it wanting to create the bucket?

I'm trying to benchmark various S3 object stores but my buckets are pre-existing. elbencho seems to want to create the bucket:

+ elbencho --s3endpoints https://s3.example.com/ --s3key KEY --s3secret SECRET -d --threads 1 --size 1048576 --files 1 --write --lat --latpercent --csvfile ./logs/fillbucket/fillbucket ise-engineering
OPERATION RESULT TYPE        FIRST DONE  LAST DONE
========= ================   ==========  =========
ERROR: Bucket creation failed. Endpoint: https://s3.example.com/; Bucket: ise-engineering; Exception: BucketAlreadyExists; Message: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Thanks

Running with iodepth reduces the performance by 10x or more

Running elbencho with iodepth=1 on our stack gives good performance as expected. Running iodepth=2 or more (in aio mode) reduces the read/write performance by 10x or more. We ran fio tool with iodepth and got almost the same performance as without aio.
Wondering if this is a known issue with elbencho? Or should we run with any additional flags to raise the performance.

Question about --size

Hi,
Thank you for sharing this useful tool. Recently, during my attempt to use it instead of fio for SSD performance analysis, I found that "elbencho -- size" cannot be set with a percentage like "fio -- size=100% ", or reporting "Unable to parse string for unit conversion: 100%". It seems that elbencho cannot recognize the capacity of an nvme device, or maybe I'm not familiar with this tool yet.

Issue generating charts using "ISO date" for x-axis

I'm running a benchmark every 15 minutes using the exact same parameters and writing all results to the same file. Once a day I want to chart the previous day's data, plotting READ/WRITE throughput vs. time of day.

Here is a sample of my datafile:

2022-04-10T23:49:48+0000,,bucket,1,1,1,1,1,10737418240,134217728,0,0,,1,,0,WRITE,286522,286522,0,0,0,0,35,35,4,4,1,1,10240,10240,286516564,286516564,286516564,1986196,3558061,7762379,,,,,,,,,,,,,,,,,2.1-0,"elbencho" "--s3endpoints" "https://s3.example.com/" "--s3key" "REDACTED" "--s3secret" "REDACTED" "--delfiles" "--threads" "1" "--dirs" "1" "--files" "1" "--size" "10737418240" "--block" "134217728" "--write" "--read" "--lat" "--cpu" "--allelapsed" "--s3objprefix" "crontest.6385" "--s3nompcheck" "--csvfile" "20220410.csv" "storage-eng-dev"
2022-04-10T23:52:52+0000,,bucket,1,1,1,1,1,10737418240,134217728,0,0,,1,,0,READ,183317,183317,0,0,0,0,55,55,5,5,1,1,10240,10240,183311998,183311998,183311998,809746,2291385,5135635,,,,,,,,,,,,,,,,,2.1-0,"elbencho" "--s3endpoints" "https://s3.example.com/" "--s3key" "REDACTED" "--s3secret" "REDACTED" "--delfiles" "--threads" "1" "--dirs" "1" "--files" "1" "--size" "10737418240" "--block" "134217728" "--write" "--read" "--lat" "--cpu" "--allelapsed" "--s3objprefix" "crontest.6385" "--s3nompcheck" "--csvfile" "20220410.csv" "storage-eng-dev"
2022-04-10T23:52:52+0000,,bucket,1,1,1,1,1,10737418240,134217728,0,0,,1,,0,RMFILES,233,233,4,4,,,,,3,3,1,1,,,227680,227680,227680,,,,,,,,,,,,,,,,,,,,2.1-0,"elbencho" "--s3endpoints" "https://s3.example.com/" "--s3key" "REDACTED" "--s3secret" "REDACTED" "--delfiles" "--threads" "1" "--dirs" "1" "--files" "1" "--size" "10737418240" "--block" "134217728" "--write" "--read" "--lat" "--cpu" "--allelapsed" "--s3objprefix" "crontest.6385" "--s3nompcheck" "--csvfile" "20220410.csv" "storage-eng-dev"

And the error I'm getting:

charting data from 20220410.csv

gnuplot> set xtics ('2022-04-10T00:04:32+0000' 0, '2022-04-10T00:22:49+0000' 1, '2022-04-10T00:36:43+0000' 2, '2022-04-10T00:49:24+0000' 3, '2022-04-10T01:06:46+0000' 4, '2022-04-10T01:22:28+0000' 5, '2022-04-10T01:36:14+0000' 6, '2022-04-10T01:49:01+0000' 7, '2022-04-10T02:06:34+0000' 8, '2022-04-10T02:21:59+0000' 9, '2022-04-10T02:33:47+0000' 10, '2022-04-10T02:50:05+0000' 11, '2022-04-10T03:03:55+0000' 12, '2022-04-10T03:23:05+0000' 13, '2022-04-10T03:33:41+0000' 14, '2022-04-10T03:50:01+0000' 15, '2022-04-10T04:04:28+0000' 16, '2022-04-10T04:23:32+0000' 17, '2022-04-10T04:34:20+0000' 18, '2022-04-10T04:49:54+0000' 19, '2022-04-10T05:04:52+0000' 20, '2022-04-10T05:24:26+0000' 21, '2022-04-10T05:36:23+0000' 22, '2022-04-10T05:51:14+0000' 23, '2022-04-10T06:04:30+0000' 24, '2022-04-10T06:22:24+0000' 25, '2022-04-10T06:35:08+0000' 26, '2022-04-10T06:51:42+0000' 27, '2022-04-10T07:05:46+0000' 28, '2022-04-10T07:22:30+0000' 29, '2022-04-10T07:33:49+0000' 30, '2022-04-10T07:49:09+0000' 31, '2022-04-10T08:03:46+0000' 32, '2022-04-10T08:24:30+0000' 33, '2022-04-10T08:35:21+0000' 34, '2022-04-10T08:50:00+0000' 35, '2022-04-10T09:04:21+0000' 36, '2022-04-10T09:24:25+0000' 37, '2022-04-10T09:34:25+0000' 38, '2022-04-10T09:49:30+0000' 39, '2022-04-10T10:05:27+0000' 40, '2022-04-10T10:24:43+0000' 41, '2022-04-10T10:35:19+0000' 42, '2022-04-10T10:50:23+0000' 43, '2022-04-10T11:04:44+0000' 44, '2022-04-10T11:24:12+0000' 45, '2022-04-10T11:36:44+0000' 46, '2022-04-10T11:51:44+0000' 47, '2022-04-10T12:05:18+0000' 48, '2022-04-10T12:22:24+0000' 49, '2022-04-10T12:35:33+0000' 50, '2022-04-10T12:50:38+0000' 51, '2022-04-10T13:06:37+0000' 52, '2022-04-10T13:23:13+0000' 53, '2022-04-10T13:34:07+0000' 54, '2022-04-10T13:49:12+0000' 55, '2022-04-10T14:04:29+0000' 56, '2022-04-10T14:21:56+0000' 57, '2022-04-10T14:34:48+0000' 58, '2022-04-10T14:49:54+0000' 59, '2022-04-10T15:05:11+0000' 60, '2022-04-10T15:24:16+0000' 61, '2022-04-10T15:34:04+0000' 62, '2022-04-10T15:51:13+0000' 63, '2022-04-10T16:04:26+0000' 64, '2022-04-10T16:24:37+0000' 65, '2022-04-10T16:36:17+0000' 66, '2022-04-10T16:51:41+0000' 67, '2022-04-10T17:04:45+0000' 68, '2022-04-10T17:24:05+0000' 69, '2022-04-10T17:34:17+0000' 70, '2022-04-10T17:51:34+0000' 71, '2022-04-10T18:05:50+0000' 72, '2022-04-10T18:22:26+0000' 73, '2022-04-10T18:36:46+0000' 74, '2022-04-10T18:49:38+0000' 75, '2022-04-10T19:04:52+0000' 76, '2022-04-10T19:22:35+0000' 77, '2022-04-10T19:34:41+0000' 78, '2022-04-10T19:51:18+0000' 79, '2022-04-10T20:06:39+0000' 80, '2022-04-10T20:22:54+0000' 81, '2022-04-10T20:33:57+0000' 82, '2022-04-10T20:48:33+0000' 83, '2022-04-10T21:04:16+0000' 84, '2022-04-10T21:23:24+0000' 85, '2022-04-10T21:34:27+0000' 86, '2022-04-10T21:51:21+0000' 87, '2022-04-10T22:05:20+0000' 88, '2022-04-10T22:22:36+0000' 89, '2022-04-10T22:34:14+0000' 90, '2022-04-10T22:49:31+0000' 91, '2022-04-10T23:04:46+0000' 92, '2022-04-10T23:22:09+0000' 93, '2022-04-10T23:34:29+0000' 94, '2022-04-10T23:49:48+0000' 95, ) rotate by 90 right
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ^
         line 0: invalid expression

It looks like some kind of off-by-one type error where it expects another value or something.

Am I doing something wrong here or is this a bug?

Thanks,
Chuck

Build failure on Ubuntu 22.04

I am trying to build elbencho (version 72261d8) on Ubuntu 22.04 and followed the README's instructions on installing dependencies. When I build though, I hit this error:

glock@glocknode:~/elbencho$ make
[OPT] CUFILE (GDS) support disabled
[OPT] CUDA support disabled
[OPT] S3 support disabled. (Enable via S3_SUPPORT=1)
[OPT] mimalloc disabled
[OPT] Alternative HTTP service disabled
[OPT] HDFS support disabled
[DEP] source/Statistics.d
[CXX] source/Statistics.o
In file included from source/workers/Worker.h:5,
                 from source/workers/WorkerManager.h:4,
                 from source/Statistics.h:7,
                 from source/Statistics.cpp:8:
./source/LatencyHistogram.h:40:22: error: ‘atomic_uint64_t’ in namespace ‘std’ does not name a type
   40 |                 std::atomic_uint64_t numStoredValuesLive{0}; // for live stats
      |                      ^~~~~~~~~~~~~~~
./source/LatencyHistogram.h:41:22: error: ‘atomic_uint64_t’ in namespace ‘std’ does not name a type
   41 |                 std::atomic_uint64_t numMicroSecsTotalLive{0}; // for live stats
      |                      ^~~~~~~~~~~~~~~
./source/LatencyHistogram.h: In member function ‘void LatencyHistogram::addLatency(uint64_t)’:
./source/LatencyHistogram.h:48:25: error: ‘numStoredValuesLive’ was not declared in this scope; did you mean ‘numStoredValues’?
   48 |                         numStoredValuesLive++;
      |                         ^~~~~~~~~~~~~~~~~~~
      |                         numStoredValues
./source/LatencyHistogram.h:49:25: error: ‘numMicroSecsTotalLive’ was not declared in this scope; did you mean ‘numMicroSecTotal’?
   49 |                         numMicroSecsTotalLive += latencyMicroSec;
      |                         ^~~~~~~~~~~~~~~~~~~~~
      |                         numMicroSecTotal
./source/LatencyHistogram.h: In member function ‘void LatencyHistogram::addAndResetAverageLiveMicroSec(uint64_t&, uint64_t&)’:
./source/LatencyHistogram.h:81:47: error: ‘numStoredValuesLive’ was not declared in this scope; did you mean ‘numStoredValues’?
   81 |                         outNumStoredValues += numStoredValuesLive;
      |                                               ^~~~~~~~~~~~~~~~~~~
      |                                               numStoredValues
./source/LatencyHistogram.h:82:49: error: ‘numMicroSecsTotalLive’ was not declared in this scope; did you mean ‘numMicroSecTotal’?
   82 |                         outNumMicroSecsTotal += numMicroSecsTotalLive;
      |                                                 ^~~~~~~~~~~~~~~~~~~~~
      |                                                 numMicroSecTotal
make: *** [Makefile:219: source/Statistics.o] Error 1

Is there something new about compiler strictness or default C++ standard that changed in 22.04 that may have broken something here?

Questions on single file read and write with multiple GPUs

Dear developers,

Thank you for the tool. I have been enjoying using it.
I have the following questions regarding the arguments relates to single file read and write.

Will affinity of CPU and GPU gaurranteed when using multiple CPU workers and multiple GPUs? For example, --direct -t 96 -gpuids 0,1,2,3,4,5,6,7? In --help-all, it says "GPU IDs will be assigned round robin to different threads".
Will read from a file into multiple GPUs duplicating all the files in all GPUs? For example, elbencho -r -b 1m -s 1g -t 32 --gpuids 0,1,2,3 --gdsbufreg --direct --cufile /some/gdsremotefile. Will all 4 GPUs have the same replica of the 1GiB file? I wonder how this is done under the hood. Does all communication done via pcie or each GPU has a different shard (via pcie) and shared the missing shard via nvlink?
If we are writing a 1GiB file using multiple GPUs, will the program first create buffers of "1GiB/num-gpus" or "1GiB" on each GPU? For exampe, elbencho -w -b -1m -s 1g -t 32 --gpuids 0,1,2,3 --direct --cufile /some/gdsremotefile. How the communication happens under the hood? Does each GPU contributes a piece of the entire file?

Relating to previous item, if I do the following: elbencho -w -b 1m -t 32 --gpuids 0,1,2,3,4,5,6,7 --direct --size 1g /some/gdsremotefile I receive the following output:

OPERATION RESULT TYPE        FIRST DONE  LAST DONE
========= ================   ==========  =========
WRITE     Elapsed ms       :        397        533
          IOPS             :       1896       1920
          Throughput MiB/s :       1896       1920
          Total MiB        :        754       1024

I wonder why the FIRST DONE only has 754 MiB data transferred?

Thank you so much! I'm looking forward to your reply.

Add more context to data verification errors

Currently, the errors lack key information that helps understanding the data verification errors:

19:49:29.140 ERROR: Data verification failed. Offset: 6979585; Expected value: 18; Actual value: 146

I'd like to see some more context the file/object name.

Option to convert milliseconds to seconds, minutes or hours?

Hi Sven! Nice chatting with you in Slack a few weeks ago. Love the tool here, I have added it to my toolkit for certain.

I was curious if there would be any way to add a switch or option where when results are printed at the end if there could be a way to have elbencho show time in seconds or minutes (or maybe even hours). Currently, once the results are in a .csv I have to go in and divide the milliseconds by 1000 for seconds or 60000 for minutes and it is just an extra step.

Not the end of the world or anything, but I thought I'd ask. Thanks!

Aggregate Throughput/IOPs results?

Hello there,

I am trying to build a benchmarking solution using elbencho for distributed storage performance testing, but the output from elbencho has me a little confused.

The tool gives: MiB/s [first] | MiB/s [last]

I gather that this refers to the first and last threads used during the test (8 threads total in this test), but how does one calculate aggregate throughput when using the tool in distributed mode (meaning multiple instances of elbencho across multiple hardware hosts in worker mode and one in manager mode all working against a large distributed storage system)?

I feel like I'm missing something.

Thank you in advance for your help!

Protect distributed mode (--service) with a secret

Hello,

I wonder if it would be possible to protect a distributed run (--service) to be hijacked by providing a secret, a path to a file containing a secret or an accessible sentinel pathname from all hosts involved in the bench (that would leverage whatever filesystem permission is in place). Maybe I've missed something already implemented ?

Right now, I'm quite nervous during the time window after I ran elbencho --service on a bunch of hosts ;)

Thank you for this great tool.

Dockerfile for nvidia gpus on arm64 based hardware.

I am looking for a arm64 based elbencho with gpu (nvidia/cuda) support.

Would one of these containers work?
https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nvhpc/tags

Unexpected short file read

Getting errors trying to do a read test:
Is there an error in the command?
elbencho version 2.1-6, RHEL 8.2, Running on 2 nodes with shared filesystem.

[root@w1 a]# elbencho --hosts w1,w2 --threads 32 --size 4G --block 128K --write --nolive --mkdir --rand --direct --iodepth 16 /mnt/scratch/data/
OPERATION RESULT TYPE FIRST DONE LAST DONE
========= ================ ========== =========
MKDIRS Elapsed ms : 2 11
Dirs/s : 10925 ### 5553
Dirs total : 32 64

WRITE Elapsed ms : 1210 7353
Files/s : 0 8
IOPS : 404015 285176
Throughput MiB/s : 50501 35647
Total MiB : 61153 262144
Files total : 1 64

[root@w1 a]#

[root@w1 a]# elbencho --hosts w1,w2 --threads 32 --size 4G --block 128K --read --nolive --mkdir --rand --direct --iodepth 16 /mnt/scratch/data/
OPERATION RESULT TYPE FIRST DONE LAST DONE
========= ================ ========== =========
MKDIRS Elapsed ms : 2 5
Dirs/s : 12825 12158
Dirs total : 32 64

=== [ HOST: w1:1611 (Rank: 0) ] ===
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r26/d0/r26-f0; Bytes read: 44957696; Expected read: 4294967296
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r16/d0/r16-f0; Bytes read: 43253760; Expected read: 4294967296
=== [ END ] ===
=== [ HOST: w2:1611 (Rank: 1) ] ===
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r41/d0/r41-f0; Bytes read: 68550656; Expected read: 4294967296
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r59/d0/r59-f0; Bytes read: 77594624; Expected read: 4294967296
=== [ END ] ===
[root@w1 a]#

2nd question. How to create a dataset and then every read will use the same dataset?

Lot of idle threads when running READ/WRITE on Lustre FS

Hi,
We are trying to run elbencho command on LustreFS with 64 threads and the filesize and the blocksize being 1MB. We are seeing lot of threads idle even though the system has 256 cores and the system CPU usage shows 100% idle. I am wondering if there are tunable parameters in the elbencho to make all the threads active at a given time.

Report metadata operation time?

Is there a way to have elbencho report the time it takes for all processes to open and/or close all files? IOR does this when there's a barrier between open/write/read/close phases of a test, but I couldn't find anything in the elbencho code that suggests these timings can be reported.

There two reasons this might be useful:

Some cloud caching file systems can read the entire file contents and cache it on a local disk as a part of open(2). So all the I/O happens during the open call, and the read performance reported by elbencho winds up being the cache performance, not the end-to-end performance.
When doing a ton of buffered writes, a lot of write I/O winds up happening in the close(2) call, so splitting out this time can give an indication of writeback performance.

Delete directories not working

I'm evaluating this package and I still very new to it.

But it looks like I found a bug around the cleanup after bench is done.
Here is a screenshot of the script and the output.

Thank you for this package and all the efforts.

support for 0-byte file creation rates?

I am trying to use elbencho to test the file create rate of a file system and hit two problems. First, when I just try to make as many files as I can within a timelimit,

$ ~/src/elbencho/bin/elbencho --timelimit 60 --threads 1 --size 0 --block 0 --write --delfiles outputfile.{0..999999}
-bash: /global/homes/g/glock/src/elbencho/bin/elbencho: Argument list too long

I know why this is happening, but I am wondering if there is another way to create a bunch of files without expressing each one on the command line.

When I reduce the number of files, I get a different error:

$ ~/src/elbencho/bin/elbencho --timelimit 60 --threads 1 --size 0 --block 0 --write --delfiles tmp.{0..100000}
ERROR: File size must not be 0 when benchmark path is a file. File: tmp.0

Is the creation of empty files unsupported? It's a common benchmark requirement for file systems (though not a very sensible one...), and I am trying to find the worst-case performance for when users scp a directory full of tiny (empty) files to different file systems (again, not a sensible thing to do, but our users love doing it).

support for S3 byte-range reads

Is there any support for 'random read' byte-range reads on s3 objects?

Thanks,
Mahmoud