gluster / glusterfs-perf Goto Github PK

as we are starting processes newly for the runs, we need to capture the gluster profile information after each run. Helps to debug later.

Once the tests are all added and different volume types, it's good to run the benchmark for the last 3-4 major releases and plot the same. Make this available in Gluster readthe docs or release notes.

right now, we are taking just 1 run as the result, as it is performance runs, we need to take the avg of each runs.

This is just a question:

We may end up with less or more 'inventory' over time, so, how are we deciding how many clients are there and how many servers are there?

Isn't it better to define some tests, and clearly identify minimum inventory required for the test? If not satisfied, the test shouldn't even run.

For each perf run, along with iops, latency etc. it's good to collect the following if possible:

• profile info
• network bandwidth usage
• disk usage
• CPU consumption of Gluster processes
• memory consumption of Gluster processes

While trying to run a test with v5.0, hit this error. Looks like we had previous glusterd workdir.

[2019-02-21 05:01:16.847028] W [rpcsvc.c:1789:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2019-02-21 05:01:16.847044] E [MSGID: 106244] [glusterd.c:1798:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2019-02-21 05:01:18.059357] E [MSGID: 106022] [glusterd-store.c:2276:glusterd_restore_op_version] 0-management: wrong op-version (70000) retrieved [Invalid argument]

the baselines should be committed to the repository itself, where as the current result should be updated from local run, and a graph image should be emailed.

We need to capture pbench[1] output as part of performance runs, and capture it somewhere on the resdir/

also see if we can expose the resdir through webserver.

Anyways, this issue is for installing and capturing details of pbench

[1] - https://github.com/distributed-system-analysis/pbench

Each run should produce a unique set of information that should be kept together. This is what I would record:

• Commit used for the test
• Date and time of the run (start and end)
• Configuration settings (any gluster options, system options, or anything else changed for the test)
• Environment (kernel, version, free memory, list of processes, cache status (dropped, not dropped), ...) both at the start and end of the test, from all nodes
• Output of all information captured during the test on all nodes (iostat, top, ...)
• stdout and stderr of all executed commands
• A user provided comment for this particular test

We should create this in some organized way so that it's easily searchable.

This is very useful when we have lots of runs and want to use them for deeper analysis or extract statistics.

It would be good if we run benchmark for both cold cache and hot cache, this will help us in identifying regressions in the cache xlators, and also give us an idea on the benefits of cache layers.

Once we run the performance tests, we have captured the files/sec, iops, bandwidth usage etc. We need to keep a history of all the runs and present in a visual format to be able to comprehend for the developers. One suggestion would be:

We have a github repo, where we push all the metrics collected on a nightly runs. The last 30 or n runs are compared plotted to get the trend of the performance.

The other option suggested is to have a database where we store all the runs metrics.

Currently we only run it for Gluster volumes' fuse mount. We should be running for Gluster block as well.

Should be able to input volume profile or options to run the tests against

Currently the tests are run for only replica 3. Need to be able to run it for different volume types

We needn't get an email if there is no regression seen.