try to trim down the events to those needed by ceph to save on capture size.

In librbdfio.py, before every test, dump_config is called to save the ceph config into a file called ceph_settings.out. I don't think this ever completes successfully because that file is never created.

Copy ceph.conf and cbt yaml configuration into archive directory, so we have the exact test configuration.

cbt checks the rados version to determine the syntax of the object size for rados bench. It does so locally on the head node (by running rados -v) which fails if the rados binary (from ceph-common on Red Hat systems) is not installed.
Instead cbt should ask the first client node for the version, if it assumes all clients to have the same rados version installed.

It would be great to have the ability to specify the clients, eg:

clients: [client1, client2, client3, client4 client5, client6]

Then also specify an array with the number of clients to use for each iteration, eg:

num_clients: [1, 2, 4, 6]

This would be useful for finding the client concurrency contention threshold for a cluster.

Can be on the lines of from a cosbench profile:

Auth type
Config details
Workstage
Work type
Workstage
Work type

When changing Ceph parameters, some of them require Ceph services, like OSD, to be restarted. CBT can only recreate a brand new cluster or use existing cluster. It would be nice to have something like use_existing, but it would restart all Ceph services and check health before starting the test(s).

Hi
I was trying to setup CBT on a fresh system and encountered the following problem

[root@ceph-node1 cbt]# python cbt.py
Traceback (most recent call last):
  File "cbt.py", line 9, in <module>
    import benchmarkfactory
  File "/root/cbt/benchmarkfactory.py", line 10, in <module>
    from benchmark.cosbench import Cosbench
  File "/root/cbt/benchmark/cosbench.py", line 8, in <module>
    import lxml.etree as ET
ImportError: No module named lxml.etree
[root@ceph-node1 cbt]#

The fix was simple yum -y install python-lxml, just for documentation purpose i am creating this issue. I will submit a PR for updating documentation.

When using use_existing, readahead is not set on the OSDs because they do not have the labels CBT expects. We should probably determine which devices are mounted in /var/lib/osd/* and then set their readahead values.

17:56:52 - DEBUG - cbt - pdsh -R ssh -w root@ceph echo 3 | sudo tee /proc/sys/vm/drop_caches
^@17:56:53 - ERROR - cbt - During tests
Traceback (most recent call last):
File "./cbt.py", line 71, in main
b.run()
File "/root/cbt/benchmark/radosbench.py", line 68, in run
self._run('write', '%s/write' % self.run_dir, '%s/write' % self.out_dir)
File "/root/cbt/benchmark/radosbench.py", line 81, in _run
rados_version_str = subprocess.check_output(["rados", "-v"])
File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output
process = Popen(stdout=PIPE, _popenargs, *_kwargs)
File "/usr/lib64/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

wrong syntax in monitoring.py causing collectl to not run:

collectl -s+mYZ -i 1:10 --rawdskfilt "+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ " -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00001024/op_size-00004094/concurrent_procs-004/iodepth-064/read/pool_monitoring/collectl
Quantifier follows nothing in regex; marked by <-- HERE in m/+ <-- HERE cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ / at /usr/share/collectl/formatit.ph line 235.

so instead
rawdskfilt = '+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ '

it should be
rawdskfilt = 'cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ '

if deploying (use_existing is false) -
Update cosbench.py to create / update the swift cosbench:operator user

--relies on rgws[] and first rgw having radosgw-admin

make example test run configs for most common workloads that can get people started.

I have installed ceph in my head and client node, and ceph cluster is up running. I have no idea which file or directory does it need!

09:41:23 - ERROR - cbt - During tests
Traceback (most recent call last):
File "./cbt.py", line 64, in main
b.initialize()
File "/root/cbt-master/benchmark/librbdfio.py", line 68, in initialize
super(LibrbdFio, self).initialize()
File "/root/cbt-master/benchmark/benchmark.py", line 44, in initialize
self.cluster.cleanup()
File "/root/cbt-master/cluster/ceph.py", line 210, in cleanup
common.pdsh(nodes, 'sudo rm -rf %s' % self.tmp_dir).communicate()
File "/root/cbt-master/common.py", line 70, in pdsh
return CheckedPopen(args,continue_if_error=continue_if_error)
File "/root/cbt-master/common.py", line 20, in init
self.popen_obj = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
File "/usr/lib64/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

potentially use some sort of uuid labeling + SQLite indexing?

check the exit code of

pdsh -w @hosts true

before actually starting the tests

Ability to trigger deep scrubs to test the impact of deep scrubbing. In hammer there is supposed to be a ceph.conf tunable that allows specifying a time. If that value can be injected, then we can probably do something along the lines of:

current_time = time
schedule_time = current_time + ramp_time

ceph tell * injectargs --some_scrubbing_schedule_key $schedule_time

wrong in script:
rawdskfilt = 'cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ '

should be ('+' on the beginning) :
rawdskfilt = '+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ '

also it is better to run collectl with sudo and with rotation of logfiles (after midnight for example), so:
common.pdsh(nodes, 'sudo collectl -s+mYZ -i 1:10 -r00:00,7 --rawdskfilt "%s" -F0 -f %s' % (rawdskfilt, collectl_dir))

Work with Bitergia. Perhaps utilize work already in CETune (Intel - Jian)?

Where is the Log after ceph running ?I can't find it

in order to improve extendability (especially setup phase) -- Radoslaw, Mirantis

send a metric with a value of 1 to a specified graphite host with a key that matches the name of the test being run (matching the directory output structure)

0000000_radosbench_etc

Probably pretty easy with something like this:

https://github.com/daniellawrence/graphitesend

Hi
while performing rdcp the script just hangs. However all files and folders are copied to the target ARCH directory, but script doesn't continue on other steps.

Any idea?

6:42:18 - DEBUG - cbt - Nodes : ceph@cbt-node1,cbt-node2,cbt-node3
16:42:18 - DEBUG - cbt - CheckedPopen continue_if_error=True args=pdsh -f 3 -R ssh -w ceph@cbt-node1,cbt-node2,cbt-node3 sudo pkill -SIGINT -f blktrace
16:42:19 - DEBUG - cbt - Nodes : ceph@cbt-node1,ceph@cbt-node2,ceph@cbt-node3,ceph@cbt-node1,cbt-node2,cbt-node3
16:42:19 - DEBUG - cbt - CheckedPopen continue_if_error=False args=pdsh -S -f 6 -R ssh -w ceph@cbt-node1,ceph@cbt-node2,ceph@cbt-node3,ceph@cbt-node1,cbt-node2,cbt-node3 sudo chown -R ceph.ceph /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-003/iodepth-064/read/*

16:42:19 - DEBUG - cbt - CheckedPopen continue_if_error=False args=rpdcp -f 10 -R ssh -w ceph@cbt-node1,ceph@cbt-node2,ceph@cbt-node3,ceph@cbt-node1,cbt-node2,cbt-node3 -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-003/iodepth-064/read/* /tmp/ARCH1/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-003/iodepth-064/read

Thanks

If osd_per_node is unset, then cbt will crash mid way through the run, even with use_existing set to true.

Hi,
just finished benchmark of our Ceph cluster with CBT tool and now I'm confused how to deal with output data. I got a large directory structure with a lot of "id" directories like:

results/00000000/id3309284191688069537/

where are fio log files and some files in json format.

My point is if there is any tool or simple way how to parse these output data to human readable format, that I would be also able to compare two or more CBT benchmark tests.

Thank you

rbdfio.py should unmap all rbd devices before beginning a run

Adding a clocksource option for fio benchmarks would be useful so that gettimeofday can be selected when there are multiple fio processes running on multiple hosts. Assuming ntp is properly configured, this would make aggregating data much easier than it would be otherwise with independent, relative timestamps.

cbt is getting idle while trying to copy collectl data by executing the rpdcp command.
Because it getting the nodes by
nodes = settings.getnodes('clients', 'osds', 'mons', 'rgws')
When same hosts are defined under clients, osds, mons, rgws it executes collectl command on each of them 4 times. Rpdcp then cannot finish copy of the files as they are still opened.

The current recovery machinery for CBT marks an OSD or group of OSDs out, then later marks it back in. It would be great is there was a way to mark an OSD or group of OSDs out, destroy them, then add them back in to the cluster.

Many benchmarks delete and recreate the test pool between runs. However, the Ceph command to delete a pool returns immediately, and the work of deleting objects in the pool takes place in the background. Unfortunately, experience has shown that disk and CPU resources used while deleting the objects is great enough that it influences the test results for the subsequent run.

One way to avoid the problem is to have the cluster.rmpool() function wait until the disk and CPU utilization on the OSD nodes drops to a reasonable level before returning to the caller. I will be issuing a pull request with this change.

update to include simple cephFS support.

Reuse rbdfio benchmark
update with new cephfsfio.py benchmark and examples
update ceph.py to create MDSs

CBT current does not support testing RBD Erasure coding (ceph 12.2.2). ceph.py needs to be updated such that when librbdfio.py calls mkpool with a data pool specified ceph.py uses the data_pool_profile instead of the pool_profile.

Since librbdfio will append "-data" to the pool name if a user had specified a data_pool the below could be a possible solution.

cbt/cluster/ceph.py:
def mkpool(self, name, profile_name, application, base_name=None):
if 'data' in name:
pool_profiles = self.config.get('data_pool_profiles', {'default': {}})
else:
pool_profiles = self.config.get('pool_profiles', {'default': {}})
. . .

It would be great to have the ability to save the results archive to a S3 bucket instead of it only being on the local filesystem of the head node.

When executing the cbt.py test suite, it is very hard to figure out which steps failed/passed.
My experience with this tool is very limited as I just started using it, but I see that the pdsh commands fail without any error, so it is hard to decipher why.

Also, the use_existing flag in cluster: configuration in the yaml file should be highlighted when using against an existing cluster. Once I go through a successful execution I will create a pull request for any doc changes if makes sense and other issues if I see.

Another issue I see is username and groupname are taken as the same which is not the case. Might be useful to add a groups filed as well.

Lastly -->
Now I think I have gotten past some of my inital hurdles and am able to execute an fio benchmark, but I am not sure what is next.

The last step I see is:

21:30:37 - DEBUG - cbt - pdsh -R ssh -w [email protected],[email protected],[email protected] sudo chown -R behzad_dastur.behzad_dastur /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-01048576/concurrent_procs-001/iodepth-064/randwrite/* 21:30:37 - DEBUG - cbt - rpdcp -f 1 -R ssh -w [email protected],[email protected],[email protected] -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-01048576/concurrent_procs-001/iodepth-064/randwrite/* /tmp/00000000/LibrbdFio/osd_ra-00004096/op_size-01048576/concurrent_procs-001/iodepth-064/randwrite

I can see logs created at:

[root@cbtvm001-d658 cbt]# ls /tmp/00000000/LibrbdFio/osd_ra-00004096/op_size-01048576/concurrent_procs-001/iodepth-064/read/ collectl.b-stageosd001-r19f29-prod.acme.symcpe.net collectl.v-stagemon-002-prod.abc.acme.net output.0.v-stagemon-001-prod.abc.acme.net collectl.v-stagemon-001-prod.abc.acme.net historic_ops.out.b-stageosd001-r19f29-prod.abc.acme.net
Are there ways to now visualize this data.

when using use_existing, being able to reuse already filled rbd volumes instead of having to write a substantial amount of data again would be a huge time saver.

In using kvmrbdfio.py to run fio on my openstack guests, the population of the fio files does not use the path to the fio executable (fio_cmd) provided in the yaml file. I define the fio command path ...

----------- yaml excerpt -----------
benchmarks:
kvmrbdfio:
fio_cmd: "/usr/local/bin/fio"
----------- end excerpt ------------

... however the populate call instead uses no path and relies on the fio executable being in the $PATH defined by sudo users, which excludes /usr/local/bin where my fio resides, resulting in the population failing on the target systems.

Use iperf for all-to-all network tests. Base code on somthing vaguely like:

Client to OSD tests:

!/bin/bash

for i in 8 9 10 11 12 13
do
val=$((62+$i))
pdsh -R ssh -w osd[$i] iperf -s -B 172.27.50.$val &
done

!/bin/bash

for i in 0 1 2 3 4 5 6 7
do
for val in 70 71 72 73 74 75
do
pdsh -R ssh -w client[$i] iperf -c 172.27.50.$val -f m -t 60 -P 1 > /tmp/iperf_client${i}to${val}.out &
done
done

!/bin/bash

for i in 8 9 10 11 12 13
do
val=$((62+$i))
pdsh -R ssh -w osd[$i] iperf -s -B 172.27.49.$val &
done

!/bin/bash

for i in 8 9 10 11 12 13
do
for val in 70 71 72 73 74 75
do
pdsh -R ssh -w osd[$i] iperf -c 172.27.49.$val -P 1 -f m -t 60 -P 1 > /tmp/iperf_${i}to${val}.out &
done
done

The librbdfio benchmark creates RBDs using node names specified in the job file. However, the clients try to access the RBDs using their own "hostname -s" name. If the names are not the same then tests will fail, with output like this in the output files:

Starting 1 process
rbd engine: RBD version: 0.1.9
rbd_open failed.
fio_rbd_connect failed.

I will be submitting a fix.

Currently only mode supported is write
"read" mode will require a prepare stage to create objects

Also should support mix mode and allow user to specify read and write ratios

A cluster composed of nodes with a mix of SSD and HDD, where the SSDs are being used as OSDs, and not for a cache tier. Currently, we can test either the flash take root with CBT, or the HDD take root, but not both concurrently. Add ability to have clients test pools with different take roots, concurrently.

Currently CBT does not support deleting both a metadata pool and data pool when use_existing_volumes is set to false, this is most common when testing with RBD erasure coding pools.

have an aggregate summary of test results.

Hi
changing the concurrent_procs option vaule in yaml file doesn't have any affect to the cbt. It always working with value '3'. I found this while using benchmark librbdfio.

THX

The radosbench test uses the 'rados' tool, which requires the client admin key to access the cluster. The rados command will fail unless the user specified in the config file has read acess to the client admin keyring.

A workaround exists, and that is to set "cmd_path: 'sudo /usr/bin/rados'" in the test configuration. But this is cumbersome and breaks the ability to run the command under valgrind.

do a best effort check to make sure CBT wont crash halfway through a test run.