Thank you very much for your previous help, I reproduced the test results on RocksDB with shinjuku policy (using the test script from the experiment), but I got strange test results. The test results of Tput and response latency on RocksDB show obvious and stable jitter, just as below:
Further, I guess this jitter is related to the number of tests, so I repeat four tests under 90000 req/s( 'e.throughputs = [90000, 90000,90000, 90000]' in ghost-userspace/experiments/scripts/shinjuku.py), I got the below results and it seems to match my guess.
bazel run //experiments/scripts:shinjuku ghost
INFO: Analyzed target //experiments/scripts:shinjuku (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //experiments/scripts:shinjuku up-to-date:
bazel-bin/experiments/scripts/shinjuku
INFO: Elapsed time: 0.728s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Running ghOSt experiments...
mount: /dev/cgroup/cpu: cgroup already mounted on /sys/fs/cgroup/systemd.
mount: /dev/cgroup/memory: cgroup already mounted on /sys/fs/cgroup/systemd.
Output Directory: /tmp/ghost_data/2022-01-26 13:29:23
{"throughputs": [90000, 90000, 90000, 90000], "output_prefix": "/tmp/ghost_data/2022-01-26 13:29:23", "binaries": {"rocksdb": "/dev/shm/rocksdb", "antagonist": "/dev/shm/antagonist", "ghost": "/dev/shm/agent_shinjuku"}, "rocksdb": {"print_format": "pretty", "print_distribution": false, "print_ns": false, "print_get": true, "print_range": true, "rocksdb_db_path": "/dev/shm/orch_db", "throughput": 20000, "range_query_ratio": 0.005, "load_generator_cpu": 10, "cfs_dispatcher_cpu": 11, "num_workers": 200, "worker_cpus": [], "cfs_wait_type": "spin", "get_duration": "10us", "range_duration": "5000us", "get_exponential_mean": "0us", "batch": 1, "experiment_duration": "15s", "discard_duration": "2s", "scheduler": "ghost", "ghost_qos": 2}, "antagonist": null, "ghost": {"firstcpu": 11, "globalcpu": 11, "ncpus": 7, "preemption_time_slice": "30us"}}
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:I0/2701289 12:J0/2701284 13:K0/2701291 14:L0/2701260 15:M0/2701295 16:N0/2701294 17:none rq_l=194
SchedState: 11:none 12:O0/2701285 13:P0/2701286 14:Q0/2701292 15:L0/2701260 16:R0/2701287 17:M0/2701295 rq_l=194
SchedState: 11:S0/2701280 12:none 13:O0/2701285 14:P0/2701286 15:K0/2701291 16:M0/2701295 17:T0/2701296 rq_l=192
SchedState: 11:none 12:M0/2701295 13:J0/2701284 14:P0/2701286 15:S0/2701280 16:R0/2701287 17:N0/2701294 rq_l=194
SchedState: 11:T0/2701296 12:none 13:U0/2701293 14:O0/2701285 15:J0/2701284 16:K0/2701291 17:N0/2701294 rq_l=194
SchedState: 11:none 12:Q0/2701292 13:V0/2701290 14:K0/2701291 15:M0/2701295 16:L0/2701260 17:P0/2701286 rq_l=194
SchedState: 11:L0/2701260 12:none 13:Q0/2701292 14:T0/2701296 15:M0/2701295 16:S0/2701280 17:K0/2701291 rq_l=193
SchedState: 11:V0/2701290 12:L0/2701260 13:P0/2701286 14:none 15:O0/2701285 16:T0/2701296 17:M0/2701295 rq_l=193
SchedState: 11:none 12:S0/2701280 13:N0/2701294 14:W0/2701288 15:P0/2701286 16:L0/2701260 17:Q0/2701292 rq_l=194
SchedState: 11:S0/2701280 12:none 13:P0/2701286 14:L0/2701260 15:O0/2701285 16:Q0/2701292 17:V0/2701290 rq_l=193
SchedState: 11:none 12:O0/2701285 13:W0/2701288 14:L0/2701260 15:S0/2701280 16:T0/2701296 17:R0/2701287 rq_l=194
SchedState: 11:K0/2701291 12:P0/2701286 13:none 14:Q0/2701292 15:R0/2701287 16:V0/2701290 17:T0/2701296 rq_l=194
SchedState: 11:none 12:S0/2701280 13:Q0/2701292 14:M0/2701295 15:W0/2701288 16:O0/2701285 17:R0/2701287 rq_l=193
SchedState: 11:M0/2701295 12:none 13:T0/2701296 14:V0/2701290 15:N0/2701294 16:O0/2701285 17:K0/2701291 rq_l=193
SchedState: 11:none 12:L0/2701260 13:M0/2701295 14:V0/2701290 15:P0/2701286 16:K0/2701291 17:T0/2701296 rq_l=193
SchedState: 11:Q0/2701292 12:L0/2701260 13:none 14:S0/2701280 15:O0/2701285 16:V0/2701290 17:P0/2701286 rq_l=194
SchedState: 11:none 12:R0/2701287 13:T0/2701296 14:P0/2701286 15:O0/2701285 16:W0/2701288 17:Q0/2701292 rq_l=194
SchedState: 11:R0/2701287 12:none 13:O0/2701285 14:K0/2701291 15:Q0/2701292 16:W0/2701288 17:S0/2701280 rq_l=194
SchedState: 11:none 12:T0/2701296 13:U0/2701293 14:N0/2701294 15:R0/2701287 16:O0/2701285 17:V0/2701290 rq_l=193
SchedState: 11:U0/2701293 12:none 13:V0/2701290 14:K0/2701291 15:M0/2701295 16:N0/2701294 17:Q0/2701292 rq_l=193
SchedState: 11:none 12:W0/2701288 13:L0/2701260 14:U0/2701293 15:V0/2701290 16:R0/2701287 17:Q0/2701292 rq_l=194
SchedState: 11:R0/2701287 12:none 13:T0/2701296 14:Q0/2701292 15:N0/2701294 16:U0/2701293 17:V0/2701290 rq_l=193
SchedState: 11:V0/2701290 12:P0/2701286 13:none 14:S0/2701280 15:M0/2701295 16:L0/2701260 17:Q0/2701292 rq_l=193
SchedState: 11:none 12:M0/2701295 13:V0/2701290 14:W0/2701288 15:K0/2701291 16:O0/2701285 17:L0/2701260 rq_l=194
SchedState: 11:M0/2701295 12:none 13:Q0/2701292 14:S0/2701280 15:N0/2701294 16:V0/2701290 17:L0/2701260 rq_l=194
SchedState: 11:none 12:S0/2701280 13:U0/2701293 14:M0/2701295 15:N0/2701294 16:J0/2701284 17:W0/2701288 rq_l=193
SchedState: 11:L0/2701260 12:none 13:O0/2701285 14:V0/2701290 15:S0/2701280 16:R0/2701287 17:N0/2701294 rq_l=194
SchedState: 11:N0/2701294 12:W0/2701288 13:none 14:O0/2701285 15:S0/2701280 16:R0/2701287 17:J0/2701284 rq_l=194
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 572656 44050 1465511 4204176 6542181 6580594 6609205 6611629
Repeatable Handle Time 572656 44050 0 0 10 10 11 42
Worker Queue Time 572656 44050 0 1 68 147 742 121093
Worker Handle Time 572656 44050 10 30 115 274 1353 1397035
Total 572656 44050 1465541 4204227 6542240 6580652 6609244 6726298
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 2831 217 1465739 4085682 6550394 6580868 6604253 6610400
Repeatable Handle Time 2831 217 0 0 10 10 11 12
Worker Queue Time 2831 217 0 1 64 115 821 1747
Worker Handle Time 2831 217 28873 36197 156347 183477 1201632 5087580
Total 2831 217 1499557 4127825 6601265 6626808 6678678 8117214
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 575487 44268 1465511 4203104 6542215 6580606 6609179 6611629
Repeatable Handle Time 575487 44268 0 0 10 10 11 42
Worker Queue Time 575487 44268 0 1 68 146 744 121093
Worker Handle Time 575487 44268 10 31 269 13562 40787 5087580
Total 575487 44268 1465541 4203424 6542390 6581029 6609531 8117214
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:I0/2701830 12:none 13:J0/2701848 14:K0/2701840 15:L0/2701811 16:M0/2701843 17:N0/2701837 rq_l=194
SchedState: 11:none 12:O0/2701850 13:P0/2701839 14:M0/2701843 15:N0/2701837 16:Q0/2701833 17:R0/2701849 rq_l=193
SchedState: 11:S0/2701851 12:none 13:K0/2701840 14:Q0/2701833 15:O0/2701850 16:R0/2701849 17:L0/2701811 rq_l=194
SchedState: 11:none 12:S0/2701851 13:K0/2701840 14:M0/2701843 15:J0/2701848 16:T0/2701845 17:P0/2701839 rq_l=194
SchedState: 11:O0/2701850 12:none 13:I0/2701830 14:N0/2701837 15:S0/2701851 16:Q0/2701833 17:T0/2701845 rq_l=193
SchedState: 11:none 12:N0/2701837 13:U0/2701844 14:O0/2701850 15:Q0/2701833 16:J0/2701848 17:T0/2701845 rq_l=194
SchedState: 11:K0/2701840 12:none 13:V0/2701846 14:O0/2701850 15:Q0/2701833 16:I0/2701830 17:J0/2701848 rq_l=194
SchedState: 11:none 12:P0/2701839 13:V0/2701846 14:U0/2701844 15:K0/2701840 16:J0/2701848 17:Q0/2701833 rq_l=194
SchedState: 11:none 12:T0/2701845 13:S0/2701851 14:J0/2701848 15:U0/2701844 16:O0/2701850 17:W0/2701829 rq_l=194
SchedState: 11:Q0/2701833 12:none 13:U0/2701844 14:J0/2701848 15:V0/2701846 16:N0/2701837 17:O0/2701850 rq_l=194
SchedState: 11:none 12:W0/2701829 13:T0/2701845 14:K0/2701840 15:J0/2701848 16:U0/2701844 17:Q0/2701833 rq_l=194
SchedState: 11:P0/2701839 12:none 13:N0/2701837 14:J0/2701848 15:Q0/2701833 16:K0/2701840 17:U0/2701844 rq_l=194
SchedState: 11:none 12:N0/2701837 13:T0/2701845 14:J0/2701848 15:O0/2701850 16:P0/2701839 17:I0/2701830 rq_l=193
SchedState: 11:S0/2701851 12:N0/2701837 13:V0/2701846 14:none 15:T0/2701845 16:W0/2701829 17:U0/2701844 rq_l=194
SchedState: 11:none 12:S0/2701851 13:V0/2701846 14:T0/2701845 15:I0/2701830 16:R0/2701849 17:J0/2701848 rq_l=194
SchedState: 11:W0/2701829 12:none 13:O0/2701850 14:V0/2701846 15:R0/2701849 16:P0/2701839 17:U0/2701844 rq_l=194
SchedState: 11:none 12:J0/2701848 13:R0/2701849 14:T0/2701845 15:U0/2701844 16:O0/2701850 17:V0/2701846 rq_l=194
SchedState: 11:U0/2701844 12:none 13:I0/2701830 14:O0/2701850 15:Q0/2701833 16:R0/2701849 17:J0/2701848 rq_l=193
SchedState: 11:none 12:Q0/2701833 13:S0/2701851 14:I0/2701830 15:V0/2701846 16:J0/2701848 17:K0/2701840 rq_l=194
SchedState: 11:R0/2701849 12:none 13:T0/2701845 14:V0/2701846 15:S0/2701851 16:W0/2701829 17:O0/2701850 rq_l=194
SchedState: 11:none 12:W0/2701829 13:R0/2701849 14:N0/2701837 15:Q0/2701833 16:O0/2701850 17:T0/2701845 rq_l=194
SchedState: 11:none 12:V0/2701846 13:P0/2701839 14:S0/2701851 15:I0/2701830 16:W0/2701829 17:Q0/2701833 rq_l=194
SchedState: 11:N0/2701837 12:none 13:P0/2701839 14:K0/2701840 15:Q0/2701833 16:U0/2701844 17:O0/2701850 rq_l=194
SchedState: 11:W0/2701829 12:U0/2701844 13:none 14:T0/2701845 15:N0/2701837 16:K0/2701840 17:O0/2701850 rq_l=193
SchedState: 11:none 12:S0/2701851 13:P0/2701839 14:U0/2701844 15:J0/2701848 16:O0/2701850 17:I0/2701830 rq_l=194
SchedState: 11:P0/2701839 12:none 13:U0/2701844 14:W0/2701829 15:K0/2701840 16:I0/2701830 17:R0/2701849 rq_l=194
SchedState: 11:none 12:P0/2701839 13:S0/2701851 14:U0/2701844 15:K0/2701840 16:W0/2701829 17:V0/2701846 rq_l=194
SchedState: 11:Q0/2701833 12:none 13:N0/2701837 14:I0/2701830 15:U0/2701844 16:T0/2701845 17:K0/2701840 rq_l=194
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 343054 26388 3243726 6277485 9107046 9141095 9173059 9173688
Repeatable Handle Time 343054 26388 0 0 10 10 11 19
Worker Queue Time 343054 26388 0 1 65 152 850 170321
Worker Handle Time 343054 26388 10 34 105 298 1863 311271
Total 343054 26388 3243780 6277531 9107184 9141132 9173101 9279417
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 1794 137 3246198 6206710 9101283 9138715 9147969 9172264
Repeatable Handle Time 1794 137 0 0 10 11 11 12
Worker Queue Time 1794 137 0 1 60 85 761 6942
Worker Handle Time 1794 137 41065 51223 218551 365308 474447 560638
Total 1794 137 3302315 6278557 9159617 9188212 9202012 9217862
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 344848 26526 3243726 6277479 9106959 9141095 9173058 9173688
Repeatable Handle Time 344848 26526 0 0 10 10 11 19
Worker Queue Time 344848 26526 0 1 65 152 859 170321
Worker Handle Time 344848 26526 10 34 313 43870 59530 560638
Total 344848 26526 3243780 6277531 9110745 9141144 9173114 9279417
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:I0/2702424 12:none 13:J0/2702442 14:K0/2702440 15:L0/2702429 16:M0/2702437 17:N0/2702422 rq_l=194
SchedState: 11:none 12:O0/2702443 13:K0/2702440 14:L0/2702429 15:P0/2702435 16:Q0/2702419 17:R0/2702444 rq_l=194
SchedState: 11:L0/2702429 12:none 13:S0/2702439 14:K0/2702440 15:T0/2702421 16:I0/2702424 17:J0/2702442 rq_l=193
SchedState: 11:none 12:R0/2702444 13:N0/2702422 14:L0/2702429 15:Q0/2702419 16:K0/2702440 17:P0/2702435 rq_l=194
SchedState: 11:I0/2702424 12:none 13:T0/2702421 14:K0/2702440 15:O0/2702443 16:L0/2702429 17:P0/2702435 rq_l=194
SchedState: 11:none 12:K0/2702440 13:N0/2702422 14:M0/2702437 15:O0/2702443 16:R0/2702444 17:J0/2702442 rq_l=193
SchedState: 11:Q0/2702419 12:none 13:K0/2702440 14:T0/2702421 15:R0/2702444 16:P0/2702435 17:N0/2702422 rq_l=193
SchedState: 11:none 12:T0/2702421 13:Q0/2702419 14:L0/2702429 15:M0/2702437 16:U0/2702446 17:P0/2702435 rq_l=194
SchedState: 11:J0/2702442 12:none 13:P0/2702435 14:Q0/2702419 15:T0/2702421 16:I0/2702424 17:K0/2702440 rq_l=194
SchedState: 11:none 12:S0/2702439 13:N0/2702422 14:M0/2702437 15:R0/2702444 16:T0/2702421 17:K0/2702440 rq_l=194
SchedState: 11:none 12:none 13:L0/2702429 14:U0/2702446 15:N0/2702422 16:T0/2702421 17:M0/2702437 rq_l=195
SchedState: 11:M0/2702437 12:N0/2702422 13:L0/2702429 14:none 15:J0/2702442 16:P0/2702435 17:U0/2702446 rq_l=194
SchedState: 11:none 12:T0/2702421 13:P0/2702435 14:O0/2702443 15:R0/2702444 16:K0/2702440 17:J0/2702442 rq_l=193
SchedState: 11:M0/2702437 12:none 13:I0/2702424 14:Q0/2702419 15:T0/2702421 16:J0/2702442 17:P0/2702435 rq_l=193
SchedState: 11:none 12:O0/2702443 13:I0/2702424 14:U0/2702446 15:P0/2702435 16:R0/2702444 17:Q0/2702419 rq_l=193
SchedState: 11:O0/2702443 12:none 13:Q0/2702419 14:P0/2702435 15:T0/2702421 16:M0/2702437 17:J0/2702442 rq_l=194
SchedState: 11:none 12:S0/2702439 13:M0/2702437 14:L0/2702429 15:O0/2702443 16:P0/2702435 17:J0/2702442 rq_l=192
SchedState: 11:U0/2702446 12:none 13:L0/2702429 14:O0/2702443 15:T0/2702421 16:R0/2702444 17:Q0/2702419 rq_l=193
SchedState: 11:none 12:M0/2702437 13:S0/2702439 14:U0/2702446 15:J0/2702442 16:T0/2702421 17:I0/2702424 rq_l=193
SchedState: 11:K0/2702440 12:none 13:T0/2702421 14:L0/2702429 15:U0/2702446 16:P0/2702435 17:R0/2702444 rq_l=193
SchedState: 11:none 12:L0/2702429 13:O0/2702443 14:K0/2702440 15:I0/2702424 16:U0/2702446 17:M0/2702437 rq_l=194
SchedState: 11:U0/2702446 12:none 13:S0/2702439 14:M0/2702437 15:R0/2702444 16:P0/2702435 17:K0/2702440 rq_l=194
SchedState: 11:T0/2702421 12:M0/2702437 13:none 14:R0/2702444 15:U0/2702446 16:L0/2702429 17:J0/2702442 rq_l=194
SchedState: 11:M0/2702437 12:P0/2702435 13:J0/2702442 14:none 15:I0/2702424 16:T0/2702421 17:N0/2702422 rq_l=194
SchedState: 11:none 12:S0/2702439 13:I0/2702424 14:J0/2702442 15:P0/2702435 16:T0/2702421 17:V0/2702426 rq_l=194
SchedState: 11:O0/2702443 12:none 13:T0/2702421 14:U0/2702446 15:P0/2702435 16:N0/2702422 17:V0/2702426 rq_l=194
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 563830 43371 1595792 4074412 6648182 6677346 6689267 6696523
Repeatable Handle Time 563830 43371 0 0 10 10 11 69
Worker Queue Time 563830 43371 0 1 66 140 787 99939
Worker Handle Time 563830 43371 10 22 132 320 1549 234801
Total 563830 43371 1595809 4074461 6648240 6677392 6689308 6697744
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 2843 218 1595867 4120598 6648470 6677351 6690197 6696509
Repeatable Handle Time 2843 218 0 0 10 11 11 13
Worker Queue Time 2843 218 0 1 68 164 377 1955
Worker Handle Time 2843 218 27416 35634 170639 228286 411674 2951477
Total 2843 218 1631843 4161835 6694658 6721512 6777858 8264193
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 566673 43590 1595792 4074442 6648232 6677349 6689277 6696523
Repeatable Handle Time 566673 43590 0 0 10 10 11 69
Worker Queue Time 566673 43590 0 1 66 140 786 99939
Worker Handle Time 566673 43590 10 22 319 29051 40378 2951477
Total 566673 43590 1595809 4074661 6648736 6677411 6689404 8264193
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:none 12:I0/2702960 13:J0/2702958 14:K0/2702982 15:L0/2702961 16:M0/2702979 17:N0/2702971 rq_l=193
SchedState: 11:O0/2702941 12:none 13:P0/2702981 14:Q0/2702972 15:R0/2702976 16:S0/2702977 17:M0/2702979 rq_l=194
SchedState: 11:none 12:K0/2702982 13:J0/2702958 14:M0/2702979 15:P0/2702981 16:T0/2702967 17:R0/2702976 rq_l=194
SchedState: 11:S0/2702977 12:none 13:M0/2702979 14:N0/2702971 15:R0/2702976 16:Q0/2702972 17:U0/2702957 rq_l=194
SchedState: 11:none 12:Q0/2702972 13:O0/2702941 14:K0/2702982 15:I0/2702960 16:N0/2702971 17:M0/2702979 rq_l=193
SchedState: 11:I0/2702960 12:none 13:R0/2702976 14:N0/2702971 15:P0/2702981 16:K0/2702982 17:U0/2702957 rq_l=194
SchedState: 11:none 12:R0/2702976 13:J0/2702958 14:N0/2702971 15:I0/2702960 16:Q0/2702972 17:L0/2702961 rq_l=193
SchedState: 11:N0/2702971 12:R0/2702976 13:K0/2702982 14:none 15:T0/2702967 16:Q0/2702972 17:I0/2702960 rq_l=194
SchedState: 11:none 12:J0/2702958 13:S0/2702977 14:P0/2702981 15:Q0/2702972 16:M0/2702979 17:I0/2702960 rq_l=194
SchedState: 11:T0/2702967 12:none 13:J0/2702958 14:Q0/2702972 15:I0/2702960 16:K0/2702982 17:M0/2702979 rq_l=194
SchedState: 11:none 12:J0/2702958 13:Q0/2702972 14:L0/2702961 15:S0/2702977 16:I0/2702960 17:R0/2702976 rq_l=192
SchedState: 11:T0/2702967 12:none 13:O0/2702941 14:N0/2702971 15:L0/2702961 16:Q0/2702972 17:K0/2702982 rq_l=194
SchedState: 11:R0/2702976 12:I0/2702960 13:none 14:L0/2702961 15:U0/2702957 16:O0/2702941 17:S0/2702977 rq_l=193
SchedState: 11:none 12:U0/2702957 13:P0/2702981 14:R0/2702976 15:K0/2702982 16:L0/2702961 17:I0/2702960 rq_l=194
SchedState: 11:none 12:S0/2702977 13:R0/2702976 14:O0/2702941 15:Q0/2702972 16:T0/2702967 17:U0/2702957 rq_l=192
SchedState: 11:J0/2702958 12:none 13:M0/2702979 14:L0/2702961 15:T0/2702967 16:K0/2702982 17:P0/2702981 rq_l=194
SchedState: 11:none 12:I0/2702960 13:R0/2702976 14:T0/2702967 15:L0/2702961 16:N0/2702971 17:O0/2702941 rq_l=193
SchedState: 11:U0/2702957 12:none 13:N0/2702971 14:M0/2702979 15:R0/2702976 16:O0/2702941 17:I0/2702960 rq_l=194
SchedState: 11:M0/2702979 12:none 13:I0/2702960 14:J0/2702958 15:Q0/2702972 16:S0/2702977 17:T0/2702967 rq_l=194
SchedState: 11:none 12:R0/2702976 13:P0/2702981 14:Q0/2702972 15:O0/2702941 16:N0/2702971 17:T0/2702967 rq_l=194
SchedState: 11:Q0/2702972 12:none 13:I0/2702960 14:M0/2702979 15:K0/2702982 16:R0/2702976 17:P0/2702981 rq_l=193
SchedState: 11:none 12:I0/2702960 13:T0/2702967 14:Q0/2702972 15:O0/2702941 16:U0/2702957 17:J0/2702958 rq_l=194
SchedState: 11:none 12:N0/2702971 13:I0/2702960 14:R0/2702976 15:K0/2702982 16:L0/2702961 17:Q0/2702972 rq_l=194
SchedState: 11:M0/2702979 12:none 13:Q0/2702972 14:S0/2702977 15:R0/2702976 16:I0/2702960 17:T0/2702967 rq_l=193
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 380337 29256 2765284 5802543 8678875 8733697 8743672 8750075
Repeatable Handle Time 380337 29256 0 0 10 10 11 20
Worker Queue Time 380337 29256 0 1 65 150 817 135405
Worker Handle Time 380337 29256 10 34 103 267 1562 230800
Total 380337 29256 2765302 5802627 8678909 8733722 8743707 8750253
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 1885 144 2765459 5766490 8687513 8725394 8745680 8749604
Repeatable Handle Time 1885 144 0 0 10 11 11 11
Worker Queue Time 1885 144 0 1 63 83 672 997
Worker Handle Time 1885 144 39139 51529 214150 291318 454397 624325
Total 1885 144 2814735 5824114 8763182 8782982 8857589 8861919
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 382222 29401 2765284 5802238 8678890 8733662 8743673 8750075
Repeatable Handle Time 382222 29401 0 0 10 10 11 20
Worker Queue Time 382222 29401 0 1 65 149 813 135405
Worker Handle Time 382222 29401 10 34 261 11885 59095 624325
Total 382222 29401 2765302 5802870 8679397 8733851 8743783 8861919