At the end of high load workloads with high latency, runs fail to complete cleanly and no output summary is generated.

The following error(s) are printed + a very large table of hex values, presumably the raw content of the buffer that was the wrong size:

17:18:27.763 [AsyncHttpClient-3-1] ERROR HistogramDeserializer - Failed to decode publish delay latency: +-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 78 9c 2c b4 0b 68 9a f9 17 2d fa 37 31 6a 8c af |x.,..h...-.71j..|
|00000010| f8 8a 51 93 a8 79 9b 97 c9 f4 94 de 12 7a 43 09 |..Q..y.......zC.|
|00000020| a5 84 52 7a 4b 08 25 94 52 7a 4a 29 a5 94 52 ca |..RzK.%.RzJ)..R.|
|00000030| 30 0c c3 20 22 5e 11 11 f1 88 88 88 88 57 44 44 |0.. "^.......WDD|
|00000040| 44 44 44 44 82 88 88 57 44 82 88 88 04 09 22 22 |DDDD...WD.....""|
|00000050| 22 41 82 88 88 84 b3 3e ff 67 be f9 3e 7f 8f bd |"A.....>.g..>...|
...
|000fffd0| 12 a8 f5 77 83 f7 4f d5 ce 82 02 f7 be 86 2a 72 |...w..O.......*r|
|000fffe0| cf 77 b5 49 a9 d5 d0 ea 93 44 f7 20 d2 4c b8 fa |.w.I.....D. .L..|
|000ffff0| a1 dd d5 11 50 75 9b 79 |....Pu.y |
+--------+-------------------------------------------------+----------------+
17:18:27.844 [main] ERROR Benchmark - Failed to run the workload '500k rate 4 producers and 4 consumers on 1 topic / 100 partition' for driver 'driver-kafka/kafka-throughput.yaml'
java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: java.lang.IllegalArgumentException: The buffer does not contain the indicated payload amount (through reference chain: io.openmessaging.benchmark.worker.commands.CumulativeLatencies["publishLatency"])
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600) ~[?:?]
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678) ~[?:?]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:919) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:?]
at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:553) ~[?:?]
at io.openmessaging.benchmark.worker.DistributedWorkersEnsemble.getCumulativeLatencies(DistributedWorkersEnsemble.java:250) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
at io.openmessaging.benchmark.WorkloadGenerator.printAndCollectStats(WorkloadGenerator.java:525) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
at io.openmessaging.benchmark.WorkloadGenerator.run(WorkloadGenerator.java:143) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
at io.openmessaging.benchmark.Benchmark.lambda$main$0(Benchmark.java:173) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at io.openmessaging.benchmark.Benchmark.lambda$main$1(Benchmark.java:153) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
at java.util.TreeMap.forEach(TreeMap.java:1002) [?:?]
at io.openmessaging.benchmark.Benchmark.main(Benchmark.java:151) [io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: java.lang.IllegalArgumentException: The buffer does not contain the indicated payload amount (through reference chain: io.openmessaging.benchmark.worker.commands.CumulativeLatencies["publishLatency"])
......

Dear maintainers,

I'd like to know which kafka driver did you use for the latency benchmark in https://developer.confluent.io/learn/kafka-performance/?

Did you use acks=1 or acks=all or any other values for the latency benchmark?

Thanks in advance!

I am running a perf test for Apache Kafka and use the following command:

bin/benchmark
--drivers driver-kafka/kafka-exactly-once.yaml
workloads/simple-workload.yaml

After the command run for about 5 mins, the error message repeatedly as below, but the process is not broken, it keeps running:
09:11:17.616 [kafka-producer-network-thread | producer-1] WARN i.o.benchmark.worker.LocalWorker - Write error on message
java.util.concurrent.CompletionException: java.lang.ArrayIndexOutOfBoundsException: value outside of histogram covered range. Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1311669 out of bounds for length 1310720
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[na:na]
at io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkProducer.lambda$sendAsync$0(KafkaBenchmarkProducer.java:49) ~[driver-kafka-0.0.1-SNAPSHOT.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1363) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:228) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:197) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:653) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:634) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:554) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$0(Sender.java:743) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:566) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:558) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325) ~[kafka-clients-2.6.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:253) ~[kafka-clients-2.6.0.jar:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

and finally the application break with message:

09:11:40.997 [main] ERROR io.openmessaging.benchmark.Benchmark - Failed to run the workload 'Simple Workload 1 producer on 1 topic' for driver 'driver-kafka/kafka-exactly-once.yaml'
java.lang.NullPointerException: null
at io.openmessaging.benchmark.Benchmark.lambda$main$0(Benchmark.java:155) ~[classes/:na]
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[na:na]
at io.openmessaging.benchmark.Benchmark.lambda$main$1(Benchmark.java:138) ~[classes/:na]
at java.base/java.util.TreeMap.forEach(TreeMap.java:1002) ~[na:na]
at io.openmessaging.benchmark.Benchmark.main(Benchmark.java:129) ~[classes/:na]

The rate at which producers produce messages (per second). Note: a value of 0 means that messages are produced as quickly as possible, with no rate limiting.
Is there any problem as described in the official openmessaging documentation?
Why was situation with no rate limiting cancelled?
If you are free, I hope you can help answer my question, thank you very much.

sudo bin/benchmark --drivers kafka/kafka-exactly-once.yaml --workers 10.0.100.195:9092,10.0.100.97:9092,10.0.101.68:

9092 workloads/1-topic-16-partitions-1kb.yaml

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/sanjay/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.13.3/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/sanjay/.m2/repository/ch/qos/logback/logback-classic/1.0.13/logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
19:09:12.208 [main] INFO - Starting benchmark with config: {
"drivers" : [ "kafka/kafka-exactly-once.yaml" ],
"workers" : [ "http://10.0.100.195:9092", "http://10.0.100.97:9092", "http://10.0.101.68:9092" ],
"workersFile" : null,
"workloads" : [ "workloads/1-topic-16-partitions-1kb.yaml" ],
"output" : null
}
19:09:12.234 [main] INFO - Workloads: {
"1-topic-16-partitions-1kb" : {
"name" : "1 producer / 1 consumers on 1 topic",
"topics" : 1,
"partitionsPerTopic" : 16,
"keyDistributor" : "NO_KEY",
"messageSize" : 1024,
"payloadFile" : "payload/payload-1Kb.data",
"subscriptionsPerTopic" : 1,
"producersPerTopic" : 1,
"consumerPerSubscription" : 1,
"producerRate" : 50000,
"consumerBacklogSizeGB" : 0,
"testDurationMinutes" : 15,
"consumerOnly" : false
}
}
19:09:12.239 [main] INFO - Workers list - producers: [http://10.0.100.195:9092]
19:09:12.239 [main] INFO - Workers list - consumers: [http://10.0.100.97:9092]
19:09:12.432 [main] ERROR - Failed to run the workload '1 producer / 1 consumers on 1 topic' for driver 'kafka/kafka-exactly-once.yaml'
java.io.FileNotFoundException: kafka/kafka-exactly-once.yaml (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:?]
at java.io.FileInputStream.open(FileInputStream.java:216) ~[?:?]
at java.io.FileInputStream.(FileInputStream.java:157) ~[?:?]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:307) ~[jackson-dataformat-yaml-2.9.3.jar:2.9.3]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14) ~[jackson-dataformat-yaml-2.9.3.jar:2.9.3]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2903) ~[jackson-databind-2.9.10.5.jar:2.9.10.5]
at io.openmessaging.benchmark.Benchmark.lambda$main$0(Benchmark.java:141) ~[classes/:?]
at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
at io.openmessaging.benchmark.Benchmark.lambda$main$1(Benchmark.java:138) ~[classes/:?]
at java.util.TreeMap.forEach(TreeMap.java:1282) [?:?]

Pls suggest me where i was wrong here in conducting performance test for kafka

In the bin directory, there is no benchmark. So how to test it? like
sudo bin/benchmark
--drivers driver-kafka/kafka.yaml
workloads/1-topic-16-partitions-1kb.yaml

Error while starting the distributed mode:

Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/jetty/util/component/Graceful$Shutdown
	at org.eclipse.jetty.server.AbstractConnector.<init>(AbstractConnector.java:156)
	at org.eclipse.jetty.server.AbstractNetworkConnector.<init>(AbstractNetworkConnector.java:44)
	at org.eclipse.jetty.server.ServerConnector.<init>(ServerConnector.java:219)
	at org.eclipse.jetty.server.ServerConnector.<init>(ServerConnector.java:94)
	at org.eclipse.jetty.server.Server.<init>(Server.java:122)
	at org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:103)
	at io.openmessaging.benchmark.Benchmark.main(Benchmark.java:139)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.component.Graceful$Shutdown
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

This commit seems to break the distributed mode because of jetty version mismatch: b163731

What's wrong: the commit has only updated jetty-server and not jetty-util

Fix: Update jetty-util in benchmark-framework/pom.xml to the same version as jetty-server: 9.4.17.v20190418

Is it possible for someone to refresh the Docker image with the latest repo? When I deploy in Kubernetes using the provided Helm chart, the image: openmessaging/openmessaging-benchmark:latest yaml files are different and appear to be an older specification than the current ones in the repo.

For example:
Repo: driver-pulsar/pulsar.yaml, line 29:
clusterName: local namespacePrefix: benchmark/ns

Pod: driver-pulsar/pulsar.yaml
namespacePrefix: benchmark/local/ns

Hi,

I'm trying without sucess perform tests on our MSK IAM Serverless clusters using Kubernetes Openmessaging pods
As per MSK documentation we created all the necessary IAM roles and permissions but when we are running the perf tests the topic creation step stuck in a loop. Look the stack below

Nov 09, 2023 3:29:29 PM com.amazonaws.http.AmazonHttpClient addProxyConfig INFO: Configuring Proxy. Proxy Host: Proxy Port: 4438 15:29:29.408 [qtp1390301622-20] INFO AbstractLogin - Successfully logged in. 15:29:29.451 [qtp1390301622-20] WARN AdminClientConfig - The configuration 'debug' was supplied but isn't a known config. 15:29:29.453 [qtp1390301622-20] INFO AppInfoParser - Kafka version: 3.2.1 15:29:29.454 [qtp1390301622-20] INFO AppInfoParser - Kafka commitId: -- 15:29:29.454 [qtp1390301622-20] INFO AppInfoParser - Kafka startTimeMs: --- 15:29:29.499 [qtp1390301622-23] INFO WorkerHandler - Received create topics request for topics: { "numberOfTopics" : 1, "numberOfPartitionsPerTopic" : 10 } 15:29:39.518 [pool-2-thread-1] INFO KafkaTopicCreator -
Created topics 0/1 ^C

We are stuck on this loop "Created topics 0/1 " even with the connection successfully with the cluster

our configs:

commonConfig: | bootstrap.servers=<msk_serverless_url>.amazonaws.com:9098 security.protocol=SASL_SSL sasl.mechanism=AWS_MSK_IAM debug=true sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="arn:aws:iam::" awsDebugCreds=true awsRoleSessionName="msk-perf-test"; sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Our IAM policy has opened all permissions on Kafka to test
We did try using serviceaccount and run the perf tests but face same issue.

As another test, we used the same serviceaccount to the benchmark pods.
From the pod I did the download for Kafka binaries and execute the topic creation and did work.
./kafka-topics.sh --bootstrap-server :9098 --create --topic test-topic --partitions 1 --replication-factor 3 --command-config client.properties

./kafka-topics.sh --bootstrap-server <url>:9098 --create --topic test-topic --partitions 1 --replication-factor 3 --command-config client.properties [2023-11-13 15:23:06,550] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2023-11-13 15:23:06,550] WARN The configuration 'sasl.client.callback.handler.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) Created topic test-topic.
./kafka-topics.sh --list --bootstrap-server <url>:9098 --command-config client.properties [2023-11-13 15:26:16,255] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2023-11-13 15:26:16,256] WARN The configuration 'sasl.client.callback.handler.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) test-topic

Looks like the issue is using the KafkaTopicCreatror

any idea what we can do to make this happen?

When user doesn't specify --output file parameter, json file won't be written because of java.lang.NullPointerException:

bin/benchmark --drivers driver-kafka/kafka-0.yaml workloads/100-partitions-1000K-rate-4-producer.yaml 2>&1 | tee test3.log && grep Aggregated test3.log
...
10:54:22.257 [main] ERROR - Failed to run the workload '100 partitions 1000K rate 4 producer' for driver 'driver-kafka/kafka-0.yaml'
java.lang.NullPointerException: null
        at io.openmessaging.benchmark.Benchmark.lambda$null$0(Benchmark.java:155) [io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.ArrayList.forEach(ArrayList.java:1541) [?:?]
        at io.openmessaging.benchmark.Benchmark.lambda$main$1(Benchmark.java:138) [io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.TreeMap.forEach(TreeMap.java:1002) [?:?]
        at io.openmessaging.benchmark.Benchmark.main(Benchmark.java:129) [io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]

Need to fix followig line:

String fileName = arguments.output.length() > 0 ? arguments.output

F.e for

boolean useOutput = (arguments.output != null) && (arguments.output.length() > 0);
String fileName = useOutput? arguments.output: String.format("%s-%s-%s.json", workloadName, driverConfiguration.name,
                                    dateFormat.format(new Date()));

The first two tasks in the ansible playbook https://github.com/confluentinc/openmessaging-benchmark/blob/master/driver-kafka/deploy/deploy.yaml appear to be invalid

Namely the - name: Collect profiles and the - name: Heap allocation profile are both invalid since they seem to refer to files in /tmp. Not exactly sure where these files come from.

In addition, the region specified on line 70 in https://github.com/confluentinc/openmessaging-benchmark/blob/master/driver-kafka/deploy/provision-kafka-aws.tf needs to be changed to us-west-2a as 2b is not supported per the CLI output.

RabbitMQ recently introduced Streams (blog post) which are closer to Kafka semantics and offers higher through put over queues.

Current benchmark has limited throughput with RabbitMQ to 30 MB/s so that latency doesn't degrade. Benchmarks should be updated with streams to represent recent developments.