Comments (5)
scholzj
commented on May 20, 2024
Please share the full log and not just a snippet.
from strimzi-kafka-operator.
see-quick
commented on May 20, 2024
Give me a moment, I will share more logs :)
Update: Here is the log from the UserOperator container [1]
[1] - https://drive.google.com/file/d/1juNl65yfXlQankK_t7eQttvoCopyjqri/view?usp=sharing
from strimzi-kafka-operator.
see-quick
commented on May 20, 2024
So based on the suggestion from @scholzj (i.e., try to increase the length of the username), we found out that it might be the problem of the LIMIT for the KafkaAdmin API request. We found out that KafkaUsers:
- with the length of 22 chars it can handle over 4800 users
- with the length of 44 chars it can handle over 4100 users
....
An interesting fact is that it's not linear (i.e., 44 chars does allow more than 2400 users to be created). So there might be some compression involved....
Here is another log from the User Operator but with 44 chars usernames [1].
This can be found in ZK pods, which could be related to large request with communication of ZK <-> Kafka
2024-04-24 14:58:18,783 WARN Closing connection to /10.128.15.132:59788 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-7-4]
java.io.IOException: Len error 1048654
at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:521)
at org.apache.zookeeper.server.NettyServerCnxn.processQueuedBuffer(NettyServerCnxn.java:405)
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.userEventTriggered(NettyServerCnxnFactory.java:324)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117)
at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:387)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:400)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
at io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396)
at io.netty.channel.AbstractChannelHandlerContext.access$500(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$6.run(AbstractChannelHandlerContext.java:381)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-04-24 14:58:18,783 DEBUG close called for session id: 0x30049a59a2and in Kafka logs one could see:
2024-04-24 15:09:18,292 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-3-1]
2024-04-24 15:09:18,292 WARN Session 0x20049a87f010001 for server cluster-f32bb7bf-zookeeper-client/172.30.106.133:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(cluster-f32bb7bf-zookeeper-client:2181)]
EndOfStreamException: channel for sessionid 0x20049a87f010001 is lost
at org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
2024-04-24 15:09:18,293 INFO [ZooKeeperClient ACL authorizer] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [/kafka-acl-changes-event-process-thread]
2024-04-24 15:09:18,293 INFO [ZooKeeperClient ACL authorizer] Connected. (kafka.zookeeper.ZooKeeperClient) [/kafka-acl-changes-event-process-thread]
2024-04-24 15:09:18,293 INFO [ZooKeeperClient ACL authorizer] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [data-plane-kafka-request-handler-7]
2024-04-24 15:09:18,293 INFO [ZooKeeperClient ACL authorizer] Connected. (kafka.zookeeper.ZooKeeperClient) [data-plane-kafka-request-handler-7]
but Pods are up and ready with Ready state:
cluster-f32bb7bf-b-c2961f8f-0 1/1 Running 0 50m
cluster-f32bb7bf-b-c2961f8f-1 1/1 Running 0 50m
cluster-f32bb7bf-b-c2961f8f-2 1/1 Running 0 50m
cluster-f32bb7bf-entity-operator-76f48ccf76-khfp8 2/2 Running 0 50m
cluster-f32bb7bf-kafka-exporter-8b5f8586-rx9zm 1/1 Running 0 50m
cluster-f32bb7bf-scraper-8549f7f979-lpz5x 1/1 Running 0 51m
cluster-f32bb7bf-zookeeper-0 1/1 Running 0 51m
cluster-f32bb7bf-zookeeper-1 1/1 Running 0 51m
cluster-f32bb7bf-zookeeper-2 1/1 Running 0 51m
...[1] - https://drive.google.com/file/d/1kAj27sWi79kGs96i3vCWegpfhrz_cBJV/view?usp=sharing
from strimzi-kafka-operator.
Related Issues (20)
- [ST] Add migration system tests with JBOD support in KRaft mode HOT 2
- [Bug]: the k8s apiserver using--goaway-chance flag resulted in createOrUpdate failed HOT 2
- [Bug]: Kafka/ZooKeeper cannot schedule due to pod antiaffinity HOT 3
- [Bug]: generateCertificatesSecret will exceed Secret max size HOT 2
- Refactor `CertUtils.buildTrustedCertificateSecret` method and move it to `ClusterCa` class HOT 1
- [DOC] Document the `strimzi.io/force-rebuild` annotation
- [Bug]: local keystore passwords appear in the process table HOT 5
- Remove KRaft JBOD storge check once support for Kafka 3.6.0 is dropped
- MM2 connector auto-restarting does not seem to work
- [Enhancement]: Allow configmaps/secrets to be used for javaSystemProperties HOT 3
- [Enhancement]: Improve tools/report.sh to make only the namespace required (make the cluster name optional) HOT 1
- [Bug]: TopicOperator removes throttled.replicas topic configuration of throttled reassignment/rebalance HOT 15
- [Docs]: Add compatibility matrix table to docs HOT 4
- [ST] MigrationST - some `KafkaTopics` are not deleted correctly after the migration tests
- [Bug]: KafkaUser secret decoding improperly HOT 2
- [Bug]: Current offset is higher than last offset
- [Bug]: Downtime due to pod readinessProbe when rolling update Kafka pod
- Logging configuration is never updated for Connect when connector operator is disabled HOT 1
- [Enhancement]: hot reload Kafka on changes in `brokerCertChainAndKey` instead of a rolling update HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from strimzi-kafka-operator.