Comments (7)
cdenneen
commented on September 17, 2024
Seeing the zk lock error flooding after received a bunch of these:
{"level":"warn","ts":1648046696.1697328,"msg":"error in OffsetResponse","type":"module","coordinator":"cluster","class":"kafka","name":"gw","sarama_error":"kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.","broker":3,"topic":"qa-logs-apms-apjson","partition":2}
{"level":"warn","ts":1648046696.1699715,"msg":"error in OffsetResponse","type":"module","coordinator":"cluster","class":"kafka","name":"gw","sarama_error":"kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.","broker":3,"topic":"dev-logs-electionsapi-syslog","partition":1}
{"level":"warn","ts":1648046696.169988,"msg":"error in OffsetResponse","type":"module","coordinator":"cluster","class":"kafka","name":"gw","sarama_error":"kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.","broker":3,"topic":"production-refeed-logs-iis","partition":1}
{"level":"warn","ts":1648046696.1701412,"msg":"error in OffsetResponse","type":"module","coordinator":"cluster","class":"kafka","name":"gw","sarama_error":"kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.","broker":3,"topic":"dev-logs-apnewsnr-apfeed","partition":2}
I can try to restart burrow but not sure what the cause here is.
Checking the consumers I see burrow-consumer_kafka in ERR state and a large TotalLag
from burrow.
cdenneen
commented on September 17, 2024
{"level":"info","ts":1648046313.133591,"msg":"send loop terminated: err=<nil>","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1648046325.1572301,"msg":"Connected to 10.x.x.x:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1648046325.221864,"msg":"authentication failed: zk: session has been expired by the server","type":"coordinator","name":"zookeeper"}
{"level":"error","ts":1648046325.2221205,"msg":"session expired","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1648046325.2221758,"msg":"stopping evaluations","type":"coordinator","name":"notifier"}
{"level":"info","ts":1648046325.2317216,"msg":"Connected to 10.x.x.y:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1648046325.2325819,"msg":"starting session","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1648046325.2409785,"msg":"authenticated: id=72057644841238557, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1648046325.2410588,"msg":"re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"warn","ts":1648046325.448404,"msg":"failed to get zk lock","type":"coordinator","name":"notifier","error":"zk: trying to acquire a lock twice"}
{"level":"warn","ts":1648046326.1202273,"msg":"failed to get zk lock","type":"coordinator","name":"notifier","error":"zk: trying to acquire a lock twice"}
{"level":"warn","ts":1648046326.241202,"msg":"failed to get zk lock","type":"coordinator","name":"notifier","error":"zk: trying to acquire a lock twice"}
from burrow.
cdenneen
commented on September 17, 2024
Everytime I restart burrow the burrow-consumer_kafka has a TotalLag over 6M.
Using kafka-consumer-groups I can't list or delete that consumer group (nor in KafkaTool).
Doing a curl -XDELETE I can delete it which will bring lag back to 0 (and then start climbing). It doesn't seem to save that consumer group update because the next time I restart burrow its back over 6M.
I can update the burrow.toml to change the client from consumer_kafka which might fix this but I'm not sure why I would need to do this.
from burrow.
cdenneen
commented on September 17, 2024
Nevermind. Even after a consumer rename the TotalLag is still over 6M:
ERR | burrow-consumer_kafka2 | 6,356,007 | gw
from burrow.
cdenneen
commented on September 17, 2024
[general]
access-control-allow-origin="*"
[logging]
level="info"
[zookeeper]
servers=["ZKHOST"]
timeout=6
[client-profile.profile]
kafka-version="2.1.0"
client-id="burrow-client"
tls="tlsprofile"
[tls.tlsprofile]
certfile="certificate.crt"
keyfile="keyfile.key"
noverify=false
[cluster.gw]
class-name="kafka"
client-profile="profile"
servers=["BROKER"]
topic-refresh=60
offset-refresh=30
groups-reaper-refresh=30
[consumer.consumer_kafka2]
class-name="kafka"
cluster="gw"
servers=["BROKER"]
client-profile="profile"
start-latest=false
offsets-topic="__consumer_offsets"
group-denylist="^(console-consumer-|python-kafka-consumer-).*$"
group-allowlist=""
[consumer.consumer_zk]
class-name="kafka_zk"
cluster="gw"
servers=["ZKHOST"]
zookeeper-timeout=30
group-denylist="^(console-consumer-|python-kafka-consumer-).*$"
group-allowlist=""
[httpserver.default]
address=":8000"from burrow.
cdenneen
commented on September 17, 2024
should I add burrow-consumer to the group-denylist? I do not see that anywhere in the instructions so I don't think that's correct.
from burrow.
muicoder
commented on September 17, 2024
You can try to deploy a ZooKeeper that is used alone for Notifier, setting start-latest=true will be better.
from burrow.
Related Issues (20)
- Cannot find storm consumers and kafka consumers
- Prevent flushing data during update
- Burrow force GOMAXPROCS to be equal to `runtime.NumCPU()` HOT 2
- Add possibility to change prometheus metrics name
- Query: Multiple Burrow instances HOT 2
- how to calculate burrow kafka_burrow_total_lag
- Using tls encrypted secrets
- Groups consuming from empty topics not listed
- Burrow does not exit after panic
- Consumer status is unpredictable when multiple topics are consumed
- Burrow report STOPPED on topic on group that no longer exist
- no matching versions for query "latest"
- Latest release of Burrow scans positive for http/2 rapid reset CVE HOT 3
- Linter needs updates after moving master to go 1.21.x
- Enable recommended default modules for golintci/revive
- add an example for deploying in kubernetes with the strimzi operator HOT 2
- Is Zookeeper mandated for using Burrow?
- Issue with Burrow from 1.6.0 to 1.7.0/1.8.0 HOT 11
- Lag is reported for deleted consumer group, if it only uses assign() API HOT 5
- Go directive in go.mod is invalid
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from burrow.