Comments (2)
Waiting for initial up-replication failed due to unavailable replicas. The breaker tripped, then reset just prior to the LoQ hit.
E240314 06:00:11.279758 2445 kv/kvserver/replica_circuit_breaker.go:167 ⋮ [T1,Vsystem,n1,s1,r36/1:‹/Table/3{3-4}›] 213 breaker: tripped with error: replica unavailable: (n1,s1):1 unable to serve request to r36:‹/Table/3{3-4}› [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1710395942.027842257,0 (2024-03-14 05:59:02); raft status: {"id":"1","term":6,"vote":"1","commit":19,"lead":"0","raftState":"StatePreCandidate","applied":19,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/33]
...
# n2, n3 join cluster around 06:05:11
06:05:15 util.go:105: waiting for initial up-replication... (<2m0s)
I240314 06:05:15.780659 2489 kv/kvserver/replica_circuit_breaker.go:142 ⋮ [T1,Vsystem,n1,s1,r36/1:‹/Table/3{3-4}›] 1488 breaker: breaker reset
W240314 06:05:15.781790 13374 sql/user.go:280 ⋮ [T1,Vsystem,n1,client=34.48.29.137:58744,hostssl,user=‹roachprod›] 1493 user lookup for ‹"roachprod"› failed: get auth info error: ‹get-login-dependencies›: replica unavailable: (n1,s1):1 unable to serve request to r36:‹/Table/3{3-4}› [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1710395942.027842257,0 (2024-03-14 05:59:02); raft status: {"id":"1","term":6,"vote":"1","commit":19,"lead":"0","raftState":"StatePreCandidate","applied":19,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/33]
from cockroach.
I'm going to consider this a testing blip, the query for waiting for up-replication could have retries and wait out the unavailability reset. It is odd that the breakers reset and the nodes rejoin but 4s later the range is still considered unavailable. Its possible that they weren't caught up yet.
from cockroach.
Related Issues (20)
- compose: TestComposeCompare failed [no locations to init GEOS] HOT 2
- corpus: TestValidateCorpuses failed HOT 3
- pkg/sql/schemachanger/corpus/corpus_test: TestValidateCorpuses failed HOT 2
- roachtest: splits/load/ycsb/a/nodes=3/obj=cpu failed
- roachtest: rebalance/by-load/leases/mixed-version failed HOT 4
- sql: don't type-check subqueries in routine argument defaults HOT 3
- ccl/changefeedccl: TestCoreChangefeedBackfillScanCheckpoint failed HOT 3
- kv/kvserver/concurrency: TestLockTableConcurrentRequests failed HOT 8
- Inconsistent results by the hint LOOKUP HOT 2
- opt: constant value projections prevent join reordering HOT 7
- roachprod: fstab entry fails with local SSDs HOT 1
- roachtest: ssh_problem failed HOT 35
- Sentry: catch.go:29: runtime error: invalid memory address or nil pointer dereference (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/u...
- roachprod: flag to enable cron service HOT 1
- raft: make follower commit index advancement safe HOT 1
- kv/kvserver/liveness: TestScanNodeVitalityFromKV failed HOT 2
- kv/kvserver: TestDiscoverIntentAcrossLeaseTransferAwayAndBack failed HOT 3
- pkg/sql/logictest/tests/fakedist-vec-off/fakedist-vec-off_test: TestLogic_alter_table failed HOT 3
- roachtest: disk-stalled/cgroup/read-write/logs-too=true failed HOT 1
- roachtest: failover/chaos/read-only failed HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.