Comments (3)
Node 1 reports slow heartbeats, and terminates after detecting disk stall:
F240515 12:54:52.692794 150690 1@util/log/file.go:269 â‹® [-] 265 disk stall detected: unable to sync log files within 20s
10s later, the roachtest could not connect to the node:
12:55:01 test_impl.go:414: test failure #1: full stack retained in failure_1.log: (allocator.go:359).func1: dial tcp 20.102.91.133:26257: connect: connection refused
from cockroach.
OTOH, the node 1 is asked to be decommissioned in this test, prior to the disk stall. I wonder if these two events correlate.
12:53:52 cluster.go:2371: running cmd `./cockroach node decommissi...` on nodes [:1]; details in run_125352.624956031_n1_cockroach-node-decom.log
12:53:53 allocator.go:403: 68 mis-replicated ranges
12:53:54 allocator.go:403: 47 mis-replicated ranges
12:53:55 allocator.go:403: 10 mis-replicated ranges
12:53:56 allocator.go:403: 0 mis-replicated ranges
12:53:57 allocator.go:374: SET CLUSTER SETTING server.time_until_store_dead = '90s'
F240515 12:54:52.692794 150690 1@util/log/file.go:269 â‹® [-] 265 disk stall detected: unable to sync log files within 20s
12:55:01 test_impl.go:414: test failure #1: full stack retained in failure_1.log: (allocator.go:359).func1: dial tcp 20.102.91.133:26257: connect: connection refused
from cockroach.
Noting that this is on Azure:
ROACHTEST_cloud=azure
Which has had a greater degree of instability in roachtests recently. I'd be pro chalking up to an infra flake.
from cockroach.
Related Issues (20)
- compose: TestComposeCompare failed [no locations to init GEOS] HOT 1
- sql/tests: TestRandomSyntaxGeneration failed
- roachtest: pg_regress failed HOT 8
- roachtest: backup-restore/round-trip failed HOT 2
- pkg/ccl/logictestccl/tests/3node-tenant-multiregion/3node-tenant-multiregion_test: TestTenantExecBuild_distsql_tenant_locality failed HOT 1
- pkg/server/application_api/application_api_test: TestStatusAPITransactions failed
- pkg/sql/logictest/tests/5node/5node_test: TestLogic_distsql_stats failed HOT 1
- cli: TestHalfOnlineLossOfQuorumRecovery failed HOT 9
- cli: TestLossOfQuorumRecovery failed HOT 16
- roachtest: schemachange/mixed-versions failed [functions cannot have more than 100 arguments] HOT 4
- ccl/backupccl: TestDataDriven_user_defined_functions_in_defaults failed HOT 1
- ccl/backupccl: TestDataDriven_procedures failed HOT 1
- : failed HOT 2
- pkg/sql/ttl/ttljob/ttljob_test: TestRowLevelTTLJobRandomEntries failed HOT 1
- server: TestDatabasesTablesV2 failed HOT 4
- kv/kvserver: TestReplicateAfterRemoveAndSplit failed HOT 7
- pkg/ccl/testccl/sqlccl/sqlccl_test: TestCancelMultipleSessions failed HOT 1
- sql/stats: TestCreateStatsProgress failed HOT 1
- roachtest: disk-stalled/wal-failover/among-stores failed HOT 14
- roachprod: fails with concurrent update on map error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.