Giter VIP home page Giter VIP logo

Comments (3)

pav-kv avatar pav-kv commented on August 26, 2024

Node 1 reports slow heartbeats, and terminates after detecting disk stall:

F240515 12:54:52.692794 150690 1@util/log/file.go:269 â‹® [-] 265  disk stall detected: unable to sync log files within 20s

10s later, the roachtest could not connect to the node:

12:55:01 test_impl.go:414: test failure #1: full stack retained in failure_1.log: (allocator.go:359).func1: dial tcp 20.102.91.133:26257: connect: connection refused

from cockroach.

pav-kv avatar pav-kv commented on August 26, 2024

OTOH, the node 1 is asked to be decommissioned in this test, prior to the disk stall. I wonder if these two events correlate.

12:53:52 cluster.go:2371: running cmd `./cockroach node decommissi...` on nodes [:1]; details in run_125352.624956031_n1_cockroach-node-decom.log
12:53:53 allocator.go:403: 68 mis-replicated ranges
12:53:54 allocator.go:403: 47 mis-replicated ranges
12:53:55 allocator.go:403: 10 mis-replicated ranges
12:53:56 allocator.go:403: 0 mis-replicated ranges
12:53:57 allocator.go:374: SET CLUSTER SETTING server.time_until_store_dead = '90s'

F240515 12:54:52.692794 150690 1@util/log/file.go:269 â‹® [-] 265  disk stall detected: unable to sync log files within 20s

12:55:01 test_impl.go:414: test failure #1: full stack retained in failure_1.log: (allocator.go:359).func1: dial tcp 20.102.91.133:26257: connect: connection refused

from cockroach.

kvoli avatar kvoli commented on August 26, 2024

Noting that this is on Azure:

ROACHTEST_cloud=azure

Which has had a greater degree of instability in roachtests recently. I'd be pro chalking up to an infra flake.

from cockroach.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.