Giter VIP home page Giter VIP logo

dev's People

Contributors

crewjam avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

bairahmat

dev's Issues

fleet failure should force a reboot or shutdown or something

We had a cluster of six machines, 3 master and 3 worker. But fleetctl list-machines showed only one worker. This came recently after the failure described in #1, so it is possible that fleet died in reaction to the etcd failure.

Regardless there are classes of failure where we should reboot/terminate/bail:

  • failure of fleet on a worker
  • failure of etcd on a master
  • ...

terminating a one of three etcd masters caused massive fail

The chain of events:

  1. Terminate a master node #1
  2. Notice that etcd on master node #2 hangs with:
$ fleetctl list-units
2014/10/11 02:38:21 ERROR fleetctl.go:171: error attempting to check latest fleet version in Registry: timeout reached
2014/10/11 02:38:21 INFO client.go:278: Failed getting response from http://127.0.0.1:4001/: cancelled
Error retrieving list of units from repository: timeout reached
  1. Notice that etcd on master node #3 fails with:
$ ssh 10.10.126.61 etcdctl ls /
Error:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
  1. Issue systemctl restart etcd on master #2
  2. Notice that fleet is all screwed up:
$ fleetctl list-units
UNIT                    MACHINE             ACTIVE      SUB
[email protected]       268251a3.../10.10.126.61    inactive    dead
[email protected]       4e283163.../10.10.146.25    inactive    dead
[email protected]       836de616.../10.10.1.191     inactive    dead
[email protected]            268251a3.../10.10.126.61    inactive    dead
[email protected]            4e283163.../10.10.146.25    inactive    dead
[email protected]            836de616.../10.10.1.191     activating  start-pre
[email protected]    268251a3.../10.10.126.61    activating  start-pre
[email protected]    4e283163.../10.10.146.25    activating  start-pre
[email protected]    836de616.../10.10.1.191     active      running
[email protected]     836de616.../10.10.1.191     active      running
[email protected]     4e283163.../10.10.146.25    activating  start-pre
[email protected]     268251a3.../10.10.126.61    activating  start-pre
[email protected]      268251a3.../10.10.126.61    activating  start-pre
[email protected]      4e283163.../10.10.146.25    activating  start-pre
[email protected]      836de616.../10.10.1.191     active      running
[email protected]         836de616.../10.10.1.191     active      running
[email protected]         4e283163.../10.10.146.25    active      running
[email protected]         268251a3.../10.10.126.61    active      running
[email protected]         836de616.../10.10.1.191     activating  start-pre
[email protected]         4e283163.../10.10.146.25    inactive    dead
[email protected]         268251a3.../10.10.126.61    failed      failed
[email protected]            268251a3.../10.10.126.61    active      running
[email protected]            4e283163.../10.10.146.25    active      running
[email protected]            836de616.../10.10.1.191     active      running
nginx-presence-dns.service      4e283163.../10.10.146.25    activating  start-pre
[email protected]        268251a3.../10.10.126.61    inactive    dead
[email protected]        4e283163.../10.10.146.25    inactive    dead
[email protected]        4e283163.../10.10.146.25    inactive    dead
[email protected]             268251a3.../10.10.126.61    activating  start-pre
[email protected]             4e283163.../10.10.146.25    activating  start-pre
[email protected]             4e283163.../10.10.146.25    activating  start-pre

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.