Add a hook for "before shutdown" about puma HOT 6 OPEN

@jmuia

Haven't had time to think about this, but one question:

it's a common pattern to fail healthcheck requests for N seconds prior to draining existing connections and closing the listener.

So, you want Puma to 'wait N seconds, then drain existing connections'? Is that delay needed? Also, often 'drain' used, but there are three types. Also, see docs for Puma::DSL#drain_on_shutdown

1. Backlog connections that haven't been accepted
2. Connections that have been accepted, but not completed the request
3. Connections that are 'keep-alive', and may have additional requests

from puma.

Yep, exactly – the pattern is effectively:

1. Receive SIGTERM
2. Continue processing new connections and requests. Only fail (e.g. 503) healthcheck requests.
3. After N seconds, close the listener and process any pending requests (let's connections that have been accepted).

The delay is needed to allow clients and load balancers to detect that this server shouldn't receive any more traffic.

DSL#drain_on_shutdown paired with checking Server#shutting_down? in the healthcheck handler (allowing us to fail healthcheck requests) does seem promising. I see two downsides:

1. For a low traffic server, the accept queue could be empty causing us to exit the accept loop. There could be a race condition with an incoming request.
2. A server will always receive healthcheck requests, which could mean that with drain_on_shutdown we never exit the accept loop (seems unlikely)

from puma.

Hmm, isn't it better to instruct your load balancers to take out the server of the rotation when you want to restart it? (Not sure what you're using but HAProxy seems to have a drain mode for this)

from puma.

Hmm, isn't it better to instruct your load balancers to take out the server of the rotation when you want to restart it?

This would be a good solution! Unfortunately, in this environment it's not an option. We're using client-side load balancers and real-time health information comes from client-side health check requests.

from puma.

Sounds cool… feel free to elaborate :)

from puma.

After some thinking, I could see another viable alternative: add a min/max drain time to Puma::DSL#drain_on_shutdown (in other words, continue the accept() loop until we've waited the minimum drain time and there are either no more connections or we've spent the max drain time).

Sounds cool… feel free to elaborate :)

Of course! We're using Envoy as a sidecar mesh proxy in this environment – it proxies requests on both the client and server hosts. In our current configuration, health checks are an important part of the "draining" process. Please let me know if there are other specifics you're interested.

from puma.