Feature: lifecycle management for zero downtime distributed hosting about reverse-proxy HOT 8 CLOSED

It seems like this has 3 parts to it:
a) Supporting graceful shutdown where clients are advised to connect to something else
b) Draining connections before actually shutting down - probably with a timeout
c) Eventing & signal model so that activating shutdown can tell the router to stop traffic, and when shutdown is complete, and the API to activate shutdown when told to by the router - assuming that app code will be listening to signals from the router.

from reverse-proxy.

The "Hosted" .NET app model (i.e. using Microsoft.Extensions.Hosting) on which ASP.NET Core is based has a graceful shutdown model and Kestrel participates in it already I believe. When shutdown is requested, Kestrel stops accepting new connections and drains existing ones (I'm not sure if it puts Connection: close in but that could be added). If a (configurable) timeout elapses, the server terminates more forcefully (yields back to Program.Main which will likely exit and terminate the process).

So I think we have (b) already and adding (a) is relatively straightforward.

For (c), we have the necessary event hooks (IApplicationLifetime.ApplicationStopping). I'm not sure there's a clear enough "pattern" that we could just implement in YARP though. I think custom code would be the primary way users would interact with that.

from reverse-proxy.

Triage: We do also have (a) it turns out.

Sounds like we need at least a sample (and maybe even a feature) for signalling health state based on IApplicationLifetime.

from reverse-proxy.

See the following as a follow on from c)
#125 - Having an API for querying and understanding proxy status
#124 - REST / gRPC endpoint to expose the data from the proxy
#123 - Dashboard showing that data

from reverse-proxy.

We achieve the first part (not ready state when starting/stopping/stopped) of c) in our services with a health check on IHostApplicationLifetime that tracks the status (starting/started/stopping/stopped) and use that in the readiness check.

The second part, removing from the upstream Load Balancer (LB) before rejecting new connections, is a bit more tricky. It either needs an active call from the service (and response from the LB) to remove the instance from rotation on the LB OR some knowledge about the configuration of health checks on the LB (ex. 3 failed checks with 10 sec. check interval) to know how long we need to report as unhealthy to the LB so it will have us removed before we proceed with Stopping and rejecting new connections.

from reverse-proxy.

Seems to be working for Island Gateway with a 3min shutdown for Kestrel (long duration due to gRPC streaming).
Should look at how to gracefully shutdown inner protocols such as:

• gRPC streaming. Eg grpc/grpc-go#2888
• Web sockets

from reverse-proxy.

Triage: We should be able to just see it working on Island Gateway prototype.
Sample may be useful for everyone

from reverse-proxy.

Triage: Scenario which didn't need any work, unclear what to do in sample. Closing.

from reverse-proxy.