Empty replies during grace/timeout period about graceful-server HOT 8 CLOSED

I can confirm that everything now works as intended, thanks @gquittet!

from graceful-server.

Hello @MichielDeMey

Happy to see another Belgian here 🇧🇪

Okay, I get the point.

Once the server crashes, it is no longer in the "ready" state.
This means that every new incoming request will be destroyed (hence the empty response).

• graceful-server/src/core/improvedServer.ts

  Line 39 in 7694570

  req.socket.destroy()

• graceful-server/src/core/improvedServer.ts

  Line 51 in 7694570

  req.socket.destroy()

The purpose of the timeout is to terminate ongoing requests, as well as to disconnect cleanly from all data sources.

So here during 30 seconds, the server will try to run the shutdown callback and wait ongoing requests to finish. The only working routes during this state are the liveness that send you a 200 and the readiness that send you a 503.

In the shutdown callback you don't necessarily have to turn off the server, the library takes care of that for you.

const gracefulServer = GracefulServer(fastify.server, {
  timeout: 30000,
  closePromises: [fastify.close]
})

What I can do is to create a new timeout that will give more time for incoming traffic.

The health check endpoints are handled by NodeJS itself (thanks to graceful server).

If you want to change your readiness by /foo, you can do it.

const gracefulServer = GracefulServer(fastify.server, {
  readinessEndpoint: '/foo',
  timeout: 30000,
  closePromises: [fastify.close]
})

I hope I have been as clear as possible.

from graceful-server.

Hi Guillaume

Thanks for the clarifications.

I think adding an additional timeout to allow for incoming requests to be handled is a good solution since that is exactly what I need.

It's not that the service is crashing, but it receives a SIGTERM from kubernetes asking to cleanly shutdown the service.
However, when coupled with ingress controllers (especially cloud load balancers) we still need to be able to serve traffic to those pods while simultaneously serving a 5xx status to the readiness endpoint.

De-registering the pod from the NEG for the load balancer takes some time and we need to be able to serve traffic for that period so we will not end up with client-side errors. 🙂

There's also some information regarding this behaviour in the Terminus README.
https://github.com/godaddy/terminus#how-to-set-terminus-up-with-kubernetes

from graceful-server.

Hello @MichielDeMey,

I just released the version 2.4.0 of graceful-server 🚀

You can now set up a period of grace with gracePeriod as you can read it in the documentation : https://github.com/gquittet/graceful-server#igracefulserveroptions

const gracefulServer = GracefulServer(fastify.server, {
  timeout: 30000,
  gracePeriod: 5
})

I just test it and everything is running fine here locally.

Can you retry with your Kube cluster to check if everything is running fine ?

Thank you ! 🙏

from graceful-server.

Hello @MichielDeMey

Without any answer from you, I'll have to close this issue tomorrow as the fix is working on my side.

Have a nice day 👋

from graceful-server.

Hey @gquittet , sorry for the late reply!

I just updated the dependency and checked again.
It does not look like the implementation is what we need for the graceful timeout to work for cloud ingress controllers.

I've used the following config (same server as before):

const gracefulServer = GracefulServer(fastify.server, {
  timeout: 5000,
  gracePeriod: 10
})

I then have 3 simple curl commands running in the background to check the API behaviour.

while sleep 1; do curl http://127.0.0.1:8080/live -vvv; done       # liveness endpoint
while sleep 1; do curl http://127.0.0.1:8080/ready -vvv; done      # readiness probe
while sleep 1; do curl http://127.0.0.1:8080/foo -vvv; done        # example endpoint, returns 200

When I send a SIGTERM to the server kill -SIGTERM 1234 (where 1234 is the pid) the following happens for the next 10 seconds:

/live: 200
/ready 200
/foo 200

After 10 seconds (graceful period), the following is returned for the next 5 seconds (timeout):

/live: 200
/ready 503
/foo connection refused

What should be happening is that the readiness endpoint returns a 503 as soon as the SIGTERM is received.
Any other API endpoints should still respond with their original response for the duration of the graceful period.

As soon as the SIGTERM is received for the next 10 seconds (graceful period):

/live: 200
/ready 503
/foo 200

This signals the loadbalancer that the endpoint is no longer ready and should be de-registered from the loadbalancer backend config.

The process of de-registering an endpoint might take some time, depending on the cloud provider and health check settings (interval, unhealthy threshold, etc.) hence the graceful period.

The key here is that all API endpoints (except the readiness route) should still serve traffic as normal when a SIGTERM is received and the graceful period is running.

I hope this makes sense. 🙂

from graceful-server.

Hello @MichielDeMey,

Glad to hear from you.
I hope you are well.

Your explanations are very clear and make sense.

I'm doing this tonight so the fix will be available in the morning 🙂

from graceful-server.

I just added a Kubernetes mode to let it manages the incoming traffic 🎉 (#11)

I removed the grace period because it is no longer useful. (#13)

All of these improvements are in the 2.5.0 version.

How to use it?

Just set the Kubernetes mode to true in the @gquittet/graceful-server option.

const gracefulServer = GracefulServer(fastify.server, {
  timeout: 5000,
  kubernetes: true
})

All is working as expected on my side 💪

I reopen this issue for a period of 14 days.

from graceful-server.