502 when under load about substra-backend HOT 5 CLOSED

Nginx retry

Note that in the example above, nginx tries hitting the backend twice. See the end of the second log line:

502, 400 be8dae30c7f82749a8b130ddf459875f

Explanation:

• nginx tries to hit the backend but the connection gets interrupted => 502
• nginx tries to hit the backend again, this time the call succeed => 400 (this is the correct, expected return code for this test)

I have no explanation as to why the request is retried. My understanding is that this shouldn't happen since there's only one backend server configured (confirmed with kubectl ingress-nginx backends -n kube-system)

from substra-backend.

I sometimes get "Bad Gateway" (502) errors when running the tests in substra-tests locally, it does not always happen to the same one I think.

My installation of Substra is with skaffold (see the installation instructions)

My setup is:

• macOS Catalina 10.15.5
• docker desktop community, 2.3.0.3 with kubernetes 1.16.5

from substra-backend.

This might be related to the fact that we currently use the cheaper algorithm.

Not sure, I added cheaper worker to solve this issue I was facing before adding them :(

from substra-backend.

I have no explanation as to why the request is retried. My understanding is that this shouldn't happen since there's only one backend server configured (confirmed with kubectl ingress-nginx backends -n kube-system)

This is a feature of nginx to retry after receiving some 500's error codes (doc: nginx#proxy_next_upstream) as long as they are not POST, PUT or something else that is not idempotent.
If you have only one pod and proxy-next-upstream-tries is set to 3 (default) it will try three times the same server (this is what I understood from this thread: kubernetes/ingress-nginx#4944), this could explain why you have a retry even if you only have one server running.
here in the default nginx-ingress code: https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/config/config.go#L784-L786 you can see it retries only on timeout.

from substra-backend.

Closing stale issue.

from substra-backend.