A few days ago I noticed https://httpbin.dmuth.org/ started hanging for no reason. My dashboards would look like this:

...and I started seeing errors like these in the logs from fly.io:

could not find a good candidate within 90 attempts at load balancing. last error: no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)

I then SSHed into the instance and saw that uvicorn was using 100% of the CPU.

I also poked around in /proc/ and saw that there was only about a dozen file descriptors open, so it's not a resource exhaustion issue.

I tried the following things so far, but have been unable to resolve it:

• ✅ Restarting the VM
• ✅ Changing the count of machines with the fly scale command to 0 and then 1 to spin up a new machine
• ✅ Running fly deploy again
• ✅ Turning off Fly's raw TCP check, thinking it was tripping up Uvicorn somehow.

I am continuing to investigate, and have a few other things to try:

• ✅ Turning off the HTTP check from Fly.io
• ✅ Adjusting the URLs that NodePing is hitting
• ✅ Upgrading FastAPI to the latest version and redeploying (this is in progress)
• ✅ Increase the number of workers to 3
• Seeing if I can capture log output from Uvicorn by setting an environment variable.
• Changing the server to Hypercorn

If I run this on the command line, I get a redirect to /redirect/1:

curl -X 'GET' \
  'https://httpbin.dmuth.org/status/301' \
  -H 'accept: application/json'

Okay, cool! The issues is that when I do that on the Swagger page, the redirects are followed, ending in a 200 page, which can confuse the user.

I think I should to three things:

1. Add a blurb in the Swagger docs warning about 3xx codes.

2. Have a parameter for /redirect/{n} called status, which contains the original status code and passes it along. For example, /status/309 would redirect to /redirect/1?status=309, and then the final message would include that status code in message and status.

3. Have an endpoint for /redirect, which says end of redirects with /redirect/1 pointing there. It should show the message and code that was passed in.

The spec file at /openapi.json appears to have some problems when loaded from the latest image at docker.io/dmuth1/fastapi-httpbin:latest. It fails to validate in an enforcement tool and editor.swagger.io reports the following:

Errors:

Semantic error at paths./cookies.delete.requestBody
DELETE operations cannot have a requestBody.
Jump to line 1265
Structural error at components.schemas.QRCode.properties.box_size.exclusiveMinimum
should be boolean
Jump to line 1849
Structural error at components.schemas.QRCode.properties.border.exclusiveMinimum
should be boolean
Jump to line 1856

It appears that, unlike httpbin, fastapi-httpbin consistently interprets a urlencode payload in a POST request as JSON.

When a POST request is sent to httpbin, with the following command:

$ curl -v -d "birthyear=1905&press=%20OK%20" http://www.httpbin.org/post

curl automatically sets the Content-Type header to application/x-www-form-urlencoded, as its default behavior. httpbin responds with the following:

"form": {
    "birthyear": "1905",
    "press": " OK "
},
"json": null

However, if the Content-Type is specified as application/json:

$ curl -v -H "content-type: application/json" -d '{"birthyear": 1905, "press": " OK "}' http://www.httpbin.org/post

The response now includes a populated "json" field:

"json": {
    "birthyear": 1905,
    "press": " OK "
},

In contrast, it appears that fastapi-httpbin always expects JSON content, even when the Content-Type is set to application/x-www-form-urlencoded:

$ curl -v -d "birthyear=1905&press=%20OK%20" https://httpbin.dmuth.org/post

This request produces an error message in the "data" field:

"data": {
    "message": "No JSON/bad JSON supplied.  If you used Swagger, you'll need to use curl on the CLI with the -d option instead for non-GET methods, or GET-method data for GET."
}

If the Content-Type is set to application/json however, it works as expected:

$ curl -v -H "content-type: application/json" -d '{"birthyear": 1905, "press": " OK "}' https://httpbin.dmuth.org/post

The response from fastapi-httpbin then successfully populates the "data" field with the submitted form data:

"data": {
    "birthyear": 1905,
    "press": " OK "
}

I'm uncertain whether the observed behavior with content-type: application/x-www-form-urlencoded, which differs from httpbin, is an intended feature.