Recently it was observed that under a specific set of pre-conditions Apache's HttpClient could be left in a non functional state. This project is a simplified reproduction of the problem.

More specifically, if basic authentication is implemented in minimalistic way passing multiple CredentialsProvider instances through same context, socket timeouts may cause client to enter a state where it always returns '401 Unauthorized' response to any request to the same service.

This example/test/demo is using traditional blocking http client 4.5.13, but identical behavior was observed using asynchronous (non-blocking) apache http client 4.1.4 (not covered by code in this demo).

Future versions of apache http client (both blocking and non-blocking) should handle timeouts in such a way that socket timeouts cause only transient errors, without long lasting effects, even if its usage may be considered unconventional. If that is not possible, then usage of http client that could lead to this issue should fail fast.

• normally apache http client issues pairs of requests for endpoints protected by basic http authentication - first one comes either with no 'Authorization' header or with previously used credentials, and then after '401 Unauthorized' response http client sends second request that uses credentials generated by provided CredentialsProvider
• if api server always uses the same credentials, then after initial '401 Unauthorized' response all requests (made with the same context) will be authorized by default, simplifying request flow and eliminating issue observed
• if api server requires different credentials for different requests (lets say, endpoint's url contains account id and different accounts require different credentials), then quite often wrong (cached on the context) credentials will be sent to the server before client rights itself by actually asking CredentialsProvider for new credentials - this is somewhat unintuitive behavior, but it is documented in javadocs, and expected
• if first request from the pair of requests returns '401 Unauthorized' as it should, but next (properly authenticated) request times out, http client enters a strange 'damaged' state where for any following requests resulting in '401 Unauthorized' response they will NEVER be followed up by another request using provided credentials; because of this, one single network timeout can take out http client out of commision forever - it stays broken until JVM restart
• if it is first request from the pair of requests that times out instead of returning '401 Unauthorized' response, http client does not enter 'damaged' state and behaves properly during next request using the same context
• occasional http timeouts will take our some http clients from the usual pooled configuration, resulting in strange intermittent '401 Unauthorized' failures, that are difficult to troubleshoot and are easy to blame on bugs in api/server implementation; a spike of timeouts that can sometime happen takes down all available http clients, resulting in inability to communicate with api until JVM restart

• use new context for every request
• add code that forces apache client to perform preemptive authentication - every call to http client results in only one http request, and timeouts are handled properly
• reset AuthState after every http call - that ensures that fresh credentials are used for every request, and it also somehow ensures that timeouts are handled properly
• do not use CredentialsProvider at all and just insert 'Authorization' header directly, effectively implementing preemptive authentication

While workarounds are a fairly good idea all by themselves (if one deals with api that uses multiple credentials, it is good idea to use one of those approaches anyway), they are not exactly intuitive and may be left out by many developers unfamiliar with the details of apache http client library.

Default behavior in this case might be slightly inefficient (with half of http requests having either no credentials or wrong credentials) but it will still function as expected, until a particular sequence of timeouts happens, resulting in either partial or complete downtime for a service.

To build/run project, you should have maven installed, and you can just execute

./run.sh

This project contains embedded web server implementing required endpoint, so it is fairly self-contained, but you have to make sure that port 8089 is available, or change SERVER_PORT constant.

Test outputs main events to stdout, so it is fairly easy to follow what happens without resorting to a debugger.

There is a variable TRIGGER_TIMEOUT in file ApacheTimeoutExample.java, you can change it from 'true' to 'false', re-run the experiment, and see how it behaves under normal conditions (without timeout). There is also flag RESET_AUTH_STATE that allows resetting AuthState after every request, you can enable it and observe disappearance of the 'sticky' '401 Unauthorized' responses.

If more debugging output from apache http client is desired, modify some pre-existing setup in script run.sh (verbose logging is disabled by default).

• Twilio https://www.twilio.com/docs/usage/api