Comments (3)
After going through clj-http
code, I write a sample java version to compare the results.
public class Main {
public static void main(String[] args) throws IOReactorException {
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(1)
.setSocketTimeout(500)
.setRedirectsEnabled(true)
.build();
HttpUriRequest httpUriRequest = new HttpGet("http://255.255.33.44:8001/");
httpUriRequest.addHeader("Connection", "close");
IOReactorConfig reactorConfig = IOReactorConfig.custom()
// .setConnectTimeout(500)
// .setSoTimeout(500)
.setShutdownGracePeriod(1)
.build();
DefaultConnectingIOReactor defaultConnectingIOReactor = new DefaultConnectingIOReactor(reactorConfig);
Registry registry = RegistryBuilder.create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.build();
PoolingNHttpClientConnectionManager connectionManager = new PoolingNHttpClientConnectionManager(defaultConnectingIOReactor, registry);
HttpClientContext context = new HttpClientContext();
context.setRequestConfig(requestConfig);
connectionManager.setMaxTotal(1);
CloseableHttpAsyncClient client = HttpAsyncClients.custom()
.setConnectionManager(connectionManager)
.build();
client.start();
long startTime = System.nanoTime();
client.execute(httpUriRequest, context, new FutureCallback<HttpResponse>() {
@Override
public void completed(HttpResponse httpResponse) {
StatusLine statusLine = httpResponse.getStatusLine();
int statusCode = statusLine.getStatusCode();
long stopTime = System.nanoTime();
System.out.print("=====completed, timeElapsed: ");
System.out.print(TimeUnit.MILLISECONDS.convert(System.nanoTime() - startTime, TimeUnit.NANOSECONDS));
System.out.println(" ms");
System.out.print(statusCode);
}
@Override
public void failed(Exception e) {
System.out.print("=====failed, timeElapsed: ");
System.out.print(TimeUnit.MILLISECONDS.convert(System.nanoTime() - startTime, TimeUnit.NANOSECONDS));
System.out.println(" ms");
System.out.print("=====error msg: " + e.getMessage());
try {
connectionManager.shutdown();
} catch (IOException ex) {
ex.printStackTrace();
}
}
@Override
public void cancelled() {
httpUriRequest.abort();
try {
connectionManager.shutdown();
} catch (IOException e) {
e.printStackTrace();
}
}
});
}
}
When setConnectTimeout(1)
, the time elapsed is ~1129ms.
When setConnectTimeout(500)
, the time elapsed is ~1142ms.
When setConnectTimeout(1000)
, the time elapsed is ~1115ms.
When setConnectTimeout(1100)
, the time elapsed is ~2099ms.
When setConnectTimeout(1200)
, the time elapsed is ~2106ms.
When setConnectTimeout(1300)
, the time elapsed is ~2091ms.
When setConnectTimeout(1500)
, the time elapsed is ~2095ms.
When setConnectTimeout(2000)
, the time elapsed is ~2110ms.
When setConnectTimeout(2500)
, the time elapsed is ~3121ms.
When setConnectTimeout(3000)
, the time elapsed is ~3098ms.
When setConnectTimeout(4000)
, the time elapsed is ~4117ms.
When setConnectTimeout(5000)
, the time elapsed is ~5110ms.
So it seems like code execute time is about ~1000ms, it takes at least 1 second from the start of the request to fail if the connect timeout is less than ~1000ms.
But synchronous requests do not have such high latency (interval from request to failure), I guess because asynchronous requests require a lot of setup code to be processed, including the creation of connection pools, so the creation of asynchronous request clients is time-consuming expensively.
from clj-http.
After some research, I figure out a way to solve my problem which used clojure.core.async
to implement the HTTP client async feature.
Here is the main code(gist):
;; lein repl
(ns test (:require [clojure.core.async :as async]
[clj-http.client :as client]))
;; define test data
(def urls ["http://www.google.com" "http://www.google1.com" "http://255.255.33.44:8001/"])
;; use async/go to build a async http client
(defn fetch-async
[url]
(let [ret (async/promise-chan)]
(async/go
(try
(let [res (client/get url {:socket-timeout 500
:connection-timeout 500})]
(async/put! ret {:result :success :msg (:status res)}))
(catch Exception e (async/put! ret {:result :error :msg (.getMessage e)}))))
ret))
;; use async/go to build a async http client but disable apache http client retry feature
(defn fetch-async-without-retry
[url]
(let [ret (async/promise-chan)]
(async/go
(try
(let [res (client/get url {:socket-timeout 500
:connection-timeout 1000
:retry-handler (fn [ex try-count http-context]
(if (> try-count 0) false true))})]
(async/put! ret {:result :success :msg (:status res)}))
(catch Exception e (async/put! ret {:result :error :msg (.getMessage e)}))))
ret))
;; test fetach a domain url
(time (async/<!! (fetch-async "http://www.google1.com")))
;; test fetach an ip url
(time (async/<!! (fetch-async "http://255.255.33.44:8001/")))
;; block main thread to get all response data
(let [res-promise (map #(fetch-async %) urls)]
(map #(async/<!! %) res-promise))
;; metric the execute time
(let [res-promise (map #(fetch-async %) urls)]
(map #(time (async/<!! %)) res-promise))
(time (let [res-promise (map #(fetch-async %) urls)]
(map #(async/<!! %) res-promise)))
(let [res-promise (map #(fetch-async %) urls)]
(time (map #(async/<!! %) res-promise)))
and the result is good.
"Elapsed time: 470.619411 msecs" ;; http://www.google.com
"Elapsed time: 882.209614 msecs" ;; http://www.google1.com, this is caused by apache HTTP client retry feature, if you disable it, the time is 0.020235 msecs
"Elapsed time: 0.018817 msecs" ;; http://255.255.33.44:8001/
From the above test results, we can see that after the first request takes 500ms (which is the connect-timeout we set), the other requests are executed asynchronously and the total time is almost 500ms at the end. Moreover, the async/go process is very lightweight and does not consume as many resources and execution time as a thread does.
from clj-http.
Note that here:
(async/go
(try
(let [res (client/get url {:socket-timeout 500
:connection-timeout 1000
:retry-handler (fn [ex try-count http-context]
(if (> try-count 0) false true))})]
(async/put! ret {:result :success :msg (:status res)}))****
you're doing a blocking call (client/get
) in the async/go
block which ties up the underlying thread. Quoting from the async/go
docstring:
go blocks should not (either directly or indirectly) perform operations
that may block indefinitely. Doing so risks depleting the fixed pool of
go block threads, causing all go block processing to stop. This includes
core.async blocking ops (those ending in !!) and other blocking IO.
from clj-http.
Related Issues (20)
- The word 'officially' is spelt wrong in the GitHub description HOT 1
- HEAD requests shouldn't discard body (in the request).
- Handling of private proxies HOT 2
- Whenever I get a 405 response from a server, a null pointer exception is thrown because the connection wasn't found HOT 1
- Nested query params
- cookie-policy none
- URL encoding of whitespace should be "%20" instead of "+" in query parameters HOT 1
- Calling `.close` on response http stream will fully read stream contents rather than terminating. HOT 2
- `:query-params` are re-ordered
- :threads is mis-named and mis-documented
- multipart posts: sending name requires a mime-type
- Async requests not really async
- AOT issues with clj_http.headers.HeaderMap HOT 4
- url path parameters HOT 2
- The :redirect-strategy doesn't seem to work for PUT and PATCH HTTP methods
- option to pre-process `:query-params` HOT 2
- Retry when service is (tempoarily) unavailable HOT 3
- `with-middleware` rebinds `http/request` with new core client
- Support async with socks proxy?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clj-http.