orthecreedence / carrier Goto Github PK
View Code? Open in Web Editor NEWA lightweight, async HTTP client
carrier's People
Contributors
Stargazers
Watchers
Forkers
nightshade427 munro fiddlerwoaroof satran sudodoki davidalphafox akanouras peoplestom kchanqvqcarrier's Issues
ssl causes faults on sbcl
There seems to be an issue where if you ssl uris with carrier and lastest cl-async it cases memory faults and other issues.
This works:
(as:with-event-loop ()
(asf:alet ((result (carrier:request "http://www.apple.com/" :return-body t)))
(format t "result: ~s" result))) This one explodes:
(as:with-event-loop ()
(asf:alet ((result (carrier:request "https://www.apple.com/" :return-body t)))
(format t "result: ~s" result)))Reusing SSL Context
I'm using carrier to send hundreds of requests per second (I hunt for an async library just for this!). However, I only get throughput around ~50 req/s. After some profiling I find that the majority of time is spent in the call ssl-ctx-set-default-verify-paths inside cl-async-ssl::tcp-ssl-connect-new. I hacked carrier:request a bit to pass through a :ssl-ctx option, and initialize a ssl-ctx outside my main loop. This increases the throughput to ~1000 req/s.
I wonder if we can implement an official support for that? My current solution feels hacky, I have to use internal symbols in cl-async-ssl to create ssl-ctx, which seems to be transparent to user by design. Any ideas?
When using chunked responses carrier returns only the first chunked response
When using chunked responses carrier returns only the first chunked response, in the following pseudo script:
- make request using carrier
- get chunked response back from uri server
- carrier reads the first chunked data and sends to fast-http to process the response
- carrier gets "our-body-callback" called with the parsed response from fast-http
- carrier reads the next chunk data and sends to fast-http to process the response
- carrier gets "our-finish-callback" called and not "our-body-callback"
- carrier closes the connection and return the processed body so far (which is the first chunk only)
- calling code gets partial response
Not sure if issue is in carrier or in fast-http. I will try and look further. Any ideas?
Any reasons not to use this for AWS SDK?
I forked aws-sdk-lisp, updated the api definitions, and have started imparting all of my terrible opinions on it since it hasn't been updated in awhile.
One of my opinions is that it should be async by default, since most AWS api interactions are completely IO-bound. I will lose connection reuse and multi-part upload out of the gate; however, I can work on those later. Are there any reasons that you know of that I shouldn't use carrier? I noticed that you had s3 and dynamo drivers in your list of future work. This should theoretically take care of those I think?
Thanks in advance for any info you can provide!
Allow for streaming request body
It would be useful to be able to stream the request body for sending large files via HTTP POST/PUT.
Perhaps we could also wrap this with a couple macros modeled after with-output-to-string i.e.: with-input-from-response and with-output-to-request? I'm not sure how tricky it'd be to make such a model asynchronous in a transparent way :)
Issue with `location` in response headers being a relative path
For some sites, that has redirects %host%/%path%/ to %host%/%path%, and location is set to just /%path%, carrier:request would fail.
Sample (CARRIER:REQUEST "http://www.bbc.com/persian/" :RETURN-BODY T)
Would return
#<BLACKBIRD-BASE:PROMISE
finished: NIL
errored: T
forward: NIL {1002EFF063}>```
Additional output would indicate that
Callback Error: the headers-complete callback failed
NIL is not a Lisp string or pointer.
Issue would seem to originate from 'location' not having specified host.
Certain SSL sites don't work
While carrier works with https://en.wikipedia.org/wiki/Lisp_(programming_language), this fails:
(cl-async:with-event-loop ()
(blackbird:catcher (blackbird:attach (carrier:request "https://api.ipify.org/" :return-body t)
(lambda (r c h)
(format t "~&RESPONSE: ~A~%CODE: ~A~%HEADERS: ~A~%"
(babel:octets-to-string r) c (and h (alexandria:hash-table-alist h)))))
(t (c) (format t "~&ERROR: ~A~%" c))))
Producing the output:
RESPONSE: #()
CODE: 0
HEADERS: NIL
0
Also, it might be useful to have some way of viewing the headers being sent and received? I tried (vom:config t :debug) but that only displayed information about cl-async-ssl and blackbird.
<DEBUG> [11:28:50] cl-async-ssl - < write: buffer: 39
<DEBUG> [11:28:50] blackbird-base - create-promise: #<PROMISE finished: NIL errored: NIL forward: NIL {1006196343}>
<DEBUG> [11:28:50] blackbird-base - promisify: #<PROMISE finished: NIL errored: NIL forward: NIL {1006196343}>
<DEBUG> [11:28:50] blackbird-base - promisify: #<PROMISE name: "attach: (CARRIER:REQUEST \"https://api.ipify.org/\" :RETURN-BODY T)" finished: NIL errored: NIL forward: NIL {100619C0E3}>
<DEBUG> [11:28:50] blackbird-base - promisify: #<PROMISE name: "attach: (CARRIER:REQUEST \"https://api.ipify.org/\" :RETURN-BODY T)" finished: NIL errored: NIL forward: NIL {100619C0E3}>
<DEBUG> [11:28:50] blackbird-base - create-promise: #<PROMISE finished: NIL errored: NIL forward: NIL {100619D8A3}>
<DEBUG> [11:28:50] cl-async-ssl - . info: before/connect initialization (16 1)
<DEBUG> [11:28:50] cl-async-ssl - . info: before/connect initialization (4097 1)
<DEBUG> [11:28:50] cl-async-ssl - . info: unknown state (4097 1)
<DEBUG> [11:28:50] cl-async-ssl - . info: unknown state (4098 -1)
<DEBUG> [11:28:50] cl-async-ssl - * exit
<DEBUG> [11:28:50] cl-async-ssl - < write: raw: 155 (conn T)
<DEBUG> [11:28:50] cl-async-ssl - . info: unknown state (4098 -1)
<DEBUG> [11:28:50] cl-async-ssl - * exit
<DEBUG> [11:28:50] cl-async-ssl - < write: raw: 155 (conn T)
<DEBUG> [11:28:50] cl-async-ssl - > read: raw: 65536
<DEBUG> [11:28:50] cl-async-ssl - . info: unknown state (16388 592)
<DEBUG> [11:28:50] cl-async-ssl - . info: unknown state (4098 0)
<DEBUG> [11:28:50] cl-async-ssl - * exit
<DEBUG> [11:28:50] cl-async-ssl - . info: unknown state (4098 -1)
<DEBUG> [11:28:50] cl-async-ssl - * exit
<DEBUG> [11:28:50] blackbird-base - finish: #<PROMISE finished: NIL errored: NIL forward: NIL {1006196343}> (#() 0 NIL)
<DEBUG> [11:28:50] blackbird-base - finish: #<PROMISE name: "attach: (CARRIER:REQUEST \"https://api.ipify.org/\" :RETURN-BODY T)" finished: NIL errored: NIL forward: NIL {100619C0E3}> (NIL)
<DEBUG> [11:28:50] blackbird-base - finish: #<PROMISE finished: NIL errored: NIL forward: NIL {100619D8A3}> (NIL)
<DEBUG> [11:28:50] blackbird-base - finish: #<PROMISE name: "attach: PROMISE" finished: NIL errored: NIL forward: NIL {100619F1F3}> (#<BLACKBIRD-BASE:PROMISE
finished: T
errored: NIL
forward: NIL {100619D8A3}>)
<DEBUG> [11:28:50] blackbird-base - rejecting an already-resolved promise: #<PROMISE finished: T errored: NIL forward: NIL {1006196343}>
HTTP request returning nil
Running this:
(cl-async:with-event-loop ()
(blackbird:attach (carrier:request "http://api.ipify.org")
(lambda (r c h)
(format t "~&data: ~a~%code: ~a~%headers: ~s"
r c
(alexandria:hash-table-alist h)))))
produces the output:
data: NIL
code: 200
headers: (("via" . "1.1 vegur") ("content-length" . 12) ("date" . "Mon, 29 Aug 2016 18:54:32 GMT") ("content-type" . "text/plain")
("connection" . "keep-alive") ("server" . "Cowboy"))
0
Rather than printing data: <my ip address>. It looks to me like the body of the http request isn't being handled correctly.
cookie support
I think the last thing I need to get off drakma is cookie support. Any plans to support this?
Multiple issues when using latest quicklisp and carrier master
I dont think carrier was brought up to date with latest cl-async/blackbird stuff and is causing issues in latest ql.
(ql:quickload "carrier")When using normal http
CL-USER> (as:with-event-loop ()
(bb:catcher
(bb:alet ((result (carrier:request "http://www.apple.com/" :return-body t)))
(format t "result: ~s" result))
(t (e) (format t "error: ~a" e))))
WARNING:
Passing event-cb as the fourth argument to tcp-connect is now deprecated. Please use the :event-cb keyword instead.
error: odd number of &KEY arguments When using https
CL-USER> (as:with-event-loop ()
(bb:catcher
(bb:alet ((result (carrier:request "https://www.apple.com/" :return-body t)))
(format t "result: ~s" result))
(t (e) (format t "error: ~a" e))))
error: odd number of &KEY argumentshangs when cannot connect
This hangs:
CL-USER> (as:with-event-loop ()
(bb:catcher
(bb:alet ((result (carrier:request "https://www.apple.com:999/" :return-body t)))
(format t "result: ~s" result))
(t (e) (format t "error: ~a" e))))Multiple issues when using latest quicklisp and carrier master
When using normal http
CL-USER> (as:with-event-loop ()
(bb:catcher
(bb:alet ((result (carrier:request "http://www.apple.com/" :return-body t)))
(format t "result: ~s" result))
(t (e) (format t "error: ~a" e))))
WARNING:
Passing event-cb as the fourth argument to tcp-connect is now deprecated. Please use the :event-cb keyword instead.
error: odd number of &KEY arguments When using https
CL-USER> (as:with-event-loop ()
(bb:catcher
(bb:alet ((result (carrier:request "https://www.apple.com/" :return-body t)))
(format t "result: ~s" result))
(t (e) (format t "error: ~a" e))))
error: odd number of &KEY argumentsssl uri sometime work sometimes dont
I'm using latest master of cl-async, carrier, balckbird, etc. If I do:
CL-USER> (as:with-event-loop (:catch-app-errors t)
(bb:catcher
(bb:multiple-promise-bind (body status headers) (carrier:request "http://s3.amazonaws.com/smart_engine/staging%2Fimports%2F68d7698667f5cb51ac2b73512fd2fe09%2Fstaging\
-imports-bd1566efd68419a547ba06dbbdb3fada-669635%2BRevB%2BSOVEE%2B%25284%2529%2B%282%29.docx" :return-body t)
(format t "~&status: ~s~&headers: ~s~&body: ~s~&" status (alexandria:hash-table-plist headers) (length body)))
(t (e) (format t "error: ~a" e))))
status: 200
headers: ("server" "AmazonS3" "content-length" 183767 "content-type"
"application/vnd.openxmlformats-officedocument.wordprocessingml.document"
"accept-ranges" "bytes" "etag" "\"7c18d605d0608d364a5ddf107df63127\""
"last-modified" "Fri, 27 Mar 2015 17:45:20 GMT" "date"
"Fri, 27 Mar 2015 19:15:43 GMT" "x-amz-request-id" "27A871B1A5837876"
"x-amz-id-2"
"rfnzv7qqE8eeaut9OPdlwqPFEHZka7rKe8XjvGkSHgKlEA8VptuWz+Exvrl9wwKqsImgauJ+SlM=")
body: 183767CL-USER> (as:with-event-loop (:catch-app-errors t)
(bb:catcher
(bb:multiple-promise-bind (body status headers) (carrier:request "https://s3.amazonaws.com/smart_engine/staging%2Fimports%2F68d7698667f5cb51ac2b73512fd2fe09%2Fstagin\
g-imports-bd1566efd68419a547ba06dbbdb3fada-669635%2BRevB%2BSOVEE%2B%25284%2529%2B%282%29.docx" :return-body t)
(format t "~&status: ~s~&headers: ~s~&body: ~s~&" status (alexandria:hash-table-plist headers) (length body)))
(t (e) (format t "error: ~a" e))))
status: 200
headers: ("server" "AmazonS3" "content-length" 183767 "content-type"
"application/vnd.openxmlformats-officedocument.wordprocessingml.document"
"accept-ranges" "bytes" "etag" "\"7c18d605d0608d364a5ddf107df63127\""
"last-modified" "Fri, 27 Mar 2015 17:45:20 GMT" "date"
"Fri, 27 Mar 2015 19:17:18 GMT" "x-amz-request-id" "DE1C47093C61E543"
"x-amz-id-2"
"RlrhpjToxsj3iQlud1mnZ4X/AeUtAAQWVcpLYG5tHM1EvFOuRPqve93g4Wt3i6bBTCp5rJec7Io=")
body: 137819 The only difference is ssl. With ssl it doesnt seem to get the whole content all the time. Sometimes it does sometimes it doesnt, which makes it all the harder to track down.
errors with latest quicklisp
carrier errors on latest quicklisp for april due to incompatability with new fast-http
Unbound variable and function
Both 'future' and 'finish' are unbound for me in https://github.com/orthecreedence/carrier/blob/master/carrier.lisp#L87
Test against httpbin
When I get around to writing tests, would be cool to test against this app.
Parses urls incorrectly
I think this one actually is a bug :)
This code:
(cl-async:with-event-loop ()
(blackbird:attach (carrier:request "http://api.ipify.org")
(lambda (r c h)
(format t "~&data: ~a~%code: ~a~%headers: ~s"
r c
(alexandria:hash-table-alist h)))))
Fails with this result:
data: NIL
code: 400
headers: (("via" . "1.1 vegur") ("connection" . "close") ("date" . "Mon, 29 Aug 2016 19:56:06 GMT") ("server" . "Cowboy"))
0
Using tcpdump, I see that carrier is making this request:
GET NIL HTTP/1.1.
Host: api.ipify.org.
Shouldn't it be defaulting to "/" if the url doesn't have a path part?
Doesnt handle query strings
When making an http call with query strings carrier drops the query string part and only uses the path.
Carrier sends:
"/about?test=this" => "/about"
Can we add this project to quicklisp?
Can we add this project to quicklisp?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.