httprb / http Goto Github PK
View Code? Open in Web Editor NEWHTTP (The Gem! a.k.a. http.rb) - a fast Ruby HTTP client with a chainable API, streaming support, and timeouts
License: MIT License
HTTP (The Gem! a.k.a. http.rb) - a fast Ruby HTTP client with a chainable API, streaming support, and timeouts
License: MIT License
In lots of cases streaming of body is not needed, so it's better to flush socket at the same time headers were received, thus underlying connection is closed for sure.
@tarcieri I believe it worth to release v0.5.1
with latest state of 0-5-release
and drop 0.5.x
support. So, 0.6.1
will become a pivot of release-0.6
branch (for backports and support and such) and master will be able to merge 0.7
changes :D like deprecations and so on.
Using the awesome http gem for parsing response from a POST request. Response is a simple JSON, ~350 bytes. Response doesn't include the Content-Length header, which is not mandatory for normal HTTP communication. Digging more in the code, problem seems to be in the body/chunk parsing code in https://github.com/tarcieri/http/blob/master/lib/http/client.rb. Here's what I am seeing
I saw a couple of open issues related to the same block of code, so the logic might have to be revisited or broken down even more.
The calling of a GET, with a chained response, gives the no 'response' method for String. Something like that:
get = Http.get("http://www.google.com").response
Any ideas on this? I've checked the unit tests and this should work normally.
https://github.com/tarcieri/http/blob/master/lib/http/response/body.rb#L17 calls the Client#read method, which doesn't exist. Would submit a pull request, but not sure if Body#read should be deleted or updated to used readpartial or something else.
… because it currently doesn't, and that seems to be the main celling point (sorry), for this gem.
The current API is probably overly simplified:
body = HTTP.get(url)
To get the response object, we have to do this:
response = HTTP.get(url).response
Really it should probably be:
response = HTTP.get(url)
body = response.body
It'd be nice if body were an HTTP::ResponseBody
object as well, then we could support streaming like this:
body = HTTP.get(url).body
chunk1 = body.readpartial
chunk2 = body.readpartial
We could also make it Enumerable so you can use it like this if you prefer:
body = HTTP.get(url).body
body.each do |chunk|
...
end
At the moment we implicitly set Host
header of request in case it wasn't provided:
@headers['Host'] ||= @uri.host
I tend to think we should enforce it instead. The only use case that might want current (implicit behavior) is pretty weird IMHO:
HTTP[:host => 'example.com'].get('http://127.0.0.1:8080/wtf?')
This is the only warning I get when I run rails c
in my whole rails app. Might this be fixable or silencable?
/Users/bat/.rvm/gems/ruby-1.9.3-p194@my_rails_app/gems/http_parser.rb-0.5.3/lib/http_parser.rb:4: warning: already initialized constant Http
Hi -- I love the library.
What do you think about putting the POST vars hash at the top level instead of pointing at it with :form
? There's nothing else in that options hash anyway, and also the word "form" kind of bothers me, since that's device-specific and in this case it's clearly not a form :)
Http.post "http://example.com/resource", {:foo => "42"}
Trying to connect to https
resp = Http.with_headers(:accept => 'application/json').post "https://api.polldaddy.com", :body => JSON(req)
I'm getting this error: OpenSSL::SSL::SSLSocket:Class
Any thoughts?
Thanks!
This seems impossible.. I have a remote app generating 2 Set-Cookie lines and another device inline inserting a third Set-Cookie line after, only the third pops into the headers hash, I need the first two.
Looks like this requires a patch to http_parser.rb to do, which was offered as a pull for a few years but never accepted. The underlying C parses out cookies into their own object.
This is an Issue(tm) for people trying to integrate into apps (like Oracle business apps) that use cookies for things they have no business using cookies for but it's unavoidable. So I guess I'm reporting this as a warning?
Seeing this behavior on both 0.5.0 and HEAD:
logger.info "HTTP.send(#{method},#{url},#{opts})"
#HTTP.send(get,http://127.0.0.1:3000/api/v1/agent_status/maximum_calls,{:params=>{:prospect_type=>:appointment}})
request = HTTP.accept(:json).with_response(:object).send(method, url, opts)
logger.info "request = #{request.inspect}"
#17:56:22 ahn.1 | [2013-12-09 17:56:22] INFO TyphoeusHelper::SharedHydra: request = #<HTTP::Response/1.1 200 OK @headers={"Content-Type"=>"application/json; charset=utf-8", "Vary"=>"Accept-Encoding", "X-Ua-Compatible"=>"IE=Edge", "Etag"=>"\"64108005385238b566cf18d4efaa73f2\"", "Cache-Control"=>"max-age=0, private, must-revalidate", "X-Request-Id"=>"930be09be03d13110e09b6e9c0cae0c8", "X-Runtime"=>"0.058730", "Connection"=>"close", "Server"=>"thin 1.5.1 codename Straight Razor"}>
logger.info "body = #{request.body}"
#EOFError: end of file reached
(Don't mind the Hyrda stuff, switching from Typheous haven't renamed the classes yet)
This is the current gem description:
HTTP so awesome it will lure Catherine Zeta Jones into your unicorn petting zoo
In my experience, this is both inaccurate and vaguely sexist/heteronormative.
I’d like to see this changed before the v0.6 release.
I'm not sure if something like this fits well streamed response bodies, but I thought it might be nice to be able to write something like this:
body = HTTP
.accepts(json: proc { |body| JSON.parse(body) })
.get("https://www.somewhere.com/index")
.body
body["people"].each do |person|
# ..
end
We're punting on keepalive and connection pooling (#72) for 0.6. We should ensure that sockets are closed correctly request completes (i.e. that we implement Connection: close
semantics correctly)
I’m trying to perform a multipart POST
with the HTTP gem but I’m having trouble getting the headers and body right. We should include an example of how to do this in the README
.
A broken implementation of proxy support was removed in #62.
I’ve added this issue to the version v0.6 milestone. IMHO, it is not a blocker for v0.6, since it was never properly working, but is a blocker for v1.0.0. Better to implement it sooner rather than later to ensure we get the API right.
I'm using Ruby 2.0.0 and HTTP 0.5.0 and using this tiny little example:
require 'http'
response = HTTP.via('192.168.1.1513', 8888).get("http://www.google.co.nz").response
puts response.body
The request goes through just fine but in reality I have no HTTP proxy sitting on that address or port. I've tried it with my real web debugger address/port as well but it still goes through.
Is there a special way to enable proxies? I should note I do have Celluloid/IO installed on my machine if it makes any difference.
See #43
If the proxy API isn't working it should be removed. We should either fix it or remove it.
If it's removed it can be placed onto a branch where someone can fix it later if they so desire.
Would be nice to have some form of Response caching here, with a simple Hash Store and Memcache API Store, seeing how Github itself doesn't count successfull Conditional Gets towards the API limit.
You should definetely make a release, there is such a large gap between 0.5.0 and master it looks like a completely different library, even if there are issues left in master it is worth pushing a release out I think since 0.5.0 has issues of its own anyway xD
is there any major issue preventing a release ?
Would be nice if Http would follow redirects, whether by default or as an option.
➔ irb > require 'http' => true > HTTP.get 'https://www.google.com/' OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:799:in `connect' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:799:in `block in connect' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/timeout.rb:54:in `timeout' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/timeout.rb:99:in `timeout' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:799:in `connect' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:755:in `do_start' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:744:in `start' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:1284:in `request' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/http-0.0.1/lib/http/client.rb:80:in `request' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/http-0.0.1/lib/http/chainable.rb:56:in `request' from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/http-0.0.1/lib/http/chainable.rb:10:in `get' from (irb):2
A root cert document could be distributed with the gem, so that it wold work with https servers. See: https://gist.github.com/996510
Redirector should allow to act in two modes: strict
(default) and non-strict
:
# strict mode with 5 max hops
HTTP.follow(true)
HTTP.follow(:strict => true)
HTTP.follow(:max_hops => 5, :strict => true)
# strict mode with unlimited hops
HTTP.follow(:max_hops => false)
HTTP.follow(:max_hops => false, :strict => true)
# non-strict mode with 5 max hops
HTTP.follow(:strict => false)
# non-strict mode with unlimited hops
HTTP.follow(:max_hops => false, :strict => false)
Thus in strict mode if original request was neither GET nor HEAD, and it received redirect 300, 301 or 302, a StateError
exception will be raised. In non-strict mode, request will be redirected with GET
verb. Notice that redirects with 307 and 308 codes will work same in both modes - redirect preserving verb of original request.
Notice, that such behavior is a mix of current RFC and proposed (1) changes (2) to that RFC.
RFC2616 says that upon any request but GET or HEAD, if response is redirect 300, 301, 302 or 307, user agent SHOULD NOT continue. Extension to that RFC says that old clients actually will change verb to GET in this case, but it also says, that correct behavior is to follow redirect with original verb if that verb is defined as "safe" (all methods are safe by default except PUT, DELETE and POST), otherwise user agent should halt. Experimental RFC that introduces redirect 308 says that 301 and 302 redirects should be followed with GET verb while 307 and 308 with original verb.
Also, notice that, main difference between 301 VS 308 (permanent redirect) and 302 VS 307 (temporary redirect) is the way how user agent should handle caching (this comes from original RFC).
Previous issue: #94
One thing I missed while reviewing #92 was that we delegate each
to @pile
but don’t mix-in Enumerable
, so methods like map
, etc. will not work. I’ll add this as well as some regression tests.
For example, it should be possible to require 'http/request'
and use the HTTP::Request
class without requiring the entire HTTP gem.
#70 added support for proxies, but there are no examples in the examples directory or documentation (that I could find).
Could you please add support for easily setting HTTP Basic Auth? Right now, it's "possible", but not very friendly:
HTTP.with('Authorization' => "Basic #{Base64.encode64('username:password')}")
But it would be nice if there was a nicer API wrapper, like:
HTTP.with_basic_auth('username', 'password')
Or something along those lines. Thoughts? /cc @sferik
When making a request, if the content type returned is application/json, the body is parsed as JSON. Sadly, servers (in this case, Google's calendar API) are happy to return HTML error pages with a JSON content type.
It's impossible to even know what the error said using the HTTP library, because the request raises a JSON parsing exception rather than returning the request in a way such that the body can be printed. :(
HTTP::Request#method
to HTTP::Request#verb
(@krainboltgreene)HTTP::ResponseBody
class (@tarcieri)HTTP::Client.request
and "friends" (#get
, #post
, etc) (@tarcieri)HTTP::Response#readpartial
(@tarcieri)HTTP::Headers
class (@ixti)HTTP::Request#redirect
(@ixti)HTTP::Response#content_type
(@ixti)HTTP::Response#mime_type
(@ixti)HTTP::Response#charset
(@ixti)HTTP::Error
namespace (@ixti)HTTP::Chainable#with_response
(@ixti)HTTP::Response::BodyDelegator
(@ixti)HTTP::Response#parsed_body
(@ixti)# Main API change you will mention is that `request` method and it's
# syntax sugar helpers like `get`, `post`, etc. now returns Response
# object instead of BodyDelegator:
response = HTTP.get "http://example.com"
raw_body = HTTP.get("http://example.com").to_s
parsed_body = HTTP.get("http://example.com/users.json").parse
# Second major change in API is work with request/response headers
# It is now delegated to `HTTP::Headers` class, so you can check it's
# documentation for details, here we will only outline main difference.
# Duckface (`[]=`) does not appends headers anymore
request[:content_type] = "text/plain"
request[:content_type] = "text/html"
request[:content_type] # => "text/html"
# In order to add multiple header values, you should pass array:
request[:cookie] = ["foo=bar", "woo=hoo"]
request[:cookie] # => ["foo=bar", "woo=hoo"]
# or call `#add` on headers:
request.headers.add :accept, "text/plain"
request.headers.add :accept, "text/html"
request[:accept] # => ["text/plain", "text/html"]
# Also, you can now read body in chunks (stream):
res = HTTP.get "http://example.com"
File.open "/tmp/dummy.bin", "wb" do |io|
while (chunk = res.readpartial)
io << chunk
end
end
At the moment HTTP method remains same as original upon redirects. It should be modified in some cases (e.g. 303): http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
It was temporarily removed in #64.
I’ve opened this issue as a blocker before we can release version v0.6.
When trying something like
Http.get("http://myserver.com/games")
I have a 400 status. Apache logs seem strange:
[Thu Oct 11 18:12:06 2012] [error] [client 194.78.223.33] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /games
It took me a long time to even figure that this was happening! I don't have an idea as to why.
When I use HTTP
to make GET requests, I'm seeing intermittent blank response bodies. The problem is not with the server response (which is never blank), and I don't see the problem with other libraries (I tested HTTParty
).
irb(main):010:0> url = "http://localhost:16000/locations/new.js"
irb(main):011:0> 1000.times.collect { |n| HTTP.get(url).to_s.length }.select { |l| l == 0 }.size
=> 4
irb(main):012:0> 1000.times.collect { |n| HTTParty.get(url).to_s.length }.select { |l| l == 0 }.size
=> 0
Per 1,000 requests with HTTP
I've seen as few as 3 blank response bodies, and as many as 22.
Is this something that could live inside http
(I stole the idea from Net::HTTP) ..?
the code is probably easy to pick this up from but the case-equality operator is implemented to provide a generic representation of the 2XX(HTTPOK), and 3XX(redirect) status codes.
I think the idea is cool and maybe it gets more interesting when you think about implementing stuff like TweetTooLong
that can decipher an API error from twitter via a response object. it might not have a place inside http
in any form though.
require "http"
module HTTPOK
def self.===(response)
return false unless response.respond_to?(:status_code)
code = response.status_code.to_s
code.start_with?("2")
end
end
module HTTPRedirect
def self.===(response)
return false unless response.respond_to?(:status_code)
code = response.status_code.to_s
code.start_with?("3")
end
end
response = HTTP.get("https://www.google.com").response
case response
when HTTPOK
puts "HTTPOK"
when HTTPRedirect
puts "HTTPRedirect"
end
As per RFC 2616 section 4.4, Content-Length
is not a required header in responses. I'm using The HTTP Gem with a server that doesn't always send a Content-Length
header.
In these cases, the body content is ""
. Combined with accept(:json)
, this usually means A JSON text must at least contain two octets! (JSON::ParserError)
exception.
See how Curl handles this:
curl -i 'https://api.honeybadger.io/v1/projects.json?auth_token=a-bad-key' -H 'Accept: application/json'
HTTP/1.1 401 Unauthorized
Server: nginx/1.4.3
Date: Mon, 04 Nov 2013 21:48:29 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 401 Unauthorized
Strict-Transport-Security: max-age=31536000
X-UA-Compatible: IE=Edge,chrome=1
X-Request-Id: b48da36d18e3c259bfec15019f3393c4
X-Runtime: 0.039962
X-Rack-Cache: miss
{"error":"Invalid authentication token."}
With The HTTP Gem:
irb(main):001:0> require 'http'
=> true
irb(main):002:0> HTTP.get("https://api.honeybadger.io/v1/projects.json", params: {auth_token: "a-bad-key"})
=> ""
A gander at the code shows that this is because @body_remaining
is derived entirely from the presence of a Content-Length
header. In some (all?) conditions, if @body_remaining
is nil
, the body is not read.
Now that v0.6.0 has shipped, I’ve started benchmarking the http
branch of the twitter
gem against master
, which uses Faraday with Net::HTTP.
While the spec suite runs about 4X faster in the http
branch, the following real-world example—which outputs the screen names of my Twitter friends that start and end with the letter “a”—performs about 2X slower (5 seconds vs. 10 seconds).
require 'twitter'
client = Twitter::REST::Client.new(consumer_key: 'abc', consumer_secret: '123')
client.friend_ids('sferik').each_slice(100) do |slice|
client.users(slice).select { |u| u.screen_name.match(/^a\w*a$/i) }.each do |user|
puts user.screen_name
end
end
When I run:
ruby-prof examples/aa_friends.rb --mode=wall --sort=self --min_percent=1
It produces the following output:
%self total self wait child calls name
60.58 10.089 10.089 0.000 0.000 189 OpenSSL::SSL::SSLSocket#sysread
25.86 4.306 4.306 0.000 0.000 6 OpenSSL::SSL::SSLSocket#connect
11.91 1.983 1.983 0.000 0.000 6 TCPSocket#initialize
As you can see, I’m initializing a new TCP socket and calling connect
for each HTTP request, which takes about a second per request. In the master
branch, I’m able to share a single HTTP connection across multiple request, which explains why it’s about 5 seconds faster.
How can I share a single connection across multiple HTTP requests with the HTTP gem? Without this feature, I won’t be able to replace faraday
with http
in the twitter
gem. 😦
I propose to remove following rubocop config in favor of default settings.
# Align with the style guide.
CollectionMethods:
PreferredMethods:
collect: 'map'
inject: 'reduce'
find: 'detect'
find_all: 'select'
I might have strange reason, but with existing config, you will not be able to use Pathname#find
(although we don't use it at the moment).
I haven't tracked down anything about this yet, but I have a failing test case.
First, click this link: http://t.co/CN60RkMG
Then:
Http.with_follow(true).get("http://t.co/CN60RkMG")
# => ""
Http.with_follow(true).get("http://t.co/CN60RkMG").response
#=> #<HTTP/1.1 200 OK @headers={"Server"=>"nginx", "Date"=>"Fri, 01 Feb 2013 22:59:53 GMT", "Content-Type"=>"text/html; charset=UTF-8", "Transfer-Encoding"=>"chunked", "Connection"=>"keep-alive", "Expires"=>"Fri, 01 Feb 2013 23:39:05 GMT", "X-Ua-Compatible"=>"IE=EmulateIE9,chrome=1", "Vary"=>"User-Agent,Accept-Encoding", "X-Frame-Options"=>"SAMEORIGIN", "X-Dns-Prefetch-Control"=>"on", "X-Varnish"=>"68307248 67584637", "Age"=>"1248", "Via"=>"1.1 varnish", "X-Varnish-Cache"=>"1", "X-Vserver"=>"10.90.128.188", "Cache-Control"=>"no-store, no-cache, must-revalidate, post-check=0, pre-check=0"}>
Expected behaviour: The html from that page should've been returned....
This is one of those dumb things that you wish the standard library was smart enough to work around, but it isnt.
Right now, HTTP takes a string as the url for the request, and converts it to a URI object internally. Thats not bad.
The problem arises if you pass it a URI object. URI.parse
does not like being given URI objects. 😒
It seems rather silly to have to convert my URI objects to strings, just to have HTTP convert them back to URIs. Maybe some class detection?
Reel in it's prerelase depends on http with a headers module but the http gem released to ruby gems doesn't have that module. Yes, they're all prereleases but still.
Would like to see HTTP gem playing well with GZip/Inflate out of the box.
There are some fixes that haven't made it into rubygems. How about publishing a new version of the gem?
Seemed like a good idea a the time. Now I'm not sure it's worth it, especially contained in the same gem.
Is anyone opposed to dropping Ruby 1.8 support in this library?
I have been trying to send some query parameters through GET but so far i don't really know how to do it with the Http gem.
Appending the params to the URL string doesn't help since the code create an URI object and calls for the host property.
Looking at the code i didn't see anything like params (Http.get URL, :params => {.. }) and using the body option throws a NoMethodError on a bytesize method.
After revisiting of bearer token builder code, I have deprecated :encode
option as it actually does not belong to Bearer token at all and I'm 100% positive nobody was (and nobody will be) using that option, unless they are under effect of serious medicine drugs.
That left bearer token options to become a single-field hash. So I tend to allow pass token directly without enclosing it into a { :token => ... }
options Hash. This leads me to question:
Do we want to support both notations of options String
and Hash[:token => String]
? Here's example:
# Hash[:token => String] notation
HTTP.auth(:bearer, :token => "ABBA")
# String notation
HTTP.auth(:bearer, "ABBA")
Please notice that BasicAuth
header is expecting #fetch
object anyway, as it needs user and pass. Thus my main question here is do we want to support both notations for BearerToken or we want to stick with only one variant, if so, which one?
PS I prefer String without Hash enclosure.
It should unless one has been explicitly specified
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.