httprb / http Goto Github PK

… because it currently doesn't, and that seems to be the main celling point (sorry), for this gem.

It took me a long time to even figure that this was happening! I don't have an idea as to why.

When I use HTTP to make GET requests, I'm seeing intermittent blank response bodies. The problem is not with the server response (which is never blank), and I don't see the problem with other libraries (I tested HTTParty).

irb(main):010:0> url = "http://localhost:16000/locations/new.js"
irb(main):011:0> 1000.times.collect { |n| HTTP.get(url).to_s.length }.select { |l| l == 0 }.size
=> 4
irb(main):012:0> 1000.times.collect { |n| HTTParty.get(url).to_s.length }.select { |l| l == 0 }.size
=> 0

Per 1,000 requests with HTTP I've seen as few as 3 blank response bodies, and as many as 22.

There are some fixes that haven't made it into rubygems. How about publishing a new version of the gem?

Seeing this behavior on both 0.5.0 and HEAD:

logger.info "HTTP.send(#{method},#{url},#{opts})"

#HTTP.send(get,http://127.0.0.1:3000/api/v1/agent_status/maximum_calls,{:params=>{:prospect_type=>:appointment}})

request = HTTP.accept(:json).with_response(:object).send(method, url, opts)
logger.info "request = #{request.inspect}"

#17:56:22 ahn.1  | [2013-12-09 17:56:22] INFO  TyphoeusHelper::SharedHydra: request = #<HTTP::Response/1.1 200 OK @headers={"Content-Type"=>"application/json; charset=utf-8", "Vary"=>"Accept-Encoding", "X-Ua-Compatible"=>"IE=Edge", "Etag"=>"\"64108005385238b566cf18d4efaa73f2\"", "Cache-Control"=>"max-age=0, private, must-revalidate", "X-Request-Id"=>"930be09be03d13110e09b6e9c0cae0c8", "X-Runtime"=>"0.058730", "Connection"=>"close", "Server"=>"thin 1.5.1 codename Straight Razor"}>

logger.info "body = #{request.body}"
#EOFError: end of file reached

(Don't mind the Hyrda stuff, switching from Typheous haven't renamed the classes yet)

Seemed like a good idea a the time. Now I'm not sure it's worth it, especially contained in the same gem.

Summary

• Rename HTTP::Request#method to HTTP::Request#verb (@krainboltgreene)
• Add HTTP::ResponseBody class (@tarcieri)
• Change API of response on HTTP::Client.request and "friends" (#get, #post, etc) (@tarcieri)
• Add HTTP::Response#readpartial (@tarcieri)
• Add HTTP::Headers class (@ixti)
• Fix and improve following redirects (@ixti)
• Add HTTP::Request#redirect (@ixti)
• Add HTTP::Response#content_type (@ixti)
• Add HTTP::Response#mime_type (@ixti)
• Add HTTP::Response#charset (@ixti)
• Improve error message upon invalid URI scheme (@ixti)
• Consolidate errors under common HTTP::Error namespace (@ixti)
• Add easy way of adding Authorization header (@ixti)
• Fix proxy support (@hundredwatt)
• Fix and improve query params handing (@jwinter)
• Change API of custom MIME type parsers (@ixti)
• Remove HTTP::Chainable#with_response (@ixti)
• Remove HTTP::Response::BodyDelegator (@ixti)
• Remove HTTP::Response#parsed_body (@ixti)
• Bump up input buffer from 4K to 16K (@tarcieri)

Details

# Main API change you will mention is that `request` method and it's
# syntax sugar helpers like `get`, `post`, etc. now returns Response
# object instead of BodyDelegator:

response = HTTP.get "http://example.com"
raw_body = HTTP.get("http://example.com").to_s
parsed_body = HTTP.get("http://example.com/users.json").parse

# Second major change in API is work with request/response headers
# It is now delegated to `HTTP::Headers` class, so you can check it's
# documentation for details, here we will only outline main difference.
# Duckface (`[]=`) does not appends headers anymore

request[:content_type] = "text/plain"
request[:content_type] = "text/html"
request[:content_type] # => "text/html"

# In order to add multiple header values, you should pass array:

request[:cookie] = ["foo=bar", "woo=hoo"]
request[:cookie] # => ["foo=bar", "woo=hoo"]

# or call `#add` on headers:

request.headers.add :accept, "text/plain"
request.headers.add :accept, "text/html"
request[:accept] # => ["text/plain", "text/html"]

# Also, you can now read body in chunks (stream):

res = HTTP.get "http://example.com"
File.open "/tmp/dummy.bin", "wb" do |io|
  while (chunk = res.readpartial)
    io << chunk
  end
end

Redirector should allow to act in two modes: strict (default) and non-strict:

# strict mode with 5 max hops
HTTP.follow(true)
HTTP.follow(:strict => true)
HTTP.follow(:max_hops => 5, :strict => true)

# strict mode with unlimited hops
HTTP.follow(:max_hops => false)
HTTP.follow(:max_hops => false, :strict => true)

# non-strict mode with 5 max hops
HTTP.follow(:strict => false)

# non-strict mode with unlimited hops
HTTP.follow(:max_hops => false, :strict => false)

Strict mode VS non-strict

Thus in strict mode if original request was neither GET nor HEAD, and it received redirect 300, 301 or 302, a StateError exception will be raised. In non-strict mode, request will be redirected with GET verb. Notice that redirects with 307 and 308 codes will work same in both modes - redirect preserving verb of original request.

Notice, that such behavior is a mix of current RFC and proposed (1) changes (2) to that RFC.

RFC2616 says that upon any request but GET or HEAD, if response is redirect 300, 301, 302 or 307, user agent SHOULD NOT continue. Extension to that RFC says that old clients actually will change verb to GET in this case, but it also says, that correct behavior is to follow redirect with original verb if that verb is defined as "safe" (all methods are safe by default except PUT, DELETE and POST), otherwise user agent should halt. Experimental RFC that introduces redirect 308 says that 301 and 302 redirects should be followed with GET verb while 307 and 308 with original verb.

Also, notice that, main difference between 301 VS 308 (permanent redirect) and 302 VS 307 (temporary redirect) is the way how user agent should handle caching (this comes from original RFC).

List of references to keep in mind:

• https://tools.ietf.org/html/rfc2616#section-10.3
• https://tools.ietf.org/html/draft-reschke-http-status-308-07
• https://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-6.4

Previous issue: #94

I’m trying to perform a multipart POST with the HTTP gem but I’m having trouble getting the headers and body right. We should include an example of how to do this in the README.

We're punting on keepalive and connection pooling (#72) for 0.6. We should ensure that sockets are closed correctly request completes (i.e. that we implement Connection: close semantics correctly)

Would be nice to have some form of Response caching here, with a simple Hash Store and Memcache API Store, seeing how Github itself doesn't count successfull Conditional Gets towards the API limit.

See #43

If the proxy API isn't working it should be removed. We should either fix it or remove it.

If it's removed it can be placed onto a branch where someone can fix it later if they so desire.

You should definetely make a release, there is such a large gap between 0.5.0 and master it looks like a completely different library, even if there are issues left in master it is worth pushing a release out I think since 0.5.0 has issues of its own anyway xD

is there any major issue preventing a release ?

  ➔ irb
> require 'http'
=> true
> HTTP.get 'https://www.google.com/'
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:799:in `connect'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:799:in `block in connect'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:799:in `connect'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:744:in `start'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:1284:in `request'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/http-0.0.1/lib/http/client.rb:80:in `request'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/http-0.0.1/lib/http/chainable.rb:56:in `request'
    from /Users/john/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/http-0.0.1/lib/http/chainable.rb:10:in `get'
    from (irb):2

A root cert document could be distributed with the gem, so that it wold work with https servers. See: https://gist.github.com/996510

Using the awesome http gem for parsing response from a POST request. Response is a simple JSON, ~350 bytes. Response doesn't include the Content-Length header, which is not mandatory for normal HTTP communication. Digging more in the code, problem seems to be in the body/chunk parsing code in https://github.com/tarcieri/http/blob/master/lib/http/client.rb. Here's what I am seeing

• Client requests a POST
• Client gets and formats the response (I am able to see the 'chunk' in response)
• Client creates a HTTP:Response instance
• JSON MimeType is wired properly and works
• When parsing the body, block in https://github.com/tarcieri/http/blob/master/lib/http/client.rb#L84 is hit once
• Whole chunk is read once and since @body_remaining is nil, whole block above is executed again, which tries to read the socket again - which results into a EOF reached error

I saw a couple of open issues related to the same block of code, so the logic might have to be revisited or broken down even more.

Now that v0.6.0 has shipped, I’ve started benchmarking the http branch of the twitter gem against master, which uses Faraday with Net::HTTP.

While the spec suite runs about 4X faster in the http branch, the following real-world example—which outputs the screen names of my Twitter friends that start and end with the letter “a”—performs about 2X slower (5 seconds vs. 10 seconds).

require 'twitter'

client = Twitter::REST::Client.new(consumer_key: 'abc', consumer_secret: '123')
client.friend_ids('sferik').each_slice(100) do |slice|
  client.users(slice).select { |u| u.screen_name.match(/^a\w*a$/i) }.each do |user|
    puts user.screen_name
  end
end

When I run:

ruby-prof examples/aa_friends.rb --mode=wall --sort=self --min_percent=1

It produces the following output:

 %self      total      self      wait     child     calls  name
 60.58     10.089    10.089     0.000     0.000      189   OpenSSL::SSL::SSLSocket#sysread 
 25.86      4.306     4.306     0.000     0.000        6   OpenSSL::SSL::SSLSocket#connect 
 11.91      1.983     1.983     0.000     0.000        6   TCPSocket#initialize 

As you can see, I’m initializing a new TCP socket and calling connect for each HTTP request, which takes about a second per request. In the master branch, I’m able to share a single HTTP connection across multiple request, which explains why it’s about 5 seconds faster.

How can I share a single connection across multiple HTTP requests with the HTTP gem? Without this feature, I won’t be able to replace faraday with http in the twitter gem. 😦

In lots of cases streaming of body is not needed, so it's better to flush socket at the same time headers were received, thus underlying connection is closed for sure.

Could you please add support for easily setting HTTP Basic Auth? Right now, it's "possible", but not very friendly:

HTTP.with('Authorization' => "Basic #{Base64.encode64('username:password')}")

But it would be nice if there was a nicer API wrapper, like:

HTTP.with_basic_auth('username', 'password')

Or something along those lines. Thoughts? /cc @sferik

One thing I missed while reviewing #92 was that we delegate each to @pile but don’t mix-in Enumerable, so methods like map, etc. will not work. I’ll add this as well as some regression tests.

I haven't tracked down anything about this yet, but I have a failing test case.

First, click this link: http://t.co/CN60RkMG

Then:

Http.with_follow(true).get("http://t.co/CN60RkMG")
# => ""

Http.with_follow(true).get("http://t.co/CN60RkMG").response
#=> #<HTTP/1.1 200 OK @headers={"Server"=>"nginx", "Date"=>"Fri, 01 Feb 2013 22:59:53 GMT", "Content-Type"=>"text/html; charset=UTF-8", "Transfer-Encoding"=>"chunked", "Connection"=>"keep-alive", "Expires"=>"Fri, 01 Feb 2013 23:39:05 GMT", "X-Ua-Compatible"=>"IE=EmulateIE9,chrome=1", "Vary"=>"User-Agent,Accept-Encoding", "X-Frame-Options"=>"SAMEORIGIN", "X-Dns-Prefetch-Control"=>"on", "X-Varnish"=>"68307248 67584637", "Age"=>"1248", "Via"=>"1.1 varnish", "X-Varnish-Cache"=>"1", "X-Vserver"=>"10.90.128.188", "Cache-Control"=>"no-store, no-cache, must-revalidate, post-check=0, pre-check=0"}>

Expected behaviour: The html from that page should've been returned....

This seems impossible.. I have a remote app generating 2 Set-Cookie lines and another device inline inserting a third Set-Cookie line after, only the third pops into the headers hash, I need the first two.

Looks like this requires a patch to http_parser.rb to do, which was offered as a pull for a few years but never accepted. The underlying C parses out cookies into their own object.

This is an Issue(tm) for people trying to integrate into apps (like Oracle business apps) that use cookies for things they have no business using cookies for but it's unavoidable. So I guess I'm reporting this as a warning?

This is the current gem description:

HTTP so awesome it will lure Catherine Zeta Jones into your unicorn petting zoo

In my experience, this is both inaccurate and vaguely sexist/heteronormative.

I’d like to see this changed before the v0.6 release.

See 7179996

cc @blambeau

After revisiting of bearer token builder code, I have deprecated :encode option as it actually does not belong to Bearer token at all and I'm 100% positive nobody was (and nobody will be) using that option, unless they are under effect of serious medicine drugs.

That left bearer token options to become a single-field hash. So I tend to allow pass token directly without enclosing it into a { :token => ... } options Hash. This leads me to question:

Do we want to support both notations of options String and Hash[:token => String]? Here's example:

# Hash[:token => String] notation
HTTP.auth(:bearer, :token => "ABBA")

# String notation
HTTP.auth(:bearer, "ABBA")

Please notice that BasicAuth header is expecting #fetch object anyway, as it needs user and pass. Thus my main question here is do we want to support both notations for BearerToken or we want to stick with only one variant, if so, which one?

PS I prefer String without Hash enclosure.

/cc @sferik @tarcieri

It was temporarily removed in #64.

I’ve opened this issue as a blocker before we can release version v0.6.

The calling of a GET, with a chained response, gives the no 'response' method for String. Something like that:

get = Http.get("http://www.google.com").response

Any ideas on this? I've checked the unit tests and this should work normally.

Is anyone opposed to dropping Ruby 1.8 support in this library?

@tarcieri I believe it worth to release v0.5.1 with latest state of 0-5-release and drop 0.5.x support. So, 0.6.1 will become a pivot of release-0.6 branch (for backports and support and such) and master will be able to merge 0.7 changes :D like deprecations and so on.

I'm using Ruby 2.0.0 and HTTP 0.5.0 and using this tiny little example:

require 'http'
response = HTTP.via('192.168.1.1513', 8888).get("http://www.google.co.nz").response
puts response.body

The request goes through just fine but in reality I have no HTTP proxy sitting on that address or port. I've tried it with my real web debugger address/port as well but it still goes through.

Is there a special way to enable proxies? I should note I do have Celluloid/IO installed on my machine if it makes any difference.

I propose to remove following rubocop config in favor of default settings.

# Align with the style guide.
CollectionMethods:
  PreferredMethods:
    collect: 'map'
    inject: 'reduce'
    find: 'detect'
    find_all: 'select'

I might have strange reason, but with existing config, you will not be able to use Pathname#find (although we don't use it at the moment).

At the moment we implicitly set Host header of request in case it wasn't provided:

@headers['Host'] ||= @uri.host

I tend to think we should enforce it instead. The only use case that might want current (implicit behavior) is pretty weird IMHO:

HTTP[:host => 'example.com'].get('http://127.0.0.1:8080/wtf?')

This is the only warning I get when I run rails c in my whole rails app. Might this be fixable or silencable?

/Users/bat/.rvm/gems/ruby-1.9.3-p194@my_rails_app/gems/http_parser.rb-0.5.3/lib/http_parser.rb:4: warning: already initialized constant Http

This is one of those dumb things that you wish the standard library was smart enough to work around, but it isnt.

Right now, HTTP takes a string as the url for the request, and converts it to a URI object internally. Thats not bad.

The problem arises if you pass it a URI object. URI.parse does not like being given URI objects. 😒

It seems rather silly to have to convert my URI objects to strings, just to have HTTP convert them back to URIs. Maybe some class detection?

Would like to see HTTP gem playing well with GZip/Inflate out of the box.

Hi -- I love the library.

What do you think about putting the POST vars hash at the top level instead of pointing at it with :form? There's nothing else in that options hash anyway, and also the word "form" kind of bothers me, since that's device-specific and in this case it's clearly not a form :)

Http.post "http://example.com/resource", {:foo => "42"}

At the moment HTTP method remains same as original upon redirects. It should be modified in some cases (e.g. 303): http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Is this something that could live inside http (I stole the idea from Net::HTTP) ..?
the code is probably easy to pick this up from but the case-equality operator is implemented to provide a generic representation of the 2XX(HTTPOK), and 3XX(redirect) status codes.

I think the idea is cool and maybe it gets more interesting when you think about implementing stuff like TweetTooLong that can decipher an API error from twitter via a response object. it might not have a place inside http in any form though.

require "http"
module HTTPOK
  def self.===(response)
    return false unless response.respond_to?(:status_code)
    code = response.status_code.to_s
    code.start_with?("2")
  end
end

module HTTPRedirect
  def self.===(response)
    return false unless response.respond_to?(:status_code)
    code = response.status_code.to_s
    code.start_with?("3")
  end
end

response = HTTP.get("https://www.google.com").response
case response
when HTTPOK
  puts "HTTPOK"
when HTTPRedirect
  puts "HTTPRedirect"
end

Would be nice if Http would follow redirects, whether by default or as an option.

#70 added support for proxies, but there are no examples in the examples directory or documentation (that I could find).

I have been trying to send some query parameters through GET but so far i don't really know how to do it with the Http gem.

Appending the params to the URL string doesn't help since the code create an URI object and calls for the host property.

Looking at the code i didn't see anything like params (Http.get URL, :params => {.. }) and using the body option throws a NoMethodError on a bytesize method.

When trying something like

Http.get("http://myserver.com/games")

I have a 400 status. Apache logs seem strange:

[Thu Oct 11 18:12:06 2012] [error] [client 194.78.223.33] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /games

I'm not sure if something like this fits well streamed response bodies, but I thought it might be nice to be able to write something like this:

body = HTTP
.accepts(json: proc { |body| JSON.parse(body) })
.get("https://www.somewhere.com/index")
.body

body["people"].each do |person|
  # ..
end

For example, it should be possible to require 'http/request' and use the HTTP::Request class without requiring the entire HTTP gem.

Trying to connect to https

resp = Http.with_headers(:accept => 'application/json').post "https://api.polldaddy.com", :body => JSON(req)

I'm getting this error: OpenSSL::SSL::SSLSocket:Class

Any thoughts?

Thanks!

Reel in it's prerelase depends on http with a headers module but the http gem released to ruby gems doesn't have that module. Yes, they're all prereleases but still.

As per RFC 2616 section 4.4, Content-Length is not a required header in responses. I'm using The HTTP Gem with a server that doesn't always send a Content-Length header.

In these cases, the body content is "". Combined with accept(:json), this usually means A JSON text must at least contain two octets! (JSON::ParserError) exception.

See how Curl handles this:

curl -i 'https://api.honeybadger.io/v1/projects.json?auth_token=a-bad-key' -H 'Accept: application/json'
HTTP/1.1 401 Unauthorized
Server: nginx/1.4.3
Date: Mon, 04 Nov 2013 21:48:29 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 401 Unauthorized
Strict-Transport-Security: max-age=31536000
X-UA-Compatible: IE=Edge,chrome=1
X-Request-Id: b48da36d18e3c259bfec15019f3393c4
X-Runtime: 0.039962
X-Rack-Cache: miss

{"error":"Invalid authentication token."}

With The HTTP Gem:

irb(main):001:0> require 'http'
=> true
irb(main):002:0> HTTP.get("https://api.honeybadger.io/v1/projects.json", params: {auth_token: "a-bad-key"})
=> ""

A gander at the code shows that this is because @body_remaining is derived entirely from the presence of a Content-Length header. In some (all?) conditions, if @body_remaining is nil, the body is not read.

https://github.com/tarcieri/http/blob/master/lib/http/response/body.rb#L17 calls the Client#read method, which doesn't exist. Would submit a pull request, but not sure if Body#read should be deleted or updated to used readpartial or something else.

A broken implementation of proxy support was removed in #62.

I’ve added this issue to the version v0.6 milestone. IMHO, it is not a blocker for v0.6, since it was never properly working, but is a blocker for v1.0.0. Better to implement it sooner rather than later to ensure we get the API right.

When making a request, if the content type returned is application/json, the body is parsed as JSON. Sadly, servers (in this case, Google's calendar API) are happy to return HTML error pages with a JSON content type.

It's impossible to even know what the error said using the HTTP library, because the request raises a JSON parsing exception rather than returning the request in a way such that the body can be printed. :(

The current API is probably overly simplified:

body = HTTP.get(url)

To get the response object, we have to do this:

response = HTTP.get(url).response

Really it should probably be:

response = HTTP.get(url)
body = response.body

It'd be nice if body were an HTTP::ResponseBody object as well, then we could support streaming like this:

body = HTTP.get(url).body
chunk1 = body.readpartial
chunk2 = body.readpartial

We could also make it Enumerable so you can use it like this if you prefer:

body = HTTP.get(url).body
body.each do |chunk|
  ...
end

It should unless one has been explicitly specified