Giter VIP home page Giter VIP logo

httpotion's Introduction

hex.pm version hex.pm downloads API Docs unlicense

HTTPotion

HTTP client for Elixir, based on ibrowse. Continues the HTTPun tradition of HTTParty, HTTPretty, HTTParrot and HTTPie.

Note: unfortunately, ibrowse seems a bit buggy and not super actively maintained. You might want to check out Tesla with hackney instead, and this cert verification setup.

Installation

Add HTTPotion to your project's dependencies in mix.exs:

  defp deps do
    [
      {:httpotion, "~> 3.1.0"}
    ]
  end

  def application do
    [ applications: [:httpotion] ]
    # Application dependency auto-starts it, otherwise: HTTPotion.start
  end

And fetch your project's dependencies:

$ mix deps.get

Usage

Note: You can load HTTPotion into the Elixir REPL by executing this command from the root of your project:

$ iex -S mix

Note: HTTPotion now enables certificate verification by default, using a few default CA bundle paths (/etc/ssl/cert.pem etc.) or the certifi package (which is not a mandatory dependency).

Some basic examples:

iex> response = HTTPotion.get "https://httpbin.org/get"
%HTTPotion.Response{
  body: "{\n…",
  headers: %HTTPotion.Headers{ hdrs: %{"connection" => "keep-alive",} },
  status_code: 200
}

iex> HTTPotion.Response.success?(response)
true

# Response headers are wrapped to allow case-insensitive access (and to support both atoms and strings)
iex> response.headers[:sErvEr]
"meinheld/0.6.1"

iex> response.headers["ConTenT-TyPe"]
"application/json"

# Response headers can have multiple values
iex> response = HTTPotion.get "https://httpbin.org/response-headers?foo=1&foo=2&bar=1"
%HTTPotion.Response{
  body: "{\n…",
  headers: %HTTPotion.Headers{ hdrs: %{"foo" => ["1", "2"], "bar" => "1"} },
  status_code: 200
}

# You can provide a map for the query string
iex> HTTPotion.get("https://httpbin.org/get", query: %{page: 2})
%HTTPotion.Response{body: "…", headers: , status_code: 200}

# Follow redirects
iex> HTTPotion.get("https://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F", follow_redirects: true)
%HTTPotion.Response{body: "…<title>Example Domain</title>…", headers: , status_code: 200}

# Send form data
iex> HTTPotion.post "https://httpbin.org/post", [body: "hello=" <> URI.encode_www_form("w o r l d !!"),
  headers: ["User-Agent": "My App", "Content-Type": "application/x-www-form-urlencoded"]]
%HTTPotion.Response{body: "…", headers: , status_code: 200}

# Use a custom method
iex> HTTPotion.request :propfind, "http://httpbin.org/post", [body: "I have no idea what I'm doing"]
%HTTPotion.Response{body: "…", headers: , status_code: 405}

# Send Basic auth credentials
iex> HTTPotion.get "https://httpbin.org/basic-auth/foo/bar", [basic_auth: {"foo", "bar"}]
%HTTPotion.Response{
  body: "…",
  headers: %HTTPotion.Headers { hdrs: %{"Access-Control-Allow-Credentials": "true",} },
  status_code: 200
}

# Pass options to ibrowse (note that it usually takes char_lists, not elixir strings)
iex> HTTPotion.get "https://check-tls.akamaized.net", [ ibrowse: [ ssl_options: [ versions, [:'tlsv1.1'] ] ] ]
%HTTPotion.Response{body: "…TLS SNI: present - Check TLS - (https,tls1.1,ipv4)…", headers: , status_code: 200}

# Change the timeout (default is 5000 ms)
iex> HTTPotion.get "https://example.com", [timeout: 10_000]

# If there is an error a `HTTPotion.ErrorResponse` is returned
iex> HTTPotion.get "http://localhost:1"
%HTTPotion.ErrorResponse{message: "econnrefused"}

# You can also raise `HTTPError` with the `bang` version of request
iex> HTTPotion.get! "http://localhost:1"
** (HTTPotion.HTTPError) econnrefused

The Response is a struct, you can access its fields like: response.body.

response.headers is a struct (HTTPotion.Headers) that wraps a map to provide case-insensitive access (so you can use response.headers[:authorization] and it doesn't matter if the server returned AuThOrIZatIOn or something).

HTTPError is an exception that happens when a bang request (request! / get! / …) fails.

Available options and their default values:

{
  body: "",                # Request's body contents, e.g. "{json: \"string\"}"
  headers: [],             # Request's headers, e.g. [Accept: "application/json"]
  query: nil,              # Query string, e.g. %{page: 1}
  timeout: 5000,           # Timeout in milliseconds, e.g. 5000
  basic_auth: nil,         # Basic auth credentials, e.g. {"username", "password"}
  stream_to: nil,          # A process to stream the response to when performing async requests
  direct: nil,             # An ibrowse worker for direct mode
  ibrowse: [],             # ibrowse options
  auto_sni: true,          # Whether TLS SNI should be automatically configured (does URI parsing)
  follow_redirects: false, # Whether redirects should be followed
}

Metaprogramming magic

You can extend HTTPotion.Base to make cool HTTP API wrappers (this example uses Poison for JSON):

defmodule GitHub do
  use HTTPotion.Base

  def process_url(url) do
    "https://api.github.com/" <> url
  end

  def process_request_headers(headers) do
    Dict.put headers, :"User-Agent", "github-potion"
  end

  def process_response_body(body) do
    body |> Poison.decode!
  end
end
iex> GitHub.get("users/unrelentingtech").body["public_repos"]
233

Read the source to see all the hooks.

Keep in mind that process_response_body and process_response_chunk get iodata. By default, they call IO.iodata_to_binary. But efficient parsers like Poison can work directly on iodata.

Asynchronous requests

You can get the response streamed to your current process asynchronously:

iex> HTTPotion.get "http://httpbin.org/get", [stream_to: self]
%HTTPotion.AsyncResponse{id: -576460752303419903}

iex> flush
%HTTPotion.AsyncHeaders{
  id: -576460752303419903,
  status_code: 200,
  headers: %HTTPotion.Headers{ hdrs: %{"connection" => "keep-alive",} }
}
%HTTPotion.AsyncChunk{
  id: -576460752303419903,
  chunk: "{\n…"
}
%HTTPotion.AsyncEnd{
  id: -576460752303419903
}

Note that instead of process_response_body, process_response_chunk is called on the chunks before sending them out to the receiver (the stream_to process).

Direct access to ibrowse workers

ibrowse allows you to use its separate worker processes directly. We expose this functionality through the direct option.

Don't forget that you have to pass the URL to the worker process, which means the worker only communicates with one server (domain!)

iex> {:ok, worker_pid} = HTTPotion.spawn_worker_process("http://httpbin.org")

iex> HTTPotion.get "httpbin.org/get", [direct: worker_pid]
%HTTPotion.Response{body: "…", headers: ["Connection": "close",], status_code: 200}

You can even combine it with async!

iex> {:ok, worker_pid} = HTTPotion.spawn_worker_process("http://httpbin.org")

iex> HTTPotion.post "httpbin.org/post", [direct: worker_pid, stream_to: self, headers: ["User-Agent": "hello it's me"]]
%HTTPotion.AsyncResponse{id: {1372,8757,656584}}

Type analysis

HTTPotion contains typespecs so your usage can be checked with dialyzer, probably via dialyxir or elixir-ls.

HTTPotion's tests are checked with dialyxir.

Contributing

Please feel free to submit pull requests!

By participating in this project you agree to follow the Contributor Code of Conduct.

License

This is free and unencumbered software released into the public domain.
For more information, please refer to the UNLICENSE file or unlicense.org.

httpotion's People

Contributors

adamkittelson avatar aforward avatar arkar-aung avatar avdgaag avatar ben-pr-p avatar bradleyd avatar cbarratt avatar ctreatma avatar cwc avatar danielberkompas avatar edgurgel avatar eidge avatar eitoball avatar ewildgoose avatar filipecabaco avatar guilleiguaran avatar henrik avatar jadlr avatar josephwilk avatar kemonomachi avatar lexmag avatar lizziepaquette avatar lowks avatar lucas-aragno avatar my-flow avatar nuxlli avatar ortuna avatar parroty avatar pragdave avatar valpackett avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

httpotion's Issues

Access protocol compilation warning

On compilation the following warning is thrown:

lib/httpotion.ex:292: warning: implementation of the Access protocol is deprecated. For customization of the dict[key] syntax, please implement the Dict behaviour instead

PR that corrects this:
#80

Why can't HTTPotion locate ibrowse?

I'm trying to consume a JSON API with my first Elixir application, but I'm stuck very early on. When I call HTTPotion.start from the elixir shell, I get the following response:

{:error, {:ibrowse, {'no such file or directory', 'ibrowse.app'}}}

Curious, I decided to see if I could invoke ibrowse directly, but calling :ibrowse.start returns

** (UndefinedFunctionError) undefined function: :ibrowse.start/0
:ibrowse.start()

I added HTTPotion to my mix.exs's dependencies and ran mix deps.get, which completed successfully. Why can't HTTPotion and my IEx shell find ibrowse?

Proxy to local address does not seem to work

Here is the request I'm trying:

HTTPotion.get("https://api-staging.healthwavehq.com", [headers: [{"Accept", "application/json"}], ibrowse: [proxy_host: "127.0.0.1", proxy_port: 8888]] )

I'm essentially trying to pass this request through Charles Proxy here is the error message:

15:49:14.912 [error] GenServer #PID<0.151.0> terminating ** (ArgumentError)
argument error (kernel) gen_tcp.erl:149: :gen_tcp.connect/4 (ibrowse)
src/ibrowse_http_client.erl:758: :ibrowse_http_client.send_req_1/8 (stdlib)
gen_server.erl:629: :gen_server.try_handle_call/4 (stdlib) gen_server.erl:661:
:gen_server.handle_msg/5 (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {:send_req, {{:url, 'https://api-staging.healthwavehq.com',
'api-staging.healthwavehq.com', 443, :undefined, :undefined, '/', :https,
:hostname}, [{'Accept', 'application/json'}], :get, "", [proxy_host:
"127.0.0.1", proxy_port: 8888], 5000}} State: {:state,
'api-staging.healthwavehq.com', 443, :undefined, #Reference<0.0.1.581>, false,
:undefined, [], true, :undefined, false, [], {[], []}, :undefined, :idle,
:undefined, "", 0, 0, [], :undefined, :undefined, :undefined, :undefined,
false, :undefined, :undefined, "", :undefined, false, 98343, 0, :undefined,
:undefined} %HTTPotion.ErrorResponse{message: "{:EXIT, {:badarg, {:gen_server,
:call, [#PID<0.151.0>, {:send_req, {{:url,
'https://api-staging.healthwavehq.com', 'api-staging.healthwavehq.com', 443,
:undefined, :undefined, '/', :https, :hostname}, [{'Accept',
'application/json'}], :get, \"\", [proxy_host: \"127.0.0.1\", proxy_port:
8888], 5000}}, 5000]}}}"}

It says badarg but I'm not sure what the bad argument would be.

How to do a multipart form post?

It's not covered in the docs but is there any way I can send some kind of mapping of keys and values for a multipart form post body to httpotion without writing the whole post body out as a string?

Get source after Javascript is run

Is it possible to get the source of a page after javascript has been run? I keep getting the pre-JS source which does not have the data I expect it to have.

Thanks.

next hex.pm release?

I was just wondering when you were planning to push another release to hex.pm? I'd like to use some of the features in master in one of my libraries but I can't have a hex package with gitref dependencies.

Thanks!
Adam

Query String Parameters

I couldn't find it on the docs, but is there a way to pass a map to the request to be converted to a query string?

String vs binary problems, giving me errors

Hi, new to Elixir but know Erlang pretty well.

I'm trying to run my application (http://github.com/arthurcolle/fxex) but when I make a simple get request it tells me:

** (Protocol.UndefinedError) protocol Access not implemented for %HTTPotion.Response{body: "{\n  \"disclaimer\": \"Exchange rates are provided for informational purposes only, and do not constitute financial advice of any kind. Although every attempt is made to ensure quality, NO guarantees are given whatsoever of accuracy, validity, availability, or fitness for any purpose - please use at your own risk. All usage is subject to your acceptance of the Terms and Conditions of Service, available at: https://openexchangerates.org/terms/\",\n  \"license\": \"Data sourced from various providers with public-facing APIs; copyright may apply; resale is prohibited; no warranties given of any kind. Bitcoin data provided by http://coindesk.com. All usage is subject to your acceptance of the License Agreement available at: https://openexchangerates.org/license/\",\n  \"timestamp\": 1429855262,\n  \"base\": \"USD\",\n  \"rates\": {\n    \"AED\": 3.673121,\n    \"AFN\": 57.8736,\n    \"ALL\": 129.7089,\n    \"AMD\": 473.443998,\n    \"ANG\": 1.78902,\n    \"AOA\": 109.0425,\n    \"ARS\": 8.89224,\n    \"AUD\": 1.285626,\n    \"AWG\": 1.7925,\n    \"AZN\": 1.051675,\n    \"BAM\": 1.812719,\n    \"BBD\": 2,\n    \"BDT\": 77.5487,\n    \"BGN\": 1.811379,\n    \"BHD\": 0.37704,\n    \"BIF\": 1560.663333,\n    \"BMD\": 1,\n    \"BND\": 1.341802,\n    \"BOB\": 6.885179,\n    \"BRL\": 2.978822,\n    \"BSD\": 1,\n    \"BTC\": 0.004319313,\n    \"BTN\": 63.3652,\n    \"BWP\": 9.937702,\n    \"BYR\": 14393.133333,\n    \"BZD\": 1.990659,\n    \"CAD\": 1.216211,\n    \"CDF\": 930.8,\n    \"CHF\": 0.957962,\n    \"CLF\": 0.024598,\n    \"CLP\": 615.879204,\n    \"CNY\": 6.178422,\n    \"COP\": 2470.311667,\n    \"CRC\": 529.702275,\n    \"CUC\": 1,\n    \"CUP\": 0.998363,\n    \"CVE\": 102.207375,\n    \"CZK\": 25.39055,\n    \"DJF\": 177.632001,\n    \"DKK\": 6.910133,\n    \"DOP\": 44.65573,\n    \"DZD\": 98.2845,\n    \"EEK\": 14.51835,\n    \"EGP\": 7.617534,\n    \"ERN\": 15.1151,\n    \"ETB\": 20.44761,\n    \"EUR\": 0.925127,\n    \"FJD\": 2.03751,\n    \"FKP\": 0.664372,\n    \"GBP\": 0.664372,\n    \"GEL\": 2.2624,\n    \"GGP\": 0.664372,\n    \"GHS\": 3.836624,\n    \"GIP\": 0.664372,\n    \"GMD\": 43.00158,\n    \"GNF\": 7234.4,\n    \"GTQ\": 7.727193,\n    \"GYD\": 206.305752,\n    \"HKD\": 7.749997,\n    \"HNL\": 21.47521,\n    \"HRK\": 7.025943,\n    \"HTG\": 47.27169,\n    \"HUF\": 278.167998,\n    \"IDR\": 12936.916667,\n    \"ILS\": 3.928771,\n    \"IMP\": 0.664372,\n    \"INR\": 63.37642,\n    \"IQD\": 1182.059217,\n    \"IRR\": 28203.666667,\n    \"ISK\": 135.959,\n    \"JEP\": 0.664372,\n    \"JMD\": 114.912801,\n    \"JOD\": 0.70833,\n    \"JPY\": 119.575699,\n    \"KES\": 94.024179,\n    \"KGS\": 61.710667,\n    \"KHR\": 3999.585016,\n    \"KMF\": 456.328781,\n    \"KPW\": 900.09,\n    \"KRW\": 1080.218315,\n    \"KWD\": 0.302314,\n    \"KYD\": 0.823171,\n    \"KZT\": 185.6891,\n    \"LAK\": 8092.595,\n    \"LBP\": 1507.888333,\n    \"LKR\": 132.6568,\n    \"LRD\": 84.553335,\n    \"LSL\": 12.17813,\n    \"LTL\": 2.933867,\n    \"LVL\": 0.651597,\n    \"LYD\": 1.363291,\n    \"MAD\": 9.926872,\n    \"MDL\": 18.47916,\n    \"MGA\": 3080.914967,\n    \"MKD\": 57.03369,\n    \"MMK\": 1076.449988,\n    \"MNT\": 1963.166667,\n    \"MOP\": 7.975066,\n    \"MRO\": 314.022833,\n    \"MTL\": 0.397267,\n    \"MUR\": 35.9687,\n    \"MVR\": 15.256538,\n    \"MWK\": 440.478202,\n    \"MXN\": 15.36043,\n    \"MYR\": 3.5993,\n    \"MZN\": 35.233333,\n    \"NAD\": 12.17823,\n    \"NGN\": 198.771401,\n    \"NIO\": 26.7191,\n    \"NOK\": 7.853823,\n    \"NPR\": 100.92236,\n    \"NZD\": 1.322367,\n    \"OMR\": 0.384969,\n    \"PAB\": 1,\n    \"PEN\": 3.13101,\n    \"PGK\": 2.686497,\n    \"PHP\": 44.22541,\n    \"PKR\": 101.4992,\n    \"PLN\": 3.70452,\n    \"PYG\": 5009.971719,\n    \"QAR\": 3.640189,\n    \"RON\": 4.092864,\n    \"RSD\": 111.8845,\n    \"RUB\": 50.83582,\n    \"RWF\": 688.7738,\n    \"SAR\": 3.749258,\n    \"SBD\": 7.743432,\n    \"SCR\": 13.47469,\n    \"SDG\": 5.953994,\n    \"SEK\": 8.65835,\n    \"SGD\": 1.341455,\n    \"SHP\": 0.664372,\n    \"SLL\": 4363.333333,\n    \"SOS\": 703.255102,\n    \"SRD\": 3.3302,\n    \"STD\": 22712.35,\n    \"SVC\": 8.734096,\n    \"SYP\": 214.72235,\n    \"SZL\": 12.18003,\n    \"THB\": 32.4593,\n    \"TJS\": 6.1587,\n    \"TMT\": 3.500067,\n    \"TND\": 1.951288,\n    \"TOP\": 2.025876,\n    \"TRY\": 2.707308,\n    \"TTD\": 6.33743,\n    \"TWD\": 31.01267,\n    \"TZS\": 1930.46,\n    \"UAH\": 22.43285,\n    \"UGX\": 2997.001667,\n    \"USD\": 1,\n    \"UYU\": 26.66482,\n    \"UZS\": 2506.599967,\n    \"VEF\": 6.318541,\n    \"VND\": 21586.666667,\n    \"VUV\": 105.771666,\n    \"WST\": 2.472739,\n    \"XAF\": 608.548875,\n    \"XAG\": 0.06319767,\n    \"XAU\": 0.00083894,\n    \"XCD\": 2.70102,\n    \"XDR\": 0.722785,\n    \"XOF\": 607.904115,\n    \"XPF\": 110.66831,\n    \"YER\": 214.9484,\n    \"ZAR\": 12.16738,\n    \"ZMK\": 5253.075255,\n    \"ZMW\": 7.359064,\n    \"ZWL\": 322.355006\n  }\n}", headers: ["Access-Control-Allow-Origin": "*", "Cache-Control": "public", Connection: "close", "Content-Length": "4402", "Content-Type": "application/json; charset=utf-8", Date: "Fri, 24 Apr 2015 06:12:04 GMT", ETag: "\"16e6e8b1c39237051c16c0d7fb174da3\"", "Last-Modified": "Fri, 24 Apr 2015 06:01:02 GMT", Server: "Apache"], status_code: 200}
    (elixir) lib/access.ex:1: Access.impl_for!/1
    (elixir) lib/access.ex:47: Access.get/2
      (fxex) lib/fxex.ex:23: Fxex.work/0
iex(1)>

Was wondering if you had any pointers?

Thanks

Can't understand the performance of various steps of making a request.

Sorry, I realize this is a little weird, but bare with me here.

I'm using HTTPotion to make requests to external resources and time how long it takes. As a simple start, I'm marking the time before I actually call HTTPotion.get, and then capturing the time when I receive %HTTPotion.AsyncHeaders{}, the first %HTTPotion.AsyncChunk{}, and finally, %HTTPotion.AsyncEnd{}.

Now, this works, and everything happens in the expected order. If I measure the time between start and end, the time looks right. What doesn't look right, however, is the time I measure of when the headers arrive.

Here's some data for grabbing the SSH keys I have set up for Github:

%{check_id: "test check",
  check_options: %{check_url: "https://github.com/tobz.keys"},
  results: %{body_end: 1471278282867, body_start: 1471278282867,
    headers_at: 1471278282630, started_at: 1471278282629}}

These times are the system time in milliseconds, so going in chronological order:

T+0: request fired off
T+1ms: headers arrived/are ready
T+238ms: first response body chunk arrives/is ready
T+238ms: response marked complete/ended

As you can see, the headers arrive uncharacteristically fast, while the response body comes in at a believable time after the request is sent. Even if we consider that the connection is already established, and all we're doing is picking up an open connection and putting the request through, 1ms is still an unreasonable response time to already have the headers back.

If I measure the system time in microseconds, things get even stranger, and the time from starting the request to getting the headers drops to an incredibly low ~80uS, which, if my math is right, would suggest GitHub has a direct fiber optic connection to my house that is under 25km long. :)

I suspect something simple is responsible here for the incongruence, but I just can't seem to figure it out. :(

Return ErrorResponse instead of raise

I was wondering if you would accept a PR to change the behavior when ibrowse sends an error tuple, httpotion would return an %ErrorResponse{} struct. It could still raise an error if the ! version of the function is called.

The main reason for this is to avoid using costly try/rescue in the calling code. Another reason, is to keep it idiomatic with other erlang/elixir libraries (least amount of surprise)

Thoughts?

Thanks for putting the work into this library!

Brad

Elixir 1.3 Support

After running mix on the latest Elixir update (1.3), I get the following error:

==> httpotion
Compiling 1 file (.ex)

== Compilation error on file lib/httpotion.ex ==
** (ArgumentError) Access is not a protocol
    (elixir) lib/protocol.ex:76: Protocol.assert_protocol!/2
    lib/httpotion.ex:292: (module)
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6

I attempted to pinpoint the issue to no avail, hopefully you can.

Thanks!

ibrowse hex package..interest?

I depend on HTTPotion and it would be nice if we could get ibrowse into hex to use that instead of git deps. Any interest?

Take care,
Brad

Does not respect `:follow_redirects` from `process_options`

When HTTPotion.Base is extended to create an API client, that API client will not follow redirects even if :follow_redirects is set to true by process_options. For example, the below API client will not follow redirects:

defmodule RedirectingClient do
  use HTTPotion.Base

  def process_options(options) do
    Keyword.put(options, :follow_redirects, true)
  end
end

Why do I get a Elixir.HTTPotion.HTTPError: retry_later?

Here's an example:

** When Server state == []
** Reason for termination ==
** {{'Elixir.HTTPotion.HTTPError','__exception__',<<"retry_later">>},
    [{'Elixir.HTTPotion',request,5,
         [{file,
              "/Users/rambo/code/elixir-code/exrachnid/deps/httpotion/lib/httpotion.ex"},
          {line,134}]},
     {'Elixir.Exrachnid.Worker',handle_cast,2,
         [{file,
              "/Users/rambo/code/elixir-code/exrachnid/lib/exrachnid/worker.ex"},
          {line,38}]},
     {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

=ERROR REPORT==== 30-Dec-2013::15:57:13 ===
** Generic server <0.1062.0> terminating
** Last message in was {'$gen_cast',
                           {crawl,
                               <<"http://www.foodpanda.sg/restaurants/index/cuisines/singaporean">>}}

Basically I'm writing a webcrawler. At the beginning of the crawl, I'm already encountering this error. What would be a reasonable of handling this? Right of the bat, I'm thinking just catching this particular error and placing it back into the queue.

Thanks!

How to handle async messages

I'm trying to handle async messages in my project, my first attempt was place a receive block inside module instead of function, the code never gets compiled.

receive do

    %HTTPotion.AsyncChunk{ id: _id, chunk: _chunk } ->
        IO.puts _chunk |> to_string
end

What I'm doing wrong? I'm assuming [stream_to: self] is sending messages to the module and a receive block should do the trick, but I can't compile the module.

Should I spawn a separate process for this module or create another module to handle the async messages?

Note: The module is cowboy request handler.

Please Bump patch version with the code that fixes the compiler warnings

Hi

We have compiler warnings as errors in our code projects and as such, the fix you have made to httpotion.ex would greatly help us, but you didn't bump the version number to 3.0.1 or higher and so we cannot get this without referencing the GitHub repo in our mix.exs file which we usually don't support.

Can you release this patch soon please?

Many thanks
Mike

HTTPotion.process_headers loses multiple values for header keys

HTTPotion.process_headers uses :orddict.from_list, to return a dictionary - but the dictionary loses information - ie,
[{"Set-Cookie","first=foo"},{"Set-Cookie", "second=bar"}]
becomes just
{"Set-Cookie": "second=bar"}

A structure like
{"Set-Cookie": ["first=foo","second=bar"]}
would be better.

Unexpected tcp_closed Error When Making GET Request

When I use HTTPotion to make an HTTP GET request to a static file server, I get a successful response. However, it crashes whatever I am running and spits out an error message like so:


=ERROR REPORT==== 17-Dec-2013::21:44:46 ===
** Generic server <0.116.0> terminating 
** Last message in was {tcp_closed,#Port<0.4249>}
** When Server state == {state,"localhost",8000,5000,#Ref<0.0.0.745>,false,
                               undefined,[],false,#Port<0.4249>,false,[],
                               {[],[]},
                               undefined,idle,undefined,<<>>,0,0,[],undefined,
                               undefined,true,undefined,false,undefined,
                               undefined,<<>>,undefined,false,196623,1,
                               undefined}
** Reason for termination == 
** connection_closed

Cannot make any get requests

When I try to make a get request (for example HTTPotion.get("http://www.google.com")) I get the following error / stacktrace:

** (ArgumentError) argument error
 stacktrace:
   :ets.lookup(:ibrowse_lb, {'www.google.nl', 80})
   src/ibrowse.erl:317: :ibrowse.send_req/6
   deps/httpotion/lib/httpotion.ex:123: HTTPotion.request/5
   lib/octolixir/parser.ex:11: Octolixir.Parser.parse_gist/1
   test/parser_test.exs:14: ParserTest."test parse_gist should return succesfully with a valid id"/1

Anyone knows what's going on?

How to use httpotion to `POST` a json body

Hello, My question is how to send a post request to server with a json body. This is my attemp.

header = [{"Accept", "application/json"},
              {"Content-Type", "application/json"}
             ]    
body = "{\"image\":{\"image_data\": \"#{base64data}\}}"
response = HTTPotion.post("#{@host}/api/image/upload", [body: body, header: header])

On the phoenix server, When I look at the console, there is an empty parameter.

[debug] Processing by ImageQuickShare.ImageController.upload/2
  Parameters: %{}
  Pipelines: [:api]

Really need your answers !

Fails to compile in 0.13

I don't think it's a big deal, but httpotion presently fails to compile in 0.13. Here's the error if you want to see it:

* Compiling httpotion
== Compilation error on file lib/httpotion.ex ==
could not compile dependency httpotion, mix compile failed. You can recompile this dependency with `mix deps.compile httpotion` or update it with `mix deps.update httpotion`
** (SyntaxError) deps/httpotion/lib/httpotion.ex:9: invalid token: \Ahttps?:\/\// do
    (elixir) lib/kernel/parallel_compiler.ex:91: anonymous fn/3 in Kernel.ParallelCompiler.spawn_compilers/8

Anyway, thought it might be helpful to have this reported. Thanks!

basic_auth doesn't appear to work

from iex

HTTPotion.get("http://localhost:8990", [ basic_auth: {"abc", "foobarbaz"} ])

netcat output

GET / HTTP/1.1
Host: localhost:8990
Content-Length: 0

from curl

curl -u "abc":"foobarbaz" http://localhost:8990

netcat output

GET / HTTP/1.1
Authorization: Basic YWJjOmZvb2JhcmJheg==
User-Agent: curl/7.37.1
Host: localhost:8990
Accept: */*

process_response_body not called when async?

I'm super new to Elixir, so this may be by design, or it may be something I'm doing wrong, but it appears that process_response_body isn't called on the chunk in async requests, whereas it does get called in a synchronous request. What's the reason for this discrepancy?

Provide example of sending nested form data

It would be great to have an example of how to properly form a request with nested data.

endpoint = "http://localhost:8080/form"
headers = ["User-Agent": "Mozilla/5/0...",  "Content-Type": "application/x-www-form-urlencoded"]

HTTPotion.post endpoint, [body: [model: [attribute_one: value_one, attribute_two: value_two]], headers: headers]

It's not entirely clear why this doesn't work. This is the resulting error message:
(Protocol.UndefinedError) protocol String.Chars not implemented for %HTTPotion.ErrorResponse

Warning with Elixir 1.0.0

Including HTTPotion in a project using Elixir 1.0.0 results in a warning:

warning: the dependency httpotion requires Elixir "~> 0.13.3 or ~> 0.14.0" but you are running on v1.0.0

I haven't noticed any issue with Elixir 1.0.0. Maybe the deps can be changed to avoid this warning?

ibrowse fail to compile on Erlang 18

==> ibrowse (compile)
Compiling src/ibrowse_lib.erl failed:
src/ibrowse_lib.erl:371: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
ERROR: compile failed while processing /home/ronen/code/octo/deps/ibrowse: rebar_abort
** (Mix) Could not compile dependency ibrowse, /home/ronen/.mix/rebar command failed. If you want to recompile this dependency, please run: mix deps.compile ibrowse

Is the only solution is to use Erlang 17?

Unicode support?

I talk to a server using HTTP get and the response is an UTF-8 encoded JSON string. When decoding it with Poison, special unicode characters do not get decoded in the right way, so I and up with odd output. Can we do something about this or is this an issue with ibrowse?

https://github.com/edgurgel/httpoison does not have this problem – it in turn uses hackney instead of ibrowse

HTTP redirects are broken

:follow_redirects will not be picked up from process_options/1 or the default option from the application env.

Also the semantics of HTTP redirects are not followed, e.g. 303 should be a GET redirect, HTTPotion uses the same HTTP method.

push new version to hex.pm

Would it be possible to push a new version to hex.pm now that the library is updated for elixir 1.0?

Thanks!
Adam

Can we have a release please?

The work done in c2ed54f will be useful to use. We can remove our exception handling of HTTPError and add ErrorResponse as just one more possible return value in our case statement.

Would you be able to release what is currently in master please?

Accept header

I'm not too familiar with HTTP headers, but I needed to pass in an "Accept: /" header to make one of my GET requests to work. Is there a reason why you do not provide default values for common headers?

ITunes sandbox url Seg faults erlang vm.

Yeah, you read that right. I can't believe it myself. Here's some of the pertinent information I've gathered:

$ iex
Erlang/OTP 17 [erts-6.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)

iex> HTTPotion.start
{:ok, []}
iex> HTTPotion.post("https://sandbox.itunes.apple.com/verifyReceipt", [body: "", headers: ["Content-Type": "application/json"]])
28805 Segmentation fault: 11

I've tried it with httpotion 2.0, 2.1, and master. All the same result.

Side note: "https://buy.itunes.apple.com/verifyReceipt" works fine

Empty status_code returned by get

If I run this code:

https://gist.github.com/pragdave/5446949

I see the following. Notice that status code seems to be [], where I was expecting 200. What am I doing wrong?

Dave

[]
["Access-Control-Allow-Credentials": "true", "Access-Control-Allow-Origin": "*", 
"Access-Control-Expose-Headers": "Link, X-RateLimit-Limit, X-RateLimit-Remaining,
 X-OAuth-Scopes, X-Accepted-OAuth-Scopes", "Cache-Control": "public, max-age=60, 
s-maxage=60", Connection: "keep-alive", "Content-Length": "68613", "Content-Type":
 "application/json; charset=utf-8", Date: "Tue, 23 Apr 2013 20:04:06 GMT", 
ETag: "\"4d720f3ae4ef8721a3862e19f34877a2\"", "Last-Modified": 
"Tue, 23 Apr 2013 20:02:20 GMT", 
Link: "<https://api.github.com/repositories/1234714/issues?page=2>; rel=\"next\", 
<https://api.github.com/repositories/1234714/issues?page=2>; 
rel=\"last\"", Server: "GitHub.com", 
Status: "200 OK", Vary: "Accept-Encoding", "X-Content-Type-Options": "nosniff", 
"X-GitHub-Media-Type": "github.beta; format=json", "X-RateLimit-Limit": "60", 
"X-RateLimit-Remaining": "53"]

Tests failing due to connection close instead of keep-alive

13 test failures. In my test run it expected keep-alive and got close.

I tried calling httpbin.org using curl and confirmed it's using HTTP/1.1 for the request, but the response is HTTP/1.0.

 13) test post binary body (HTTPotionTest)
     test/httpotion_test.exs:21
     Assertion with == failed
     code: response.headers()[:Connection] == "keep-alive"
     lhs:  "close"
     rhs:  "keep-alive"
     stacktrace:
       test/httpotion_test.exs:102: HTTPotionTest.assert_response/2
       test/httpotion_test.exs:24

Finished in 20.7 seconds (0.2s on load, 20.5s on tests)
16 tests, 13 failures

Compilation error with Elixir v1.3.0-rc.0

==> httpotion
Compiling 1 file (.ex)

== Compilation error on file lib/httpotion.ex ==
** (ArgumentError) Access is not a protocol
    (elixir) lib/protocol.ex:76: Protocol.assert_protocol!/2
    lib/httpotion.ex:292: (module)
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6

Could not compile

  • Compiling httpotion
    == Compilation error on file lib/httpotion.ex ==
    could not compile dependency httpotion, mix compile failed. If you want to recompile this dependency, please run: mix deps.compile httpotion
    ** (CompileError) deps/httpotion/lib/httpotion.ex:120: function to_string/1 undefined
    lists.erl:1323: :lists.foreach/2
    /private/tmp/elixir-Juds/lib/elixir/lib/kernel/parallel_compiler.ex:82: Kernel.ParallelCompiler."-spawn_compilers/8-fun-0-"/3

(I'm using Elixir 0.10.2-dev)

Example doesn't work

iex(2)> HTTPotion.get "httpbin.org/get"
** (ArgumentError) argument error
    (stdlib) :ets.lookup(:ibrowse_lb, {'httpbin.org', 80})
             /Users/jcoyne/workspace/phoenix_recipe/deps/ibrowse/src/ibrowse.erl:328: :ibrowse.send_req/6
             lib/httpotion.ex:355: HTTPotion.request/3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.