Giter VIP home page Giter VIP logo

halite's Introduction

halite-logo

Halite

Language Tag Source Document Build Status

HTTP Requests with a chainable REST API, built-in sessions and middleware written by Crystal. Inspired from the awesome Ruby's HTTP/RESTClient and Python's requests.

Build in Crystal version >= 1.0.0, this document valid with latest commit.

Index

Installation

Add this to your application's shard.yml:

dependencies:
  halite:
    github: icyleaf/halite

Usage

require "halite"

Making Requests

Make a GET request:

# Direct get url
Halite.get("http://httpbin.org/get")

# Support NamedTuple as query params
Halite.get("http://httpbin.org/get", params: {
  language: "crystal",
  shard: "halite"
})

# Also support Hash as query params
Halite.get("http://httpbin.org/get", headers: {
    "Private-Token" => "T0k3n"
  }, params: {
    "language" => "crystal",
    "shard" => "halite"
  })

# And support chainable
Halite.header(private_token: "T0k3n")
      .get("http://httpbin.org/get", params: {
        "language" => "crystal",
        "shard" => "halite"
      })

See also all chainable methods.

Many other HTTP methods are available as well:

  • get
  • head
  • post
  • put
  • delete
  • patch
  • options

Passing Parameters

Query string parameters

Use the params argument to add query string parameters to requests:

Halite.get("http://httpbin.org/get", params: { "firstname" => "Olen", "lastname" => "Rosenbaum" })

Form data

Use the form argument to pass data serialized as form encoded:

Halite.post("http://httpbin.org/post", form: { "firstname" => "Olen", "lastname" => "Rosenbaum" })

File uploads (via form data)

To upload files as if form data, construct the form as follows:

Halite.post("http://httpbin.org/post", form: {
  "username" => "Quincy",
  "avatar" => File.open("/Users/icyleaf/quincy_avatar.png")
})

It is possible to upload multiple files:

Halite.post("http://httpbin.org/post", form: {
  photos: [
    File.open("/Users/icyleaf/photo1.png"),
    File.open("/Users/icyleaf/photo2.png")
  ],
  album_name: "samples"
})

Or pass the name with []:

Halite.post("http://httpbin.org/post", form: {
  "photos[]" => [
    File.open("/Users/icyleaf/photo1.png"),
    File.open("/Users/icyleaf/photo2.png")
  ],
  "album_name" => "samples"
})

Multiple files can also be uploaded using both ways above, it depend on web server.

JSON data

Use the json argument to pass data serialized as body encoded:

Halite.post("http://httpbin.org/post", json: { "firstname" => "Olen", "lastname" => "Rosenbaum" })

Raw String

Use the raw argument to pass raw string as body and set the Content-Type manually:

# Set content-type to "text/plain" by default
Halite.post("http://httpbin.org/post", raw: "name=Peter+Lee&address=%23123+Happy+Ave&language=C%2B%2B")

# Set content-type manually
Halite.post("http://httpbin.org/post",
  headers: {
    "content-type" => "application/json"
  },
  raw: %Q{{"name":"Peter Lee","address":"23123 Happy Ave","language":"C++"}}
)

Passing advanced options

Auth

Use the #basic_auth method to perform HTTP Basic Authentication using a username and password:

Halite.basic_auth(user: "user", pass: "p@ss").get("http://httpbin.org/get")

# We can pass a raw authorization header using the auth method:
Halite.auth("Bearer dXNlcjpwQHNz").get("http://httpbin.org/get")

User Agent

Use the #user_agent method to overwrite default one:

Halite.user_agent("Crystal Client").get("http://httpbin.org/user-agent")

Headers

Here are two way to passing headers data:

1. Use the #headers method
Halite.headers(private_token: "T0k3n").get("http://httpbin.org/get")

# Also support Hash or NamedTuple
Halite.headers({ "private_token" => "T0k3n" }).get("http://httpbin.org/get")

# Or
Halite.headers({ private_token: "T0k3n" }).get("http://httpbin.org/get")
2. Use the headers argument in the available request method:
Halite.get("http://httpbin.org/anything" , headers: { private_token: "T0k3n" })

Halite.post("http://httpbin.org/anything" , headers: { private_token: "T0k3n" })

Cookies

Passing cookies in requests

The Halite.cookies option can be used to configure cookies for a given request:

Halite.cookies(session_cookie: "6abaef100b77808ceb7fe26a3bcff1d0")
      .get("http://httpbin.org/headers")
Get cookies in requests

To obtain the cookies(cookie jar) for a given response, call the #cookies method:

r = Halite.get("http://httpbin.org/cookies?set?session_cookie=6abaef100b77808ceb7fe26a3bcff1d0")
pp r.cookies
# => #<HTTP::Cookies:0x10dbed980 @cookies={"session_cookie" =>#<HTTP::Cookie:0x10ec20f00 @domain=nil, @expires=nil, @extension=nil, @http_only=false, @name="session_cookie", @path="/", @secure=false, @value="6abaef100b77808ceb7fe26a3bcff1d0">}>

Redirects and History

Automatically following redirects

The Halite.follow method can be used for automatically following redirects(Max up to 5 times):

# Set the cookie and redirect to http://httpbin.org/cookies
Halite.follow
      .get("http://httpbin.org/cookies/set/name/foo")
Limiting number of redirects

As above, set over 5 times, it will raise a Halite::TooManyRedirectsError, but you can change less if you can:

Halite.follow(2)
      .get("http://httpbin.org/relative-redirect/5")
Disabling unsafe redirects

It only redirects with GET, HEAD request and returns a 300, 301, 302 by default, otherwise it will raise a Halite::StateError. We can disable it to set :strict to false if we want any method(verb) requests, in which case the GET method(verb) will be used for that redirect:

Halite.follow(strict: false)
      .post("http://httpbin.org/relative-redirect/5")
History

Response#history property list contains the Response objects that were created in order to complete the request. The list is ordered from the oldest to most recent response.

r = Halite.follow
          .get("http://httpbin.org/redirect/3")

r.uri
# => http://httpbin.org/get

r.status_code
# => 200

r.history
# => [
#      #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/relative-redirect/2" ...>,
#      #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/relative-redirect/1" ...>,
#      #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/get" ...>,
#      #<Halite::Response HTTP/1.1 200 OK    {"Content-Type" => "application/json" ...>
#    ]

NOTE: It contains the Response object if you use history and HTTP was not a 30x, For example:

r = Halite.get("http://httpbin.org/get")
r.history.size # => 0

r = Halite.follow
          .get("http://httpbin.org/get")
r.history.size # => 1

Timeout

By default, the Halite does not enforce timeout on a request. We can enable per operation timeouts by configuring them through the chaining API.

The connect timeout is the number of seconds Halite will wait for our client to establish a connection to a remote server call on the socket.

Once our client has connected to the server and sent the HTTP request, the read timeout is the number of seconds the client will wait for the server to send a response.

# Separate set connect and read timeout
Halite.timeout(connect: 3.0, read: 2.minutes)
      .get("http://httpbin.org/anything")

# Boath set connect and read timeout
# The timeout value will be applied to both the connect and the read timeouts.
Halite.timeout(5)
      .get("http://httpbin.org/anything")

HTTPS

The Halite supports HTTPS via Crystal's built-in OpenSSL module. All you have to do in order to use HTTPS is pass in an https://-prefixed URL.

To use client certificates, you can pass in a custom OpenSSL::SSL::Context::Client object containing the certificates you wish to use:

tls = OpenSSL::SSL::Context::Client.new
tls.ca_certificates = File.expand_path("~/client.crt")
tls.private_key = File.expand_path("~/client.key")

Halite.get("https://httpbin.org/anything", tls: tls)

Response Handling

After an HTTP request, Halite::Response object have several useful methods. (Also see the API documentation).

  • #body: The response body.
  • #body_io: The response body io only available in streaming requests.
  • #status_code: The HTTP status code.
  • #content_type: The content type of the response.
  • #content_length: The content length of the response.
  • #cookies: A HTTP::Cookies set by server.
  • #headers: A HTTP::Headers of the response.
  • #links: A list of Halite::HeaderLink set from headers.
  • #parse: (return value depends on MIME type) parse the body using a parser defined for the #content_type.
  • #to_a: Return a Hash of status code, response headers and body as a string.
  • #to_raw: Return a raw of response as a string.
  • #to_s: Return response body as a string.
  • #version: The HTTP version.

Response Content

We can read the content of the server's response by call #body:

r = Halite.get("http://httpbin.org/user-agent")
r.body
# => {"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"}

The gzip and deflate transfer-encodings are automatically decoded for you. And requests will automatically decode content from the server. Most unicode charsets are seamlessly decoded.

JSON Content

There’s also a built-in a JSON adapter, in case you’re dealing with JSON data:

r = Halite.get("http://httpbin.org/user-agent")
r.parse("json")
r.parse # simplily by default
# => {
# =>   "user-agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
# => }

Parsing Content

Halite::Response has a MIME type adapter system that you can use a decoder to parse the content, we can inherit Halite::MimeTypes::Adapter to make our adapter:

# Define a MIME type adapter
class YAMLAdapter < Halite::MimeType::Adapter
  def decode(string)
    YAML.parse(string)
  end

  def encode(obj)
    obj.to_yaml
  end
end

# Register to Halite to invoke
Halite::MimeType.register YAMLAdapter.new, "application/x-yaml", "yaml", "yml"

# Test it!
r = Halite.get "https://raw.githubusercontent.com/icyleaf/halite/master/shard.yml"
r.parse("yaml") # or "yml"
# => {"name" => "halite", "version" => "0.4.0", "authors" => ["icyleaf <[email protected]>"], "crystal" => "0.25.0", "license" => "MIT"}

Binary Data

Store binary data (eg, application/octet-stream) to file, you can use streaming requests:

Halite.get("https://github.com/icyleaf/halite/archive/master.zip") do |response|
  filename = response.filename || "halite-master.zip"
  File.open(filename, "w") do |file|
    IO.copy(response.body_io, file)
  end
end

Error Handling

  • For any status code, a Halite::Response will be returned.
  • If request timeout, a Halite::TimeoutError will be raised.
  • If a request exceeds the configured number of maximum redirections, a Halite::TooManyRedirectsError will raised.
  • If request uri is http and configured tls context, a Halite::RequestError will raised.
  • If request uri is invalid, a Halite::ConnectionError/Halite::UnsupportedMethodError/Halite::UnsupportedSchemeError will raised.

Raise for status code

If we made a bad request(a 4xx client error or a 5xx server error response), we can raise with Halite::Response.raise_for_status.

But, since our status_code was not 4xx or 5xx, it returns nil when we call it:

urls = [
  "https://httpbin.org/status/404",
  "https://httpbin.org/status/500?foo=bar",
  "https://httpbin.org/status/200",
]

urls.each do |url|
  r = Halite.get url
  begin
    r.raise_for_status
    p r.body
  rescue ex : Halite::ClientError | Halite::ServerError
    p "[#{ex.status_code}] #{ex.status_message} (#{ex.class})"
  end
end

# => "[404] not found error with url: https://httpbin.org/status/404 (Halite::Exception::ClientError)"
# => "[500] internal server error error with url: https://httpbin.org/status/500?foo=bar (Halite::Exception::ServerError)"
# => ""

Middleware

Halite now has middleware (a.k.a features) support providing a simple way to plug in intermediate custom logic in your HTTP client, allowing you to monitor outgoing requests, incoming responses, and use it as an interceptor.

Available features:

  • Logging (Yes, logging is based on feature, cool, aha!)
  • Local Cache (local storage, speed up in development)

Write a simple feature

Let's implement simple middleware that prints each request:

class RequestMonister < Halite::Feature
  @label : String
  def initialize(**options)
    @label = options.fetch(:label, "")
  end

  def request(request) : Halite::Request
    puts @label
    puts request.verb
    puts request.uri
    puts request.body

    request
  end

  Halite.register_feature "request_monster", self
end

Then use it in Halite:

Halite.use("request_monster", label: "testing")
      .post("http://httpbin.org/post", form: {name: "foo"})

# Or configure to client
client = Halite::Client.new do
  use "request_monster", label: "testing"
end

client.post("http://httpbin.org/post", form: {name: "foo"})

# => testing
# => POST
# => http://httpbin.org/post
# => name=foo

Write a interceptor

Halite's killer feature is the interceptor, Use Halite::Feature::Chain to process with two result:

  • next: perform and run next interceptor
  • return: perform and return

So, you can intercept and turn to the following registered features.

class AlwaysNotFound < Halite::Feature
  def intercept(chain)
    response = chain.perform
    response = Halite::Response.new(chain.request.uri, 404, response.body, response.headers)
    chain.next(response)
  end

  Halite.register_feature "404", self
end

class PoweredBy < Halite::Feature
  def intercept(chain)
    if response = chain.response
      response.headers["X-Powered-By"] = "Halite"
      chain.return(response)
    else
      chain
    end
  end

  Halite.register_feature "powered_by", self
end

r = Halite.use("404").use("powered_by").get("http://httpbin.org/user-agent")
r.status_code               # => 404
r.headers["X-Powered-By"]   # => Halite
r.body                      # => {"user-agent":"Halite/0.6.0"}

For more implementation details about the feature layer, see the Feature class and examples and specs.

Advanced Usage

Configuring

Halite provides a traditional way to instance client, and you can configure any chainable methods with block:

client = Halite::Client.new do
  # Set basic auth
  basic_auth "username", "password"

  # Enable logging
  logging true

  # Set timeout
  timeout 10.seconds

  # Set user agent
  headers user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
end

# You also can configure in this way
client.accept("application/json")

r = client.get("http://httpbin.org/get")

Endpoint

No more given endpoint per request, use endpoint will make the request URI shorter, you can set it in flexible way:

client = Halite::Client.new do
  endpoint "https://gitlab.org/api/v4"
  user_agent "Halite"
end

client.get("users")       # GET https://gitlab.org/api/v4/users

# You can override the path by using an absolute path
client.get("/users")      # GET https://gitlab.org/users

Sessions

As like requests.Session(), Halite built-in session by default.

Let's persist some cookies across requests:

client = Halite::Client.new
client.get("http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0")
client.get("http://httpbin.org/cookies")
# => 2018-06-25 18:41:05 +08:00 | request | GET    | http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0
# => 2018-06-25 18:41:06 +08:00 | response | 302    | http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0 | text/html
# => <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
# => <title>Redirecting...</title>
# => <h1>Redirecting...</h1>
# => <p>You should be redirected automatically to target URL: <a href="/cookies">/cookies</a>.  If not click the link.
# => 2018-06-25 18:41:06 +08:00 | request | GET    | http://httpbin.org/cookies
# => 2018-06-25 18:41:07 +08:00 | response | 200    | http://httpbin.org/cookies | application/json
# => {"cookies":{"private_token":"6abaef100b77808ceb7fe26a3bcff1d0"}}

All it support with chainable methods in the other examples list in requests.Session.

Note, however, that chainable methods will not be persisted across requests, even if using a session. This example will only send the cookies or headers with the first request, but not the second:

client = Halite::Client.new
r = client.cookies("username": "foobar").get("http://httpbin.org/cookies")
r.body # => {"cookies":{"username":"foobar"}}

r = client.get("http://httpbin.org/cookies")
r.body # => {"cookies":{}}

If you want to manually add cookies, headers (even features etc) to your session, use the methods start with with_ in Halite::Options to manipulate them:

r = client.get("http://httpbin.org/cookies")
r.body # => {"cookies":{}}

client.options.with_cookie("username": "foobar")
r = client.get("http://httpbin.org/cookies")
r.body # => {"cookies":{"username":"foobar"}}

Streaming Requests

Similar to HTTP::Client usage with a block, you can easily use same way, but Halite returns a Halite::Response object:

r = Halite.get("http://httpbin.org/stream/5") do |response|
  response.status_code                  # => 200
  response.body_io.each_line do |line|
    puts JSON.parse(line)               # => {"url" => "http://httpbin.org/stream/5", "args" => {}, "headers" => {"Host" => "httpbin.org", "Connection" => "close", "User-Agent" => "Halite/0.8.0", "Accept" => "*/*", "Accept-Encoding" => "gzip, deflate"}, "id" => 0_i64}
  end
end

Warning:

body_io is avaiabled as an IO and not reentrant safe. Might throws a "Nil assertion failed" exception if there is no data in the IO (such like head requests). Calling this method multiple times causes some of the received data being lost.

One more thing, use streaming requests the response will always enable redirect automatically.

Logging

Halite does not enable logging on each request and response too. We can enable per operation logging by configuring them through the chaining API.

By default, Halite will logging all outgoing HTTP requests and their responses(without binary stream) to STDOUT on DEBUG level. You can configuring the following options:

  • logging: Instance your Halite::Logging::Abstract, check Use the custom logging.
  • format: Output format, built-in common and json, you can write your own.
  • file: Write to file with path, works with format.
  • filemode: Write file mode, works with format, by default is a. (append to bottom, create it if file is not exist)
  • skip_request_body: By default is false.
  • skip_response_body: By default is false.
  • skip_benchmark: Display elapsed time, by default is false.
  • colorize: Enable colorize in terminal, only apply in common format, by default is true.

NOTE: format (file and filemode) and logging are conflict, you can not use both.

Let's try with it:

# Logging json request
Halite.logging
      .get("http://httpbin.org/get", params: {name: "foobar"})

# => 2018-06-25 18:33:14 +08:00 | request  | GET    | http://httpbin.org/get?name=foobar
# => 2018-06-25 18:33:15 +08:00 | response | 200    | http://httpbin.org/get?name=foobar | 381.32ms | application/json
# => {"args":{"name":"foobar"},"headers":{"Accept":"*/*","Accept-Encoding":"gzip, deflate","Connection":"close","Host":"httpbin.org","User-Agent":"Halite/0.3.2"},"origin":"60.206.194.34","url":"http://httpbin.org/get?name=foobar"}

# Logging image request
Halite.logging
      .get("http://httpbin.org/image/png")

# => 2018-06-25 18:34:15 +08:00 | request  | GET    | http://httpbin.org/image/png
# => 2018-06-25 18:34:15 +08:00 | response | 200    | http://httpbin.org/image/png | image/png

# Logging with options
Halite.logging(skip_request_body: true, skip_response_body: true)
      .post("http://httpbin.org/get", form: {image: File.open("halite-logo.png")})

# => 2018-08-28 14:33:19 +08:00 | request  | POST   | http://httpbin.org/post
# => 2018-08-28 14:33:21 +08:00 | response | 200    | http://httpbin.org/post | 1.61s | application/json

JSON-formatted logging

It has JSON formatted for developer friendly logging.

Halite.logging(format: "json")
      .get("http://httpbin.org/get", params: {name: "foobar"})

Write to a log file

# Write plain text to a log file
Log.setup("halite.file", backend: Log::IOBackend.new(File.open("/tmp/halite.log", "a")))
Halite.logging(for: "halite.file", skip_benchmark: true, colorize: false)
      .get("http://httpbin.org/get", params: {name: "foobar"})

# Write json data to a log file
Log.setup("halite.file", backend: Log::IOBackend.new(File.open("/tmp/halite.log", "a")))
Halite.logging(format: "json", for: "halite.file")
      .get("http://httpbin.org/get", params: {name: "foobar"})

# Redirect *all* logging from Halite to a file:
Log.setup("halite", backend: Log::IOBackend.new(File.open("/tmp/halite.log", "a")))

Use the custom logging

Creating the custom logging by integration Halite::Logging::Abstract abstract class. Here has two methods must be implement: #request and #response.

class CustomLogging < Halite::Logging::Abstract
  def request(request)
    @logger.info { "| >> | %s | %s %s" % [request.verb, request.uri, request.body] }
  end

  def response(response)
    @logger.info { "| << | %s | %s %s" % [response.status_code, response.uri, response.content_type] }
  end
end

# Add to adapter list (optional)
Halite::Logging.register "custom", CustomLogging.new

Halite.logging(logging: CustomLogging.new)
      .get("http://httpbin.org/get", params: {name: "foobar"})

# We can also call it use format name if you added it.
Halite.logging(format: "custom")
      .get("http://httpbin.org/get", params: {name: "foobar"})

# => 2017-12-13 16:40:13 +08:00 | >> | GET | http://httpbin.org/get?name=foobar
# => 2017-12-13 16:40:15 +08:00 | << | 200 | http://httpbin.org/get?name=foobar application/json

Local Cache

Local cache feature is caching responses easily with Halite through an chainable method that is simple and elegant yet powerful. Its aim is to focus on the HTTP part of caching and do not worrying about how stuff stored, api rate limiting even works without network(offline).

It has the following options:

  • file: Load cache from file. it conflict with path and expires.
  • path: The path of cache, default is "/tmp/halite/cache/"
  • expires: The expires time of cache, default is never expires.
  • debug: The debug mode of cache, default is true

With debug mode, cached response it always included some headers information:

  • X-Halite-Cached-From: Cache source (cache or file)
  • X-Halite-Cached-Key: Cache key with verb, uri and body (return with cache, not file passed)
  • X-Halite-Cached-At: Cache created time
  • X-Halite-Cached-Expires-At: Cache expired time (return with cache, not file passed)
Halite.use("cache").get "http://httpbin.org/anything"     # request a HTTP
r = Halite.use("cache").get "http://httpbin.org/anything" # request from local storage
r.headers                                                 # => {..., "X-Halite-Cached-At" => "2018-08-30 10:41:14 UTC", "X-Halite-Cached-By" => "Halite", "X-Halite-Cached-Expires-At" => "2018-08-30 10:41:19 UTC", "X-Halite-Cached-Key" => "2bb155e6c8c47627da3d91834eb4249a"}}

Link Headers

Many HTTP APIs feature Link headers. GitHub uses these for pagination in their API, for example:

r = Halite.get "https://api.github.com/users/icyleaf/repos?page=1&per_page=2"
r.links
# => {"next" =>
# =>   Halite::HeaderLink(
# =>    @params={},
# =>    @rel="next",
# =>    @target="https://api.github.com/user/17814/repos?page=2&per_page=2"),
# =>  "last" =>
# =>   Halite::HeaderLink(
# =>    @params={},
# =>    @rel="last",
# =>    @target="https://api.github.com/user/17814/repos?page=41&per_page=2")}

r.links["next"]
# => "https://api.github.com/user/17814/repos?page=2&per_page=2"

r.links["next"].params
# => {}

Help and Discussion

You can browse the API documents:

https://icyleaf.github.io/halite/

You can browse the all chainable methods:

https://icyleaf.github.io/halite/Halite/Chainable.html

You can browse the Changelog:

https://github.com/icyleaf/halite/blob/master/CHANGELOG.md

If you have found a bug, please create a issue here:

https://github.com/icyleaf/halite/issues/new

How to Contribute

Your contributions are always welcome! Please submit a pull request or create an issue to add a new question, bug or feature to the list.

All Contributors are on the wall.

You may also like

  • totem - Load and parse a configuration file or string in JSON, YAML, dotenv formats.
  • markd - Yet another markdown parser built for speed, Compliant to CommonMark specification.
  • poncho - A .env parser/loader improved for performance.
  • popcorn - Easy and Safe casting from one type to another.
  • fast-crystal - 💨 Writing Fast Crystal 😍 -- Collect Common Crystal idioms.

License

MIT License © icyleaf

halite's People

Contributors

007lva avatar donovanglover avatar icyleaf avatar imgbotapp avatar jkthorne avatar kalinon avatar megatux avatar oprypin avatar tylerwbrown avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

jkthorne 007lva megatux chipper1 yorci qszhu tylerwbrown oprypin lucasintel kalinon harryrh grkek

halite's Issues

request option values repeated when reusing clients

I'm not sure whether this is the expected behavior.

It's normal when not reusing client:

r = Halite.get("http://httpbin.org/get", params: {"foo"=> "bar"})
puts "1st response", r.body

r = Halite.get("http://httpbin.org/get", params: {"foo"=> "bar"})
puts "2nd response", r.body
Output 
1st response
{
  "args": {
    "foo": "bar"
  }, 
  "headers": {
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "Halite/0.10.0"
  }, 
  "origin": "xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx", 
  "url": "https://httpbin.org/get?foo=bar"
}
2nd response
{
  "args": {
    "foo": "bar"
  }, 
  "headers": {
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "Halite/0.10.0"
  }, 
  "origin": "xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx", 
  "url": "https://httpbin.org/get?foo=bar"
}

Reusing with absolute path:

client = Halite::Client.new do
  endpoint "http://httpbin.org"
end

r = client.get("/get", params: {"foo"=> "bar"})
puts "1st response", r.body

r = client.get("/get", params: {"foo"=> "bar"})
puts "2nd response", r.body
Output 
1st response
{
  "args": {
    "foo": "bar"
  }, 
  "headers": {
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "Halite/0.10.0"
  }, 
  "origin": "xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx", 
  "url": "https://httpbin.org/get?foo=bar"
}
2nd response
{
  "args": {
    "foo": [
      "bar", 
      "bar"
    ]
  }, 
  "headers": {
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "Halite/0.10.0"
  }, 
  "origin": "xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx", 
  "url": "https://httpbin.org/get?foo=bar&foo=bar"
}

The query parameters is repeated in this case.

Reusing client with relative path:

r = client.get("get", params: {"foo"=> "bar"})
puts "1st response", r.body

r = client.get("get", params: {"foo"=> "bar"})
puts "2nd response", r.body
Output 
1st response
{
  "args": {
    "foo": "bar"
  }, 
  "headers": {
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "Halite/0.10.0"
  }, 
  "origin": "xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx", 
  "url": "https://httpbin.org/get?foo=bar"
}
2nd response
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>

I guess the URL path is repeated in this case so 404 is returned.

Can POST JSON payloads with a sub hash with a value of type Array(String)

Hi, I think I've encountered a bug or an undocumented support for some JSON payloads.

If I try to perform this POST request Halite.post("https://my-app.com", json: {"asd" => { "asd" => ["asd", "sad"] }}) I end up with the following compiler error:

/lib/halite/src/halite/options.cr:356, line 84:
...
  68.       private def parse_json(raw : (Hash(String, _) | NamedTuple)?) : Hash(String, Options::Type)
  69.         new_json = {} of String => Type
  70.         return new_json unless json = raw
  71. 
  72.         if json.responds_to?(:each)
  73.           json.each do |k, v|
  74.             pp k, v
  75.             new_json[k.to_s] =
  76.               case v
  77.               when Array
  78.                 pp "CICCIO"
  79.                 v.each_with_object([] of Type) do |e, obj|
  80.                   obj << e.as(Type)
  81.                 end
  82.               when Hash
  83.                 v.each_with_object({} of String => Type) do |(ik, iv), obj|
> 84.                   obj[ik.to_s] = iv.as(Type)
  85.                 end
  86.               when NamedTuple
  87.                 hash = {} of String => Type
  88.                 v.each do |nk, nv|
  89.                   hash[nk.to_s] = nv.as(Type)
  90.                 end
  91.                 hash
  92.               else
  93.                 v.as(Type)
  94.               end
  95.           end
  96.         end
  97. 
  98.         new_json
  99.       end
 100.     

can't cast Array(String) to Halite::Options::Type

Btw I'm using the latest version of the shard, i.e. 0.7.2.
For any other information I remain obviously at total disposal :)

Set cookie from session

Is there a way to tell Halite to automatically update it's own cookie-jar when it get a Set-Cookie header?

client.get(URL_WITH_SET_COOKIE) # response cookie 123
client.get(OTHER_URL) # request cookie 123

JSON post issue

I have a cURL command that seems to work fine and also works from Postman, but I can seem to format it into Halite in such a way that it works.

The cURL is
curl --location --request POST 'https://api-qa.employers.com/DigitalAgencyServices/quote'
--header 'appKey: xxxxxx'
--header 'appToken: xxxxxxx'
--header 'Content-Type: application/json'
--data-raw '{"id": "baca57f1-8bcc-4e21-b26b-27f0f3567567", "effectiveDate": "2022-04-19", "expirationDate": "2022-05-19", "yearsInBusiness": 5, "agency": {"agencyCode": "0001878"}, "agent": {"customerNumber": "0001878-CR00034268"}, "namedInsureds": [{"name": "Test+Restaurant", "fein": "333333334", "locations": [{"primary": true, "numberOfEmployees": 3, "address": {"state": "MA", "zipCode": "01864"}, "rateClasses": [{"classCode": "907908", "classCodeDescription": "", "payrollAmount": 100000}]}]}], "questions": [{"questionCode": "EMPCO-1000", "value": "false"}, {"questionCode": "EMPCO-1100", "value": "false"}, {"questionCode": "EMPCO-1001", "value": "false"}, {"questionCode": "EMPCO-1101", "value": "false"}, {"questionCode": "EMPCO-1002", "value": "false"}, {"questionCode": "EMPCO-1007", "value": "false"}, {"questionCode": "EMPCO-1004", "value": "false"}, {"questionCode": "EMPCO-1104", "value": "false"}, {"questionCode": "EMPCO-1105", "value": "false"}, {"questionCode": "EMPCO-1005", "value": "false"}, {"questionCode": "EMPCO-1006", "value": "false"}, {"questionCode": "EMPCO-3001", "value": "false"}, {"questionCode": "EMPCO-3011", "value": "false"}, {"questionCode": "EMPCO-163", "value": "false"}, {"questionCode": "EMPCO-164", "value": "false"}, {"questionCode": "EMPCO-3012", "value": "false"}, {"questionCode": "EMPCO-3003", "value": "false"}, {"questionCode": "EMPCO-3009", "value": "false"}, {"questionCode": "EMPCO-3109", "value": "false"}, {"questionCode": "EMPCO-3209", "value": "false"}, {"questionCode": "EMPCO-3010", "value": "false"}, {"questionCode": "EMPCO-3013", "value": "false"}, {"questionCode": "EMPCO-512", "value": "false"}, {"questionCode": "EMPCO-3014", "value": "false"}, {"questionCode": "EMPCO-3015", "value": "false"}, {"questionCode": "EMPCO-3007", "value": "false"}, {"questionCode": "EMPCO-3008", "value": "false"}, {"questionCode": "EMPCO-3016", "value": "false"}, {"questionCode": "EMPCO-3017", "value": "false"}, {"questionCode": "EMPCO-3018", "value": "false"}, {"questionCode": "EMPCO-3118", "value": "false"}, {"questionCode": "EMPCO-3218", "value": "false"}, {"questionCode": "EMPCO-3019", "value": "false"}]}'

I've tried variations on

# Set content-type manually

response = Halite.post("https://api-qa.employers.com/DigitalAgencyServices/quote",
headers: {"content-type" => "application/json", "appKey=" => "xxxx", "appToken" => "xxxxx"},
raw: quote.to_json
)

where quote is the data in any number of different formats.

What would be the correct appoach?

Many thanks in advance!

Two memory leaks 

require "json"
require "http/client"
require "halite"
require "kemal"

URL = "http://localhost:3000"

case ARGV[0]?
when "server" # needs to run in a separate process (vs. just another fiber), otherwise the "too many open files" exception will not happen
    post "/" do |env|
        input = env.params.json["input"].as(String)
        args = env.params.json["args"]?
        # puts("Received #{input}")
        {"result" => "ok"}.to_json
    end
    Kemal.run do |config|
        server = config.server.not_nil!
        server.bind_tcp("0.0.0.0", URI.parse(URL).port.not_nil!)
    end

when "client"
    cnt = 0
    spawn do
        while true
            form = {"input" => "foo"}

            # HTTP::Client.post doesn't really allow to change the Content-Type; .exec does allow
            # res = HTTP::Client.exec("POST", URL, headers: HTTP::Headers{"Content-Type"=>"application/json"}, body: form.to_json) # (1)
            res = Halite.post(URL, json: form) # xor (2)

            # puts([res.status_code, JSON.parse(res.body)["result"]])
            # GC.collect # (3): comment in and out
            cnt += 1
            # sleep(0.02) # (4) remove sleep to speed things up
        end
    end
    while cnt < 100000 # put any number or endless loop
        v1 = `cat /proc/#{Process.pid}/status |grep VmRSS`.chomp
        v2 = `lsof -i`.split("\n").select(&.=~ /CLOSE_WAIT/).select(&.=~ /crystal/).size # exception if exceeds ulimit -n ("too many open files")
        puts([cnt, v1, v2])
        sleep(1)
    end
else
    puts("call with either 'server' or 'client'")
end

# halite w/o GC.collect: (2) -> exception
# [16951, "VmRSS:\t   93216 kB", 981]
# Unhandled exception in spawn: Hostname lookup for localhost failed: System error (Halite::Exception::ConnectionError)
# Unhandled exception: Could not create pipe: Too many open files (IO::Error)

# halite w/ GC.collect: (2) and (3) -> ~2kB leak per post
# [49976, "VmRSS:\t   98928 kB", 4]
# [99988, "VmRSS:\t  183928 kB", 3]

# http/client, w/o GC.collect: (1) -> no leak
# [50040, "VmRSS:\t   12284 kB", 2]
# [97617, "VmRSS:\t   12228 kB", 2]

You have to run the sample in two separate processes. If you run it in the default configuration as above (2), the "too many open files" exception shows up.
This can be worked around by inserting an GC.collect (3). Then the real memory leak shows up, every post consuming about two kilobytes of memory.
This compares against the standard crystal post method (1), which doesn't expose either weakness.

Using Crystal 1.2.1 [4e6c0f26e] (2021-10-21), LLVM: 10.0.0, Default target: x86_64-unknown-linux-gnu
and halite 0.12.0
on Ubuntu 18.04

TLS\SSL

Is there a way to pass in a context to use for SSL connection ?
I looked at Halite::Options but didn't find it.

Bug: can not use logging with bool 

Halite.logging(true).get("...")

# Or
Halite::Client.new
  logging true
end

It returns:

in src/halite/chainable.cr:390: no overload matches 'Halite::Options#with_logging' with types Bool, file: Nil, filemode: String, skip_request_body: Bool, skip_response_body: Bool, skip_benchmark: Bool, colorize: Bool
Overloads are:
 - Halite::Options#with_logging(format : String, **opts)
 - Halite::Options#with_logging(logging : Halite::Logging::Abstract)

This effected only in Crystal 0.28.

Endpoint joining is broken

halite/spec/support/mock_server.cr

Line 46 in ba22d2a

File.join(endpoint, path)

For endpoint http://example.com if I make a request http://other.example.com/foo, this will try to request http://example.com/http://other.example.com/foo. That's probably not good.

On Windows, what, maybe it'll request http://example.com\http://other.example.com/foo??

Pleeease just use URI#resolve. Its documentation is very good, so I don't need to explain its merits myself.

Bug when using `timeout()` inside `Halite::Client.new do`

I think this line has a bug because it looks different from other lines around it:

halite/src/halite/chainable.cr

Lines 223 to 225 in e7c7476

def timeout(timeout : Int32? | Float64? | Time::Span?)
timeout ? timeout(timeout, timeout, timeout) : branch
end

I have this code

c = Halite::Client.new do
  endpoint("https://example.com/")
  timeout(1.seconds)
  logging(skip_request_body: true, skip_response_body: true)
end

I get this error

 92 | timeout(1.seconds)
      ^------
Error: instantiating 'timeout(Time::Span)'


In lib/halite/src/halite/chainable.cr:224:54

 224 | timeout ? timeout(timeout, timeout, timeout) : branch
                                                      ^-----
Error: instantiating 'branch()'


In lib/halite/src/halite/client.cr:275:23

 275 | oneshot_options.merge!(options)
                       ^-----
Error: no overload matches 'Halite::Options#merge!' with type Nil

Overloads are:
 - Halite::Options#merge!(other : Halite::Options)

Some errors when building against crystal HEAD 

in src/protocols/http/crawler.cr:33: instantiating 'Halite::Client:Class#new()'

        client = Halite::Client.new
                                ^~~

in lib/halite/src/halite/client.cr:42: instantiating 'Halite::Options:Class#new()'

    def initialize(@options : Halite::Options = Options.new)
                                                        ^~~

in lib/halite/src/halite/options.cr:30: instantiating 'parse_headers(Hash(String, Nil))'

      @headers = parse_headers(options).merge!(default_headers)
                 ^~~~~~~~~~~~~

in lib/halite/src/halite/options.cr:171: no overload matches 'HTTP::Headers.escape' with type Nil
Overloads are:
 - Halite::Ext::HTTPHeaders::Escape#escape(data : Hash(String, _) | NamedTuple)

        HTTP::Headers.escape(headers)
                      ^~~~~~

feature: local cache

Better to debug and improve the speed for developing third party API.

Logic

  • cache key: hash url with params as cache key with MD5 or use given filename with path.
  • cache body: stores full response(status code, headers and body) to local file, at least stores the body of response.
  • cache directory: You can set it
  • expires time: stored forever if not set
  • update strategy: check cache related headers or hash body and compares, i don't know.

Cache key

generate algorithm md5(verb + url + body)

Cache response

Store response into two files:

  • metadata: Stored status_code and headers in JSON format.
  • body: Original body of response, so it can be binary data.

metadata JSON structure:

{
  "status_code": 200,
  "headers": {
    "content-type": "application/json",
  },
}

Expires time

Supports Int32, Time::Span is enough.

Examples

Use it with single call

Halite.cache.get "https://httpbin.ogr/get"
Halite.cache(directory: "cache", expired: 1.hour).get "https://httpbin.ogr/get"

# Loads the content of "cache/get.txt" 
Halite.cache(file: "get.txt").get "https://httpbin.ogr/get"

Use it with instance client:

client = Halite::Client.new do |options|
  options.cache = true
end

client = Halite::Client.new do |options|
  options.cache(directory: "cache", expired: 1.hour)
end

client.get "https://httpbin.ogr/get"

This request will store into cache path: "cache/{cache_key}" with two files:

metadata.json:

{
  "status_code": 200,
  "headers": {
    "content-type": "application/json",
  },
}

{cache_key}.cache

{"args":{"name":"foobar"},"headers":{"Accept":"*/*","Accept-Encoding":"gzip, deflate","Connection":"close","Host":"httpbin.org","User-Agent":"Halite/0.3.2"},"origin":"60.206.194.34","url":"http://httpbin.org/get?name=foobar"}

Load from file

It supports load cache from file, just passes file argument, Cache will load the content of file and return to Halite::Response no matter request whatever is.

PR

#35

Discuss: Decode json body with JSON logger?

before:

{
  "created_at": "2018-07-03T10:48:04+08:00:00",
  "entry": {
    "request": {
      "body": "name=icyleaf",
      "headers": {
        "User-Agent": "Halite/0.4.0",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Content-Type": "application/x-www-form-urlencoded"
      },
      "method": "POST",
      "url": "https://httpbin.org/post",
      "timestamp": "2018-07-03T10:48:05+08:00:00"
    },
    "response": {
      "body": "{\"args\":{},\"data\":\"\",\"files\":{},\"form\":{\"name\":\"icyleaf\"},\"headers\":{\"Accept\":\"*/*\",\"Accept-Encoding\":\"gzip, deflate\",\"Connection\":\"close\",\"Content-Length\":\"12\",\"Content-Type\":\"application/x-www-form-urlencoded\",\"Host\":\"httpbin.org\",\"User-Agent\":\"Halite/0.4.0\"},\"json\":null,\"origin\":\"60.206.194.34\",\"url\":\"https://httpbin.org/post\"}\n",
      "header": {
        "Connection": "keep-alive",
        "Server": "gunicorn/19.8.1",
        "Date": "Tue, 03 Jul 2018 02:48:05 GMT",
        "Content-Type": "application/json",
        "Content-Length": "333",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Credentials": "true",
        "Via": "1.1 vegur"
      },
      "status_code": 200,
      "http_version": "HTTP/1.1",
      "timestamp": "2018-07-03T10:48:05+08:00:00"
    }
  }
}

after:

{
  "created_at": "2018-07-03T10:48:04+08:00:00",
  "entry": {
    "request": {
      "body": "name=icyleaf",
      "headers": {
        "User-Agent": "Halite/0.4.0",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Content-Type": "application/x-www-form-urlencoded"
      },
      "method": "POST",
      "url": "https://httpbin.org/post",
      "timestamp": "2018-07-03T10:48:05+08:00:00"
    },
    "response": {
      "body": {
        "args": {},
        "data": "",
        "files": {},
        "form": {
          "name": "icyleaf"
        },
        "headers": {
          "Accept": "*/*",
          "Accept-Encoding": "gzip, deflate",
          "Connection": "close",
          "Content-Length": "12",
          "Content-Type": "application/x-www-form-urlencoded",
          "Host": "httpbin.org",
          "User-Agent": "Halite/0.4.0"
        },
        "json": null,
        "origin": "60.206.194.34",
        "url": "https://httpbin.org/post"
      },
      "header": {
        "Connection": "keep-alive",
        "Server": "gunicorn/19.8.1",
        "Date": "Tue, 03 Jul 2018 02:48:05 GMT",
        "Content-Type": "application/json",
        "Content-Length": "333",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Credentials": "true",
        "Via": "1.1 vegur"
      },
      "status_code": 200,
      "http_version": "HTTP/1.1",
      "timestamp": "2018-07-03T10:48:05+08:00:00"
    }
  }
}

path/filename missmatch

Found a missmatch:

Halite.post("http://127.0.0.1:3000/upload", form: {
  "username" => "Quincy",
  "avatar" => File.open("public/2.png")
})

were "public/2.png" becomes the filename, so, when trying to save this, i got:

Exception: Error opening file './data/uploads/public/2.png' with mode 'w': No such file or directory (Errno)

so "public/" is to much; it should only be "2.png"

and my path is correct with: './data/uploads/2.png' ;-)

Add endpoint as configuration option

It would be nice if we could specify an endpoint when creating a Halite::Client instance so that all requests going through that client reference that endpoint.

Something like:

cli = Halite::Client.new do |client|
  client.endpoint = "https://api.github.com"
end

response = cli.get("/orgs/octokit/repos")

Change tls to Bool or OpenSSL::SSL::Context::Client

Change tls to Bool | OpenSSL::SSL::Context::Client should make easy pass default ssl context if true given.

Halite.get url, tls: true

# | |
# | |
# \ /

client = HTTP::Client.new(domain, tls)

May be returns OpenSSL::SSL::Context::Client without verify mode?

tls = case tls
        when true
          context = OpenSSL::SSL::Context::Client.new
          context.verify_mode = OpenSSL::SSL::VerifyMode::NONE
          context
        else
          tls
        end

Option to send raw post data

Hi,

first, thanks for great job on halite!
I was looking for a way to perform POST request with raw string body, for handling cases that are neither json or form encoded body (handling elasticsearch bulk api in my case). Is that somehow possible ?

thanks,

Matej

Middleware(features) support

  • middlewares(features) support with #use
    • request/response handler
    • interceptor
  • Refactoring #logger to middleware

Features

Usage:

Halite.use("logger").get "https://httpbin.org/get"
Halite.use("logger", format: "json").get "https://httpbin.org/get"
Halite.use("logger", logger: Logger.new).get "https://httpbin.org/get"

Design:

abstract class Feature
  def initialize(**options)
  end

  # Cooks with request
  def request(request : Request) : Request
    request
  end

  # Cooking with response
  def response(response : Response) : Response
    response
  end

  # Interceptor
  #
  # Intercept and cooking request and response
  def intercept(chain : Interceptor::Chain) : Interceptor::Chain
    chain
  end
end

Logger

Keep #logger in Chainable methods.

PRs

#31, #32, #36

raw output (to String or ...)

Are you funny to implement that the complete response (with headers, 200 OK, ... etc.) is completely copyied to a String? I would need this to read all the bytes my server sends, like cookies too and output it myself with p, pp or simply puts :)

I mean, actually we got this via puts:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
</head>
<body>
  <!-- ... -->
</body>
</html>

and i need this:

HTTP/1.1 200 OK
Date: Mon, 27 Jul 2009 12:28:53 GMT
Server: Apache/2.2.14 (Win32)
Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT
Content-Length: 88
Content-Type: text/html
Connection: Closed

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
</head>
<body>
  <!-- ... -->
</body>
</html>

prefered as String :)

This were nice 👍

Unhandled exception: HTTP::Client::Response#body_io cannot be nil (NilAssertionError) 

Crystal 0.35.1 (2020-06-19)

LLVM: 10.0.0
Default target: x86_64-apple-macosx

We have an API that returns an empty body and when making a http request we receive this error:

Unhandled exception: HTTP::Client::Response#body_io cannot be nil (NilAssertionError)
  from ../../../../../usr/local/Cellar/crystal/0.35.1/src/http/client/response.cr:9:3 in 'body_io'
  from lib/halite/src/halite/response.cr:111:34 in 'to_s'
  from lib/halite/src/halite/response.cr:119:13 in 'to_s'
  from ../../../../../usr/local/Cellar/crystal/0.35.1/src/io.cr:174:5 in '<<'
  from ../../../../../usr/local/Cellar/crystal/0.35.1/src/io.cr:231:5 in 'puts'
  from ../../../../../usr/local/Cellar/crystal/0.35.1/src/kernel.cr:377:3 in 'puts'
  from src/x-client/cli.cr:110:11 in 'run'
  from scripts/x-client:5:1 in '__crystal_main'
  from ../../../../../usr/local/Cellar/crystal/0.35.1/src/crystal/main.cr:105:5 in 'main_user_code'
  from ../../../../../usr/local/Cellar/crystal/0.35.1/src/crystal/main.cr:91:7 in 'main'
  from ../../../../../usr/local/Cellar/crystal/0.35.1/src/crystal/main.cr:114:3 in 'main'

Our code is just:

        Halite
          .basic_auth(user: username, pass: password)
          .post(url, json: payload)

Wrapping the request in a rescue block doesn't seem to do anything:

      begin
        Halite
          .basic_auth(user: username, pass: password)
          .post(url, json: payload)
      rescue
        payload
      end

Any thoughts? The request is made successfully but HTTP client seems unhappy about having an empty response body.

Get amount of data dowloaded, info ...

Hello, first thank you very much for your work. I would like to use your library for my project. I'm actually writing a package manager for Linux with Crystal, but I was using wget because HTTP:Client doesn't support redirection.

I have a question, because I would like to make a nice output when my package manager is downloading a file. How can I get the amount of data downloaded, speed ... etc ? Can I have an example please ?

It's my project: https://github.com/Fulgurance/ISM

The README isn't up to date, but it show the most important

Confict with exception_page

In Halite, it overwrites HTTP::Params.to_h method, so it will confict with other shard which use it.

in lib/exception_page/src/exception_page.cr:39: instance variable '@headers' of ExceptionPage must be Hash(String, Array(String)), not Hash(String, Array(String) | String)

    @headers = context.response.headers.to_h
    ^~

Basic Auth creates bad headers in crystal 0.29.0

The following error is thrown when trying to use the Halite.basic_auth function.

       Header content contains invalid character '\n' (ArgumentError)
         from /usr/local/Cellar/crystal/0.29.0/src/http/headers.cr:303:7 in 'check_invalid_header_content'
         from /usr/local/Cellar/crystal/0.29.0/src/http/headers.cr:108:5 in 'add'
         from lib/halite/src/halite/ext/http_headers_encode.cr:27:11 in 'encode'
         from lib/halite/src/halite/options.cr:387:9 in 'parse_headers'
         from lib/halite/src/halite/options.cr:130:23 in 'with_headers'
         from lib/halite/src/halite/chainable.cr:163:7 in 'headers'
         from lib/halite/src/halite/chainable.cr:130:7 in 'auth'
         from lib/halite/src/halite/chainable.cr:120:7 in 'basic_auth'

This is due to not using Base64.strict_encode which does not place a newline at the end of the output.

Timeouts to download binary data with redirections

It returns gateway timeout use uri: https://github.com/icyleaf/halite/archive/master.zip but it works with redirect uri (https://codeload.github.com/icyleaf/halite/zip/master)

Headers info with curl.

$ curl -L -I https://github.com/icyleaf/halite/archive/master.zip
HTTP/1.1 302 Found
Date: Tue, 10 Oct 2017 16:00:33 GMT
Content-Type: text/html; charset=utf-8
Server: GitHub.com
Status: 302 Found
Cache-Control: no-cache
Vary: X-PJAX
Location: https://codeload.github.com/icyleaf/halite/zip/master
X-UA-Compatible: IE=Edge,chrome=1
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Sat, 10 Oct 2037 16:00:33 -0000; secure; HttpOnly
Set-Cookie: _gh_sess=eyJzZXNzaW9uX2lkIjoiZTVkZTlmZjk4YTI3ODM2YmNkNjhjZGYyYTFmZjI4YTkiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUwNzY1MTIzMzkyNiwic3B5X3JlcG8iOiJpY3lsZWFmL2hhbGl0ZSIsInNweV9yZXBvX2F0IjoxNTA3NjUxMjMzfQ%3D%3D--ee361d1140eb2973415f14468179cc708808e6ad; path=/; secure; HttpOnly
X-Request-Id: 5ff7e430d9449fa32c538e34d20df358
X-Runtime: 0.052454
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src render.githubusercontent.com; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
Public-Key-Pins: max-age=5184000; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="RRM1dGqnDFsCJXBTHky16vi1obOlCgFFn/yOhI/y+ho="; pin-sha256="k2v657xBsOVe1PQRwOsHsw3bsGT2VzIqz5K+59sNQws="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="IQBnNBEiFuhj+8x6X8XLgh01V9Ic5/V3IRQLNFFc7v4="; pin-sha256="iie1VXtL7HzAMF+/PVPR9xzT80kQxdZeJ+zduCB3uj0="; pin-sha256="LvRiGEjRqfzurezaWuj8Wie2gyHMrW5Q06LspMnox7A="; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Runtime-rack: 0.059370
Vary: Accept-Encoding
X-GitHub-Request-Id: 768B:28D99:51FB1CC:7DEE5B1:59DCEEA1

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Access-Control-Allow-Origin: https://render.githubusercontent.com
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
Strict-Transport-Security: max-age=31536000
Vary: Authorization,Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
ETag: "cc28467b25df3dd57bf2329bde0d80d85cb7c0d4"
Content-Type: application/zip
Content-Disposition: attachment; filename=halite-master.zip
X-Geo-Block-List:
Date: Tue, 10 Oct 2017 16:00:35 GMT
X-GitHub-Request-Id: 76B0:1D38E:A8177:199908:59DCEEA2

Feature: HTTP Proxy

in v0.8.0 Halite add local cache feature(#24), though this does not debug friendly, it could not review the raw directly(#15, #22)

Halite use many chainable methods and high duplice from HTTP::Client, I decide to rewrite a new HTTP Connection and manage itself inside Halite.

Reference:

Can't cast from NamedTuple to Halite::Options::Type failed

accepts NamedTuple in params/json/form:

Halite.get "http url", params: {name: "foo", skill: ["a", "b"], hobbit: {work: "abc"}}

It throws an exception:

Unhandled exception: cast from NamedTuple(work: String) to Halite::Options::Type failed, at src/halite/options.cr:356:5:19 (TypeCastError)
  from src/halite/options.cr:0:5 in 'parse_params'
  from src/halite/options.cr:92:7 in 'initialize:headers:cookies:params:form:json:raw:timeout:follow:ssl:logging'
  from src/halite/options.cr:78:5 in 'new:headers:cookies:params:form:json:raw:timeout:follow:ssl:logging'
  from src/halite/options.cr:73:7 in 'new:headers:params:form:json:raw:ssl'
  from src/halite/chainable.cr:423:7 in 'options_with'
  from src/halite/chainable.cr:385:7 in 'request:headers:params:raw:ssl'
  from src/halite/chainable.cr:5:5 in 'get:params'
  from main.cr:18:1 in '__crystal_main'
  from /usr/local/Cellar/crystal/0.26.1/src/crystal/main.cr:97:5 in 'main_user_code'
  from /usr/local/Cellar/crystal/0.26.1/src/crystal/main.cr:86:7 in 'main'
  from /usr/local/Cellar/crystal/0.26.1/src/crystal/main.cr:106:3 in 'main'

Is Halite Fiber-safe?

When firing multiple requests inside Fibers, eventually I will get a Unhandled exception in spawn: SSL_connect: ZERO_RETURN (OpenSSL::SSL::Error)

Support persistant connections

Persisting connections can dramatically reduce request time. I'm not sure if Halite is already doing this, but it might be something to look into if not. Here is an example of a Ruby gem that does it.

Issue while redirecting

It seems Halite had a problem composing the target from this redirect:

screenshot at 15-00-57

Error:

No address found for 172.16.131.135index.php:80 over TCP (Halite::ConnectionError)

Proxy for Web Connection

Hello.
Coming from Mechanize@Ruby, I want to use Halite@Crystal now.

However, I can't find the possibility to specify a web proxy at Halite::Client for my requests ... Can this be implemented?

Missing i

;-)

screenshot_2018-06-21-12-20-04

uuups ... very big screenshot from mobile :)

New instance client with block behavior

It must to change instance client with block to return self instead of options.

Before:

Any method was call must be exists in Halite::Options, and it is not recommend to use, because it was wrapper by chainable methods.

client = Halite::Client.new do |options|
  options.headers = {
    user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
  }

  # Enable logging
  options.logging = true

  # Set read timeout to one minute
  options.read_timeout = 1.minutes
end

After:

client = Halite::Client.new do
  headers user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
  basic_auth "name", "foo"
  logging true
end

For new change, we can call any methods in chainable module.

multipart/form-data multiple file upload

This awesome tool 💯 really needs parsing for multipart/form-data via post, including multiple files with one tag name (like <input multiple>)

This would be very nice :)

As example:

This exists:

Halite.post("http://127.0.0.1:3000/upload", form: {
  "username" => "Quincy",
  "avatar" => File.open("image.png")
})

This is standard:

Halite.post("http://127.0.0.1:3000/upload", form: {
  "username" => "Quincy",
  "avatar" => File.open("image1.png"),
  "bvatar" => File.open("image2.png"),
  "cvatar" => File.open("image3.png"),
  "dvatar" => File.open("image4.png")
})

This is <input multiple>

Halite.post("http://127.0.0.1:3000/upload", form: {
  "username" => "Quincy",
  "avatar" => [File.open("1.png"), File.open("2.png"), File.open("3.png"), File.open("4.png")]
})

Client request option incorrect when used as class variable

It's OK to use client as instance variables:

require "halite"

class ClientA
  def initialize
    @client = Halite::Client.new do
      endpoint "http://httpbin.org/get"
      logging true
    end
  end
  def test
    @client.get("", params: {"test"=>"a"})
  end
end

class ClientB
  def initialize
    @client = Halite::Client.new do
      endpoint "http://httpbin.org/post"
      logging true
    end
  end
  def test
    @client.post("", params: {"test"=>"b"})
  end
end

ClientA.new.test
ClientB.new.test

Output:

2019-06-10 07:22:19 +08:00 | request  | GET    | http://httpbin.org/get?test=a
2019-06-10 07:22:20 +08:00 | response | 200    | http://httpbin.org/get?test=a | 1.01s | application/json
...
2019-06-10 07:22:20 +08:00 | request  | POST   | http://httpbin.org/post?test=b
2019-06-10 07:22:20 +08:00 | response | 200    | http://httpbin.org/post?test=b | 612.35ms | application/json

If used as class variables:

require "halite"

class ClientA
  @@client = Halite::Client.new do
    endpoint "http://httpbin.org/get"
    logging true
  end
  def self.test
    @@client.get("", params: {"test"=>"a"})
  end
end

class ClientB
  @@client = Halite::Client.new do
    endpoint "http://httpbin.org/post"
    logging true
  end
  def self.test
    @@client.post("", params: {"test"=>"b"})
  end
end

ClientA.test
ClientB.test

Output

2019-06-10 07:36:06 +08:00 | request  | GET    | http://httpbin.org/post?test=a
2019-06-10 07:36:07 +08:00 | response | 405    | http://httpbin.org/post?test=a | 836.37ms | text/html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

2019-06-10 07:36:07 +08:00 | request  | POST   | http://httpbin.org/post?test=b
2019-06-10 07:36:07 +08:00 | response | 200    | http://httpbin.org/post?test=b | 612.47ms | application/json
...

It seems late endpoint definitions will overwrite previous ones when used in class variables. Also note the path needs not be empty.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.