Giter VIP home page Giter VIP logo

scalaj-http's Introduction

Maven Central Javadocs FOSSA Status

THIS LIBRARY IS DEPRECATED

As of April 2022 there are many supported options for making HTTP calls in scala:

  • sttp or http4s might be good scala first options
  • The native HttpClient library if you're using JDK 11+ and want minimal deps

If you've inherited a codebase that's using this library, I would suggest just copying the source code (it's only four files, and no depedencies) and make whatever fixes/changes you want.

Good luck!

Simplified Http

This is a fully featured http client for Scala which wraps java.net.HttpURLConnection

Features:

  • Zero dependencies
  • Cross compiled for Scala 2.10, 2.11, 2.12, and 2.13-M3
  • OAuth v1 request signing
  • Automatic support of gzip and deflate encodings from server
  • Easy to add querystring or form params. URL encoding is handled for you.
  • Multipart file uploads

Non-Features:

  • Async execution
    • The library is thread safe. HttpRequest and HttpResponse are immutable. So it should be easy to wrap in an execution framework of your choice.

Works in Google AppEngine and Android environments.

Big differences:

  • Executing the request always returns a HttpResponse[T] instance that contains the response-code, headers, and body
  • Exceptions are no longer thrown for 4xx and 5xx response codes. Yay!
  • Http(url) is the starting point for every type of request (post, get, multi, etc)
  • You can easily create your own singleton instance to set your own defaults (timeouts, proxies, etc)
  • Sends "Accept-Encoding: gzip,deflate" request header and decompresses based on Content-Encoding (configurable)
  • Redirects are no longer followed by default. Use .option(HttpOptions.followRedirects(true)) to change.

Installation

in your build.sbt

libraryDependencies +=  "org.scalaj" %% "scalaj-http" % "2.4.2"

maven

<dependency>
  <groupId>org.scalaj</groupId>
  <artifactId>scalaj-http_${scala.version}</artifactId>
  <version>2.4.2</version>
</dependency>

If you're including this in some other public library. Do your users a favor and change the fully qualified name so they don't have version conflicts if they're using a different version of this library. The easiest way to do that is just to copy the source into your project :)

Usage

Simple Get

import scalaj.http._

val response: HttpResponse[String] = Http("http://foo.com/search").param("q","monkeys").asString
response.body
response.code
response.headers
response.cookies

Immutable Request

Http(url) is just shorthand for a Http.apply which returns an immutable instance of HttpRequest. You can create a HttpRequest and reuse it:

val request: HttpRequest = Http("http://date.jsontest.com/")

val responseOne = request.asString
val responseTwo = request.asString

Additive Request

All the "modification" methods of a HttpRequest are actually returning a new instance. The param(s), option(s), header(s) methods always add to their respective sets. So calling .headers(newHeaders) will return a HttpRequest instance that has newHeaders appended to the previous req.headers

Simple form encoded POST

Http("http://foo.com/add").postForm(Seq("name" -> "jon", "age" -> "29")).asString

OAuth v1 Dance and Request

Note: the .oauth(...) call must be the last method called in the request construction

import scalaj.http.{Http, Token}

val consumer = Token("key", "secret")
val response = Http("https://api.twitter.com/oauth/request_token").postForm(Seq("oauth_callback" -> "oob"))
  .oauth(consumer).asToken

println("Go to https://api.twitter.com/oauth/authorize?oauth_token=" + response.body.key)

val verifier = Console.readLine("Enter verifier: ").trim

val accessToken = Http("https://api.twitter.com/oauth/access_token").postForm.
  .oauth(consumer, response.body, verifier).asToken

println(Http("https://api.twitter.com/1.1/account/settings.json").oauth(consumer, accessToken.body).asString)

Parsing the response

Http("http://foo.com").{asString, asBytes, asParams}

Those methods will return an HttpResponse[String | Array[Byte] | Seq[(String, String)]] respectively

Advanced Usage Examples

Parse the response InputStream directly

val response: HttpResponse[Map[String,String]] = Http("http://foo.com").execute(parser = {inputStream =>
  Json.parse[Map[String,String]](inputStream)
})

Post raw Array[Byte] or String data and get response code

Http(url).postData(data).header("content-type", "application/json").asString.code

Post multipart/form-data

Http(url).postMulti(MultiPart("photo", "headshot.png", "image/png", fileBytes)).asString

You can also stream uploads and get a callback on progress:

Http(url).postMulti(MultiPart("photo", "headshot.png", "image/png", inputStream, bytesInStream,
  lenWritten => {
    println(s"Wrote $lenWritten bytes out of $bytesInStream total for headshot.png")
  })).asString

Stream a chunked transfer response (like an event stream)

Http("http://httpbin.org/stream/20").execute(is => {
  scala.io.Source.fromInputStream(is).getLines().foreach(println)
})

note that you may have to wrap in a while loop and set a long readTimeout to stay connected

Send https request to site with self-signed or otherwise shady certificate

Http("https://localhost/").option(HttpOptions.allowUnsafeSSL).asString

Do a HEAD request

Http(url).method("HEAD").asString

Custom connect and read timeouts

These are set to 1000 and 5000 milliseconds respectively by default

Http(url).timeout(connTimeoutMs = 1000, readTimeoutMs = 5000).asString

Get request via a proxy

val response = Http(url).proxy(proxyHost, proxyPort).asString

Other custom options

The .option() method takes a function of type HttpURLConnection => Unit so you can manipulate the connection in whatever way you want before the request executes.

Change the Charset

By default, the charset for all param encoding and string response parsing is UTF-8. You can override with charset of your choice:

Http(url).charset("ISO-8859-1").asString

Create your own HttpRequest builder

You don't have to use the default Http singleton. Create your own:

object MyHttp extends BaseHttp (
  proxyConfig = None,
  options = HttpConstants.defaultOptions,
  charset = HttpConstants.utf8,
  sendBufferSize = 4096,
  userAgent = "scalaj-http/1.0",
  compress = true
)

Full API documentation

scaladocs here

Dealing with annoying java library issues

Overriding the Access-Control, Content-Length, Content-Transfer-Encoding, Host, Keep-Alive, Origin, Trailer, Transfer-Encoding, Upgrade, Via headers

Some of the headers are locked by the java library for "security" reasons and the behavior is that the library will just silently fail to set them. You can workaround by doing one of the following:

  • Start your JVM with this command line parameter: -Dsun.net.http.allowRestrictedHeaders=true
  • or, do this first thing at runtime: System.setProperty("sun.net.http.allowRestrictedHeaders", "true")

License

FOSSA Status

scalaj-http's People

Contributors

andrewl102 avatar antonfagerberg avatar dmaicher avatar eranation avatar felixreuthlingerbmw avatar fntz avatar fossabot avatar hoffrocket avatar jonshea avatar jorgeortiz85 avatar jrouly avatar lucasjosh avatar mukel avatar odisseus avatar pelamfi avatar pfhayes avatar savulchik avatar sciss avatar tanishiking avatar tonyd3 avatar vmorarian avatar xuwei-k avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scalaj-http's Issues

Unable to remove default Accept header from POST requests

Hi

I am trying to do a POST request and some headers are added by default by Http's implementation (I assume by the underlying Java implementation). The Accept header is making my endpoint reject my POST request with a 406.

Below my code:

val result = Http(eventStoreUrl)
  .proxy("127.0.0.1", 8888)
  .header("Content-Type", "application/json")
  .header("ES-EventType", "UserSearchedHotelAvailability")
  .header("ES-EventId", UUID.randomUUID().toString)
  .postData(payload)
  .asString

and here a trace of the actual post request from Fiddler:

POST myUrl HTTP/1.1
Accept-Encoding: gzip,deflate
ES-EventId: 6bed603d-e6aa-4d9f-9e04-3b5e19f372d1
ES-EventType: UserSearchedHotelAvailability
Content-Type: application/json
User-Agent: scalaj-http/1.0
Host: 127.0.0.1:2113
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 10

{'Type':1}

I could not find a way to remove the Accept header from there. I tried setting it to null or empty string, but that meant it's just added twice, first with my value, then with this default one.

PUT possible

Hi there,

is it possible to do HTTP PUT commands? Something like:

Http.put("http://foo.com/add").data(""" {"some": "data"} """)

Regards,
Sven

java 6 support for binaries?

I'm trying to use scalaj-http but I'm getting errors classfile version errors (looks like the artifacts on sonatype are v51). I'm deploying on macs that default to java 1.6.

Leak of open connections in CLOSE_WAIT

This sample program will keep many connections in CLOSE_WAIT:

import scalaj.http.{HttpOptions, Http}

object Runner {

  def main(args: Array[String]) {
    val data =
      """
        |{
        |"test": "test"
        |}
      """.stripMargin

    for (i <- 0 to 10000) {
      println(s"Making request $i...")

      val request = Http
        .postData("http://google.com", data)
        .header("Content-type", "application/json")
        .header("Connection", "close")

      request.responseCode
    }

  }

}

How to use input stream to download a file?

This is the kind of code I am using

   request.apply {
      inputStream =>
        copyInputToOutputImperative(inputStream, tempOutstream, bufferSize);
    }

The input stream is empty!
How to I trigger the request?
Normally I would do request.responseCode or request.asString

But now I want the inputstream and also I want the response code to know if the inputstream has anything useful or not.
Thanks!

SSL handshake exception

SSL handshake exception always pops up while trying to do the following. Any idea would be appreciated.
val timeout = 5000
val path = "some curl-able path"
// Have set proper java opts etc
Http(path).option(HttpOptions.connTimeout(timeout)).option(HttpOptions.readTimeout(timeout)).responseCode

Wonder if I am missing anything?

Cross Domain

Hi I tried to invoke a cross domain request using scalaj which eventually failed.
is there any hack to make it work
please let us know
Thanks in Advance

scalaj-http doesn't set default "User-Agent" value into HTTP request headers.

scala> val html = Http("http://www.questionablecontent.net").asString
scalaj.http.HttpException: 403: Forbidden
at scalaj.http.Http$Request.process(Http.scala:125)
at scalaj.http.Http$Request.apply(Http.scala:102)
at scalaj.http.Http$Request.asString(Http.scala:151)
at .(:11)
at .()
at .(:7)
at .()
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
...

kypeli@iceberg:~$ curl -I http://www.questionablecontent.net
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Sun, 21 Jul 2013 11:40:50 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.4.13
Set-Cookie: __cfduid=dd4e453443f2d6171976787db3fdf55c81374406850; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.questionablecontent.net
CF-RAY: 9179e60760306dc

get response even if response code is not in 200-399 range

When i make an http request(get) and the response code is not in 200-399 range, i get an exception. Wouldnt it be better if you returned the a pair with the http response code and response body or an object with response code and response body as members. This can be useful for debugging error responses

readString seems buggy with Scala 2.10.0 (ok with 2.9.2)

readString does not seems to works as expected with 2.10.0
I suggest using bos.appendAll instead of bos.append

def readString(is: InputStream): String = {
val in = new InputStreamReader(is, charset)
val bos = new StringBuilder
val ba = new ArrayChar

def readOnce {
  val len = in.read(ba)
  if (len > 0) bos.appendAll(ba, 0, len)
  if (len >= 0) readOnce
}

readOnce

bos.toString()

}

Multipart is missing the dataparts

How could someone implement the data parts of a multipart ?

These are a seperate section from files.

For example my web service framework is using a method like multipartBody.dataParts("some key in here") to get data parts and multipartBody.file("filename") to get files.

To make more clear what I am talking about I will note this from official specificaiton: A form data part of a multipart looks like that at the end (if boundary is *****):

--*****
Content-Disposition: form-data; name="some name"\n
\n
value here\n
\n

Adding this library to library dependencies results in ClassNotFoundException for the App object

Hi

I have a project seeded from here: https://github.com/spray/spray-template

I have added this line to my librariesDependencies: "org.scalaj" %% "scalaj-http" % "1.1.4" and now I get a runtime error Exception in thread "main" java.lang.ClassNotFoundException: com.example.Boot

This is the Boot object:

object Boot extends App {
...
}
where App is scala.App

As soon as I remove the line from libraries dependencies, everything runs fine again. I don't even have to add a call to something from scalaj, it just immediately breaks when I add this as a dependency.

Other dependencies I have are:
"io.spray" %% "spray-can" % sprayV,
"io.spray" %% "spray-routing" % sprayV,
"io.spray" %% "spray-testkit" % sprayV % "test",
"io.spray" %% "spray-json" % "1.3.1",
"com.typesafe.akka" %% "akka-actor" % akkaV,
"com.typesafe.akka" %% "akka-testkit" % akkaV % "test",
"org.specs2" %% "specs2-core" % "2.3.11" % "test",
"com.typesafe.slick" %% "slick" % "2.1.0",
"com.typesafe.slick" %% "slick-extensions" % "2.1.0",
"c3p0" % "c3p0" % "0.9.1.2",
"org.slf4j" % "slf4j-api" % "1.7.10",
"org.slf4j" % "slf4j-simple" % "1.7.10"

Async scalaj: scalaj fail with futures

Hi...I'm wrapping my request inside a Future{blocking{ Http .... }} and I have a list of 200 urls for call freebase service

val foo=List[String].map(myHttpFutureMethod))

this return a List of Future[String]...so far so good
now when I do Await.result(Future.sequence(foo),10 minutes) I get

java.net.SocketTimeoutException: connect timed out
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
  at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
  at sun.net.www.http.HttpClient.New(HttpClient.java:308)
  at sun.net.www.http.HttpClient.New(HttpClient.java:326)
  at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1167)
  at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1146)
  at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:987)
  at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:985)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.security.AccessController.doPrivileged(AccessController.java:713)
  at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:984)
  at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:931)
  at scalaj.http.Http$$anonfun$1.apply(Http.scala:296)
  at scalaj.http.Http$$anonfun$1.apply(Http.scala:296)
  at scalaj.http.Http$Request.process(Http.scala:159)
  at scalaj.http.Http$Request.apply(Http.scala:141)
  at scalaj.http.Http$Request.asString(Http.scala:207)
  at $anonfun$YQL4$1$$anonfun$apply$1$$anonfun$apply$2.apply(<console>:29)
  at $anonfun$YQL4$1$$anonfun$apply$1$$anonfun$apply$2.apply(<console>:29)
  at scala.util.Try$.apply(Try.scala:191)
  at .retry(<console>:23)
  at $anonfun$YQL4$1$$anonfun$apply$1.apply(<console>:26)
  at $anonfun$YQL4$1$$anonfun$apply$1.apply(<console>:26)
  at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$2$$anon$4.block(ExecutionContextImpl.scala:48)
  at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
  at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$2.blockOn(ExecutionContextImpl.scala:45)
  at scala.concurrent.package$.blocking(package.scala:54)
  at $anonfun$YQL4$1.apply(<console>:25)
  at $anonfun$YQL4$1.apply(<console>:25)
  at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
  at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
  at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


I test the code with only 10 url and works good...but when I use much more then fails
I've played with HttpOptions.connTimeout and HttpOptions.readTimeout and change the number of thread pools in Executors.newFixedThreadPool(N) but always fails, dont care what number I put

how could fix it??...thanks!!

How to include a body in a request?

I see you can already have a body (for example json) in a request using "postData" method.
But If I want to perform a DELETE verb/method I start the command like that:

Http(link).option(HttpOptions.method("DELETE")).
header("Content-Type", "application/json"). ..... ?

How could I include the json body?
Thank you!

Multipart uploading files larger than 2Go: negative long returned

Hi, thanks for this lib!

This issue is related to a previous one: #38.
I explained then that I couldn't uplaod files of >2Go using multipart. That was fixed but there's still a problem:
the Long (length written) returned and handled by writeCallBack: Long => Unit is negative and decreasing after 2Go (2147483648 as Long) has been reached.
It should still be growing until reaching the file length.

I upgraded to version 1.1.4.

Add a Changelog

It'd be very helpful to see what's changed from release to release, without getting bogged down in the git commit log. The commit log is helpful, but it doesn't really let one know about the severity of bug fixes, breaking API changes, etc. I realize it's a bit more work to maintain, but it'd be much appreciated.

Multipart File Upload Causing "java.io.IOException: insufficient data written"

We're trying to move from dispatch to scalaj, and during a file upload against spray (and, later against play) we run into this issue:

java.io.IOException: insufficient data written
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3214)
at scalaj.http.Http$$anonfun$3.apply(Http.scala:399)
at scalaj.http.Http$$anonfun$3.apply(Http.scala:312)
at scalaj.http.Http$Request.process(Http.scala:159)
at scalaj.http.Http$Request.apply(Http.scala:141)
at scalaj.http.Http$Request.asString(Http.scala:190)
at rest.DocumentProcessorClient$$anonfun$1.apply(DocProcClient.scala:27)
at rest.DocumentProcessorClient$$anonfun$1.apply(DocProcClient.scala:21)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

This is consistent across Mac and PC, and both using a stream or a byte array. We're running scala 2.10.3, scalaj 0.3.14, and java 1.7.0_51. Our code looks like this:

val responseFuture = Future {
log.debug("Uploading file " + fileName)
val r = Http.multipart(putUrl, MultiPart(fileName, fileName, "application/none", byteArray))
.option(HttpOptions.connTimeout(10000))
.option(HttpOptions.readTimeout(60000))
(r.responseCode, r.asString)
}

However, note, we've also tried this outside of a future, same result.

On the server side, we see a java.nio.channels.ClosedChannelException when we try to send an Ok back, which could be related?

Any advice would be much appreciated!

Multipart uploading files larger than 2Go

Hi;
I'm using scalaj-http Multipart to upload files with a form to a server. But I currently can't send files larger than 2Go for the field "numBytes" is an Int.
I fixed it by (forking, then) changing it into a Long in Http.scala:

case class MultiPart(val name: String, val filename: String, val mime: String, val data: InputStream, val numBytes: Long,
val writeCallBack: Long => Unit)

and precising further:

val totalBytesToSend: Long = {
...
0L + paramsLength + filesLength + finaleBoundaryLength
}

Could it be stable in scalaj-http ? should be an improvement. Thanks!

Multiple posts side effect on accessing request methods

I use scalaj-http in my functional tests to test a post to a service that I'm writing. I post the request, no problem. Then I check the status code by doing a request.responseCode. I then want to pick apart the response for an id returned in the response. I do a request.asString, only to get a second post to the service, which is an unwanted side effect.

As a result my test is failing due to the duplicate post. Am I doing something wrong when interacting with the scalaj-http api?

Basic Auth

Does or can Scalaj support Basic Auth header? Is there a way to do it?

Binary data is truncated by .asString conversion

This is a pretty unusual use case, so I'll understand if this is something not considered worth fixing.

I'm using an HTTP connection to connect to a bittorrent tracker, which returns mostly ASCII text, with some raw binary data. Most characters above 0x80 get truncated to 0x3F when I use Http(foo).asString.

To get around this, I'm using java.net.URL instead of scalaj-http, and avoiding any conversions that assume some charset. If there's any interest, I can submit a pull request if I can find some spare time, but I don't know that anyone actually cares about pulling raw binary data over HTTP

DNS exception not managed

First, thanks for you work !

When I try to use this lib with an incorrect URL, it blocks.
I can see, with Wireshark, the dns response error : No such name.
I have solved the problem by adding a try/catch (see below).
It's a little bit ugly, but it seems to work.

def exec[T](parser: (Int, Map[String, String], InputStream) => T): HttpResponse[T] = {
    val urlToFetch: URL = new URL(urlBuilder(this))
    try{
      proxyConfig.map(urlToFetch.openConnection).getOrElse(urlToFetch.openConnection) match {
        case conn: HttpURLConnection =>
          conn.setInstanceFollowRedirects(false)
          HttpOptions.method(method)(conn)
          if (compress) {
            conn.setRequestProperty("Accept-Encoding", "gzip,deflate")
          }
          headers.reverse.foreach{ case (name, value) =>
            conn.setRequestProperty(name, value)
          }
          options.reverse.foreach(_(conn))

          connectFunc(this, conn)
          try {
            toResponse(conn, parser, conn.getInputStream)
          } catch {
            case e: java.io.IOException if conn.getResponseCode > 0 =>
              toResponse(conn, parser, conn.getErrorStream)
          } finally {
            closeStreams(conn)
          }
      }
    }catch{
      case e: Exception =>
        HttpResponse(e.asInstanceOf[T], 502, Map())
    }
  }

Missing tags

The tag list stops at version 0.2.5, but 0.2.8 seems to be the latest release. I had to dig through the maven repository to work this out because installation failed when using 0.2.5 with scala 2.9.0.1.

content-encoding

twitter is sending the content-encoding header in lower case and this isn't being detected by scalaj (unless I'm doing something wrong). This means that for twitter I am decoding manually but for facebook no problem.

Can it (should it) be made case insensitive?

Important feature is missing in new Multipart

The new multipart introduced in version 0.3.9 where you can upload large files using input stream would be nice to have a couple of extra features.
1)I must be able to set the buffer/chunck size
2)If there was a callback function on each buffer/chunck uploaded someone could fill a progress bar properly or do some other similar action. Thank you

What happened to HttpException ?

Hi there! After a long time I have decided to upgrade to the latest (1.1.4) version from 0.3.x

On the old version of scalaj I used to have a line that looked like that

case e: HttpException => MyCustomErrorResultCaseClass(e.body, e.message)

Is HttpException used at all in the latest version? I only see it referenced once in the source code

couldn't resolve dependencies 0.3.3 and 0.3.4

[info] Resolving commons-codec#commons-codec;1.5 ...                                                
[warn]  [NOT FOUND  ] commons-codec#commons-codec;1.5!commons-codec.src (1003ms)                    
[warn] ==== public: tried                                                                           
[warn]   http://repo1.maven.org/maven2/commons-codec/commons-codec/1.5/commons-codec-1.5-sources.src
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::                                              
[warn]  ::              FAILED DOWNLOADS            ::                                              
[warn]  :: ^ see resolution messages for details  ^ ::                                              
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::                                              
[warn]  :: commons-codec#commons-codec;1.5!commons-codec.src                                        
[warn]  :::::::::::::::::::::::::::::::::::::::::::::: 

0.3.3 and 0.3.4 pom.xml included

<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.5</version>
<classifier>sources</classifier>
<type>src</type>
</dependency>

and

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
<classifier>sources</classifier>
<type>src</type>
</dependency>

http://repo1.maven.org/maven2/org/scalaj/scalaj-http_2.9.2/0.3.4/scalaj-http_2.9.2-0.3.4.pom
http://repo1.maven.org/maven2/org/scalaj/scalaj-http_2.9.2/0.3.3/scalaj-http_2.9.2-0.3.3.pom

I think these dependencies should not be include.

Cancelling connection would be really useful

Having set a val request = Http.postData(...
for instance when you call request.asHeadersAndParse(... then the request is actually executed in order to get the response.

Could you have an request.asHeadersAndParse method run asynchronously (a future or something) and also have a request.cancelConnection method to cancel the current connection.

I am not implying to interrupt a certain thread, only to terminate connection and get thrown an HttpException as you would normally get after connection or read timeout.

If the above is possible would be really useful.
Thanks
pligor

Unexpected java.net.SocketTimeoutException from URL "http://www.dilbert.com"

It seems I can make GET requests successfully to a given site - except to "http://www.dilbert.com". Then I get a java.net.SocketTimeoutException, although the site seems to be responding to HTTP requests normally.

Please see below:

Welcome to Scala version 2.10.0 (Java HotSpot(TM) Client VM, Java 1.7.0_17).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import scalaj.http.Http
import scalaj.http.Http

scala> val html = Http("http://www.dilbert.com").asString
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)

But the site is up:

kypeli@iceberg:~$ curl -I http://www.dilbert.com
HTTP/1.1 200 OK
Date: Sun, 21 Jul 2013 09:26:08 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.10
Set-Cookie: PHPSESSID=3oeodcel2pacdgd7cpoel95vo5; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=5nvtfb4m3qsm1hiknsglo6lmq4; path=/
Set-Cookie: PHPSESSID=69t1p2kc5afs3de3gi75ouuoq2; path=/
Set-Cookie: PHPSESSID=tbfjgtkn6dk6mgogtdv6evl0p1; path=/
Set-Cookie: PHPSESSID=5punq0q7rsm8b9q4357l2lb8a1; path=/
Connection: close
Content-Type: text/html; charset=utf-8lias

And I can make requests to other sites:

scala> val html = Http("http://www.google.com").asString
html: String =
<!doctype html><title>Google</title><script>(function(){
window.google={kEI:"TKnrUYe4I-zZ0QXx7IGIDg",getEI:function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI},https:function(){return"https:"==window.location.

Exceptions in README examples

I was trying to run the README examples and in several I've received exceptions. For instance, testing:

val request: Http.Request = Http("http://date.jsontest.com/")

val resultOne = request.asString
val resultTwo = request.asString

throws an: java.lang.NoSuchMethodError: scala.collection.mutable.StringBuilder.append([CII)Lscala/collection/mutable/StringBuilder;

Am I missing some configuration or this project is no longer updated?

Scala 2.10?

Hey, your library seems to be awesome, but I can't find it for Scala 2.10.
Are you supporting it? Will it work?

Cheers,
A.

Nullpointer Exception

I get a very weird nullpointer exception which happens when I append a spaces to parameters in querystrings but only on certain URLs. I can't find a pattern or cause but I can reproduce it by doing this:

// OK: Works
println(Http("https://www.google.com/?a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)

// OK: 404 HttpException
println(Http("https://www.google.com/?a= b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/12895238/?a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/ 12895238/?a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/12895238/?a=b ").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)

// NOT OK: NullPointerException
println(Http("https://api.instagram.com/v1/users/12895238/?a =b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)
println(Http("https://api.instagram.com/v1/users/12895238/? a=b").options(HttpOptions.connTimeout(10000), HttpOptions.readTimeout(10000)).asString)

It might be related to Issue #8 but it seems to happen in another part of the code:

Exception in thread "main" java.lang.NullPointerException
    at scalaj.http.Http$.tryParse(Http.scala:155)
    at scalaj.http.Http$Request.liftedTree1$1(Http.scala:107)
    at scalaj.http.Http$Request.process(Http.scala:103)
    at scalaj.http.Http$Request.apply(Http.scala:90)
    at scalaj.http.Http$Request.asString(Http.scala:133)

Multipart POST no longer works with android less than target android-19 (4.4.2)

First, thank you for this library, it has simplified my life greatly.

(Also, I was able to resolve my issue by using 0.3.12, so not a big problem for me)

The method HttpURLConnection.setFixedLengthStreamingMode(long contentLength) is only available on android-19 or above, previously only taking an 'int contentLength'. So, on android (scaloid) when sending a multipart post, the method cannot be found with that signature. I am not sure if it is possible to support both long and int signatures, but I thought I should bring this to your attention. (Http.scala, line 326)

expected 30 bytes but received 4096 [org.apache.harmony.luni.internal.net.www.protocol.http.FixedLengthOutputStream.write]

I tried to execute a POST request with 2 multiparts in Android. Something like that:

Http.multipart(url, Multipart(name, filename, mime, data string), Multipart(name, filename, mime, file input stream, chunk size, {
bytesWritten =>
publishProgress(bytesWritten.toFloat / fileLen.toFloat)
}).responseCode

I got a FixedLengthOutputStream exception. This is the log output:

Caused by: java.io.IOException: expected 30 bytes but received 4096
 15399         AndroidRuntime  E    at org.apache.harmony.luni.internal.net.www.protocol.http.FixedLengthOutputStream.write(FixedLengthOutputStream.java:38)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3$$anonfun$apply$8.readOnce$3(Http.scala:381)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3$$anonfun$apply$8.apply(Http.scala:391)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3$$anonfun$apply$8.apply(Http.scala:367)
 15399         AndroidRuntime  E    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
 15399         AndroidRuntime  E    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3.apply(Http.scala:367)
 15399         AndroidRuntime  E    at scalaj.http.Http$$anonfun$3.apply(Http.scala:312)
 15399         AndroidRuntime  E    at scalaj.http.Http$Request.process(Http.scala:159)
 15399         AndroidRuntime  E    at scalaj.http.Http$Request.responseCode(Http.scala:178)

Perhaps this link could be the solution for this issue: http://answer.techwikihow.com/977999/upload-files-post-server-outofmemory.html

Default read and connection timeouts

These are currently set at:

Connection timeout = 0.1 seconds
Read timeout = 0.5 seconds

Surely these are unrealistically low?

Furthermore, your README says your defaults are set at 30s (connection timeout) and 60s (read timeout).

Cannot use "default http proxy"

Because the default proxy option is Proxy.NO_PROXY, there's no easy way to specify "use the proxy that http.proxyHost, http.proxyPort, and http.nonProxyHosts (and the https equivalents) point to".

A straightforward fix for this should be to default to using the Java default proxy - I'll have a PR up shortly.

getResponseHeaders should return a list of values

Headers could contains many values for same key (see Set-Cookie - http://en.wikipedia.org/wiki/Http_cookie)

As s suggestion here is a dummy impl:

def getResponseHeaders(conn: HttpURLConnection): Map[String, List[String]] = {
  // according to javadoc, there can be a headerField value where the HeaderFieldKey is null
  // at the 0th row in some implementations.  In that case it's the http status line
  val map = Stream.from(0).map(i => i -> conn.getHeaderField(i)).takeWhile(_._2 != null).map{ case (i, value) =>
    Option(conn.getHeaderFieldKey(i)).getOrElse("Status") -> value
  }.toList.groupBy(_._1)
  for ( (k,v) <- map ) yield  k -> v.map(_._2)
}

SocketTimeoutException with Http.param and Http.params

Hi,

I seem to be getting a SocketTimeoutException every time I try to use param or params.

Here's my code:

println(Http("https://api.twitter.com/1/statuses/mentions.json").param("include_entities", "true").asString)

This also does not work:

println(Http("https://api.twitter.com/1/statuses/mentions.json?include_entities=true").asString)

Exception:

(run-main) java.net.SocketTimeoutException: connect timed out
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
at java.net.Socket.connect(Socket.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:564)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:395)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:272)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:329)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:172)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:158)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:133)
at scalaj.http.Http$$anonfun$1.apply(Http.scala:192)
at scalaj.http.Http$$anonfun$1.apply(Http.scala:192)
at scalaj.http.Http$Request.process(Http.scala:96)
at scalaj.http.Http$Request.apply(Http.scala:84)
at scalaj.http.Http$Request.asString(Http.scala:112)
at budogo.Runner$.main(Runner.scala:13)
at budogo.Runner.main(Runner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
java.lang.RuntimeException: Nonzero exit code: 1

I'm using version: 0.2.9

If I leave out the parameters everything works fine. Any ideas?

Cannot send large files with multipart

Inside documentation I can find this:
Http.multipart(url, MultiPart("photo","headshot.png", "image/png", fileBytes)).responseCode

But misses the fact that some devices like android cannot handle large files.

Could it be refactored so that the fourth argument is not a byte array but a file?

And of course this file must be handled streamfully with chuncks of bytes.
A simple statement that simply reads a file into a byte array will throw an OutOfMemoryException

NPE in finally block of tryParse

Under certain circumstances a NPE is thrown when trying to get the response string:

java.lang.NullPointerException
at scalaj.http.Http$.tryParse(Http.scala:128)
at scalaj.http.Http$Request.process(Http.scala:95)
at scalaj.http.Http$Request.apply(Http.scala:78)
at scalaj.http.Http$Request.asString(Http.scala:106)

Seems that the conn.getInputStream is null. Is that an expected condition?

sbt.ResolveException: download failed: junit#junit;4.10!junit.src

I am trying to take scalaj-http into use in my Scala Play2 application. This is how I set my dependency for sbt:

"org.scalaj" % "scalaj-http_2.10" % "0.3.8"

This, however, yields the error message in the title:
[warn] [NOT FOUND ] junit#junit;4.10!junit.src (130ms)
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/junit/junit/4.10/junit-4.10-sources.src
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: junit#junit;4.10!junit.src
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
sbt.ResolveException: download failed: junit#junit;4.10!junit.src
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:114)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:114)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:102)
...

Add Scala 2.9.1 artifacts

Scala 2.9.1 has been out for a little while now. The installation instructions use a cross-build configuration, but that fails on Scala 2.9.1 because there aren't any 2.9.1 artifacts yet. I locked down to an older version for now, but it'd be nice to fix this on the maven repos.

Host headers can't be set

Love this library and use it for everything, but I've hit a weird edge case that prevents me from using it in one particular case.

For reasons I won't go into, I have no DNS lookup available so I have to use the IP address, and the service I'm trying to reach is on a subdomain. I've deduced that the way around this using curl or some other posting tool is to set the host header to what would be the subdomain.domain.tld I'd like to reach.

Attempting to set the host header using the header method does not work, apparently this is a security feature of the underlying java library. Ok fair enough.

I tried the same trick with another library (Play's WS) and setting the host is also not allowed there, however, they provide a method called withVirtualHost("subdomain.domain.tld") and that does the magic.

I'd love it if something similar appeared in this library so I can ditch WS! :)

If I get some time I might have a go but it's unlikely :( so I thought I should capture this in an issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.