when http.configure is called, InternalDefaults.client is no longer maintained, and causing "AsyncHttpClient.close() hasn't been invoked, which may produce file descriptor leaks" about reboot HOT 11 CLOSED

Hi, I'm working on the configuration for issue #58 and am also checking this out. I just tested calling configure and was still able to exit without calling shutdown. We are creating a new client (and I think this is the only safe thing to do) but both the new and old one will use the deamon thread factories by default.

Can you provide an example where this doesn't work like you expect?

from reboot.

This is the code I used to use, and works -

  val respPromise = dispatch.Http(req.>(f => (f.getStatusCode(), f.getResponseBody())))
  respPromise()

and I want to do different timeout configuration per clients, so I use -
http = new dispatch.Http {
import com.ning.http.client._
val builder = new AsyncHttpClientConfig.Builder()
builder.setConnectionTimeoutInMs(3000).setRequestTimeoutInMs(10000)
override val client = new AsyncHttpClient(builder.build())
}
val respPromise = http(req.>(f => (f.getStatusCode(), f.getResponseBody())))

and I found the leaks (by stressing it running 4 threads keep calling above code in a 100 times for loop). My scenario is, I have 3-4 http connections to different service, and I want the client (and underlying http connection) to be reused if it hits the same server.
Is there anyway to achieve my goal?

from reboot.

and I found the leaks (by stressing it running 4 threads keep calling above code in a 100 times for loop).

Yeah if you want to use arbitrarily many clients you'll need to shut them down on your own, that's the assumption of the underlying client.

But I don't think you're using configure as intended. Should be more like

def buildHttp() = dispatch.Http.configure { builder =>
  builder.setConnectionTimeoutInMs(3000).setRequestTimeoutInMs(10000)
}
val myHttp1 = buildHttp()
val myHttp2= buildHttp()

from reboot.

Thanks, yes, I think I've used the code above, but like you said, I can't reuse the underlying connection pooling.
Is there anyway to reuse the pooling? Say I don't want to create new client every time, and want to share underlying connection per timeout configuration.
I think if I don't change the timeout and use the default, the pooling works pretty well even multiple clients connect to multiple servers.
does that make sense?

from reboot.

I am also encountering this issue. I am calling configure in order to setFollowRedirects. To do this as transparently to my other code as possible, I have defined an Http function, which gets called in place of dispatch.Http:

def Http = dispatch.Http.configure(_ setFollowRedirects true)

Is there a better way to do this? I'd rather not put code in all of my handlers to follow 301/302 status codes, if I can help it. For my needs, I will always want to follow such redirects, so I thought this would make things simpler.

EDIT

I just took another look at dispatch.Http, and it was immediately apparent what I should do. By declaring my intermediate Http as a value (instead of a function) I will reuse the client, which has its setFollowRedirects set.

from reboot.

declaring my intermediate Http as a value (instead of a function)

Yep, that's what I was coming here to say but I'm glad you figured it out already.

from reboot.

It is even worse for me.

object Test extends App {

  import dispatch._, Defaults._

  val svc = url("http://api.hostip.info/country.php")

  val country = Http(svc OK as.String)

  for (c <- country)
    println(c)
}

If I run the above code, sbt will hang. I am not calling Http.configure even. And if I explicitly call Http.shutdown(), it would be fine.

And sometime I might get even stranger error:

ga:master:0.1-SNAPSHOT> last compile:run-main
[info] Running ...
java.lang.RuntimeException: Nonzero exit code returned from runner: 133
        at scala.sys.package$.error(package.scala:27)
        at sbt.BuildCommon$$anonfun$toError$1.apply(Defaults.scala:1356)
        at sbt.BuildCommon$$anonfun$toError$1.apply(Defaults.scala:1356)
        at scala.Option.foreach(Option.scala:197)
        at sbt.BuildCommon$class.toError(Defaults.scala:1356)
        at sbt.Defaults$.toError(Defaults.scala:34)
        at sbt.Defaults$$anonfun$runMainTask$2$$anonfun$apply$26.apply(Defaults.scala:513)
        at sbt.Defaults$$anonfun$runMainTask$2$$anonfun$apply$26.apply(Defaults.scala:512)
        at sbt.Scoped$$anonfun$hf4$1.apply(Structure.scala:580)
        at sbt.Scoped$$anonfun$hf4$1.apply(Structure.scala:580)
        at scala.Function1$$anonfun$compose$1.apply(Function1.scala:49)
        at sbt.Scoped$Reduced$$anonfun$combine$1$$anonfun$apply$12.apply(Structure.scala:311)
        at sbt.Scoped$Reduced$$anonfun$combine$1$$anonfun$apply$12.apply(Structure.scala:311)
        at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:41)
        at sbt.std.Transform$$anon$5.work(System.scala:71)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:232)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:232)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
        at sbt.Execute.work(Execute.scala:238)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:232)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:232)
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)
[error] (compile:run-main) Nonzero exit code returned from runner: 133

Something wrong with my code?

from reboot.

I also ran into this and was kind of banging my head on it for awhile.

package asdf

import dispatch._, Defaults._, as._

import java.io.FileOutputStream

object Main extends App {

  for (i <- 1 to 100) {
    HTTPDownloader.testdl("http://a.thumbs.redditmedia.com/SkXU16rGPG93eG6f.png");
    Thread.sleep(1000)
  }
}

object HTTPDownloader {

  val httpWait = Http.configure(_.setConnectionTimeoutInMs(2000))
  def httpWaitBugged(n: Int) = Http.configure(_.setConnectionTimeoutInMs(n))

  def testdl(file:String) {
    httpWaitBugged(2000)(url(file) OK as.Bytes) map { bytes =>
    //httpWait(url(file) OK as.Bytes) map { bytes =>
      val out = new FileOutputStream("file.png")
      out.write(bytes)
      out.close
    }
  }
}

When I run it in sbt it leaks file descriptors very gradually, but when I use one-jar or assembly to make it into a jar, it leaks about 40 descriptors every loop. That really hit me hard. It looks like such a safe method to call but it so isn't.

from reboot.

I'm hitting this too, except my app is huge, and runs with hundreds of local ips, so I hit descriptor limits in seconds :(

from reboot.

+1

from reboot.

This has been resolved in Dispatch 0.12.x and Dispatch 0.13.x and releases to Maven Central are forthcoming.

from reboot.