current git / current drakma git compilation failed about drakma-async HOT 22 CLOSED

Is this this latest Drakma from quicklisp? I'll update everything and see if I can reproduce this.

from drakma-async.

I tried with drakma from git

from drakma-async.

This is fixed, however it re-exposed the SBCL compilation bug you found. Closing this, re-opening "failed AVER" bug...

from drakma-async.

Nevermind, there was only a small error in the library caused by the packaging fixes that I did. I fixed this, and now all tests are passing in SBCL. Can you give it a shot when you have a minute?

(ql:quickload :drakma-async-test)
(drakma-async-test:run-tests)

Thanks for your help on this!

from drakma-async.

CL-USER> (drakma-async-test:run-tests)

....X....
Did 9 checks.
Pass: 8 (88%)
Skip: 0 ( 0%)
Fail: 1 (11%)

Failure Details:

WANT-STREAM [Makes sure :want-stream returns a stream that can be operated on again, and
also makes sure that :want-stream keeps the stream open]:
Unexpected Error: #
Argument X is not a NUMBER: NIL..

from drakma-async.

Would it be possible for you to run the code from the want-stream test directly and provide a backtrace? I'm having trouble reproducing.

(ql:quickload :drakma-async)

(defpackage :simple-drakma-test
  (:use :cl :cl-async-future))
(in-package :simple-drakma-test)

(as:with-event-loop ()
  (let ((socket1 nil))
    (multiple-future-bind (stream)
        (das:http-request "http://www.google.com/"
                          :want-stream t)
      (setf socket1 (das::get-underlying-socket stream))
      (multiple-future-bind (stream)
          (das:http-request "http://www.google.com/"
                            :stream stream
                            :want-stream t)
        (let ((sockets-equal-p (equal socket1 (das::get-underlying-socket stream))))
          (format t "equal? ~a~%" sockets-equal-p))
        (close stream)))))

from drakma-async.

Also, if the above code actually works, please try without the (close stream) at the end and see if the error pops up again. Thank you.

from drakma-async.

Same with and without close:

Closed TCP socket being operated on: #<SOCKET {100869A453}>.
[Condition of type CL-ASYNC:SOCKET-CLOSED]

Restarts:
0: [RETRY] Retry SLIME REPL evaluation request.
1: [*ABORT] Return to SLIME's top level.
2: [ABORT] Abort thread (#<THREAD "repl-thread" RUNNING {1003308063}>)

Backtrace:
0: (CL-ASYNC::CHECK-SOCKET-OPEN #<CL-ASYNC:SOCKET {100869A453}>)
1: (CL-ASYNC:WRITE-SOCKET-DATA #<CL-ASYNC:SOCKET {100869A453}> NIL :READ-CB #<CLOSURE (LAMBDA (DRAKMA-ASYNC::SOCK STREAM) :IN DRAKMA-ASYNC::HTTP-REQUEST-COMPLETE-STREAM) {10087379CB}> :WRITE-CB NIL :EVEN..
2: (DRAKMA-ASYNC::HTTP-REQUEST-COMPLETE-STREAM "www.google.com.ua" 80 #<CLOSURE (LAMBDA (STREAM) :IN DRAKMA-ASYNC:HTTP-REQUEST) {100873578B}> #<CLOSURE (LAMBDA (DRAKMA-ASYNC::EV) :IN DRAKMA-ASYNC:HTTP-RE..
3: (DRAKMA-ASYNC:HTTP-REQUEST #<PURI:URI http://www.google.com.ua/?gws_rd=cr> #<unavailable &REST argument> :PROTOCOL # :METHOD # :FORCE-SSL # :CERTIFIC..
4: ((LAMBDA NIL :IN DRAKMA-ASYNC::HTTP-REQUEST-ASYNC))
5: ((LAMBDA (STREAM) :IN DRAKMA-ASYNC:HTTP-REQUEST) #<CL-ASYNC:ASYNC-IO-STREAM {100869A4B3}>)
6: ((LAMBDA (DRAKMA-ASYNC::SOCK STREAM) :IN DRAKMA-ASYNC::HTTP-REQUEST-COMPLETE-STREAM) #<CL-ASYNC:SOCKET {100869A453}> #<CL-ASYNC:ASYNC-IO-STREAM {100869A4B3}>)
7: ((LAMBDA (SB-ALIEN::ARGS-POINTER SB-ALIEN::RESULT-POINTER FUNCTION) :IN "/home/avatar/.cache/common-lisp/sbcl-1.1.10-linux-x64/home/avatar/quicklisp/dists/quicklisp/software/cl-async-20130720-git/tcp...
8: ("foreign function: call_into_lisp")
9: ("foreign function: funcall3")
10: ("foreign function: #x20101840")
11: ("foreign function: #x200CD311C0")
12: (SB-INT:SIMPLE-EVAL-IN-LEXENV (CL-ASYNC:WITH-EVENT-LOOP NIL (LET ((SOCKET1 NIL)) (MULTIPLE-FUTURE-BIND (STREAM) (DRAKMA-ASYNC:HTTP-REQUEST "http://www.google.com/" :WANT-STREAM T) (SETF SOCKET1 (DRAKM..
13: (EVAL (CL-ASYNC:WITH-EVENT-LOOP NIL (LET ((SOCKET1 NIL)) (MULTIPLE-FUTURE-BIND (STREAM) (DRAKMA-ASYNC:HTTP-REQUEST "http://www.google.com/" :WANT-STREAM T) (SETF SOCKET1 (DRAKMA-ASYNC::GET-UNDERLYING-..
14: (SWANK::EVAL-REGION "(as:with-event-loop () ..)
15: ((LAMBDA NIL :IN SWANK::REPL-EVAL))
16: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA NIL :IN SWANK::REPL-EVAL) {10085A54BB}>)
17: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation request." #<CLOSURE (LAMBDA NIL :IN SWANK::REPL-EVAL) {10085A53DB}>)
18: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA NIL :IN SWANK::REPL-EVAL) {10085A53BB}>)
19: (SWANK::REPL-EVAL "(as:with-event-loop () ..)
20: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(as:with-event-loop () ..)
21: (EVAL (SWANK:LISTENER-EVAL "(as:with-event-loop () ..)
22: (SWANK:EVAL-FOR-EMACS (SWANK:LISTENER-EVAL "(as:with-event-loop () ..)
23: (SWANK::PROCESS-REQUESTS NIL)
24: ((LAMBDA NIL :IN SWANK::HANDLE-REQUESTS))
25: ((LAMBDA NIL :IN SWANK::HANDLE-REQUESTS))
26: (SWANK-BACKEND::CALL-WITH-BREAK-HOOK # #<CLOSURE (LAMBDA NIL :IN SWANK::HANDLE-REQUESTS) {100331011B}>)
27: ((FLET SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK :IN "/home/avatar/.emacs.d/elpa/slime-20130626.1151/swank-sbcl.lisp") # #<CLOSURE (LAMBDA NIL :IN SWANK::HANDLE-REQUEST..
28: (SWANK::CALL-WITH-BINDINGS ((STANDARD-OUTPUT . #1=#<SWANK-BACKEND::SLIME-OUTPUT-STREAM {10032EA8C3}>) (STANDARD-INPUT . #2=#<SWANK-BACKEND::SLIME-INPUT-STREAM {10031F75B3}>) (TRACE-OUTPUT . #1#)..
29: (SWANK::HANDLE-REQUESTS #<SWANK::MULTITHREADED-CONNECTION {1005B81DE3}> NIL)
30: ((FLET #:WITHOUT-INTERRUPTS-BODY-1120 :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
31: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
32: ((FLET #:WITHOUT-INTERRUPTS-BODY-537 :IN SB-THREAD::CALL-WITH-MUTEX))
33: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE) {7FFFF468EC3B}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THR..
34: (SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE #<SB-THREAD:THREAD "repl-thread" RUNNING {1003308063}> #S(SB-THREAD:SEMAPHORE :NAME "Thread setup semaphore" :%COUNT 0 :WAITCOUNT 0 :MUTEX #<SB-THREAD:MU..
35: ("foreign function: call_into_lisp")
36: ("foreign function: new_thread_trampoline")

from drakma-async.

Although I can't reproduce, I think this may be fixed. Can you pull and give it another shot? Thanks!

from drakma-async.

CL-USER> (drakma-async-test:run-tests)
.....f....
Did 10 checks.
Pass: 9 (90%)
Skip: 0 ( 0%)
Fail: 1 (10%)

Failure Details:

WANT-STREAM [Makes sure :want-stream returns a stream that can be operated on again, and
also makes sure that :want-stream keeps the stream open]:
SOCKET-EQUAL-P did not return a true value.

NIL

SIMPLE-DRAKMA-TEST> (as:with-event-loop ()
(let ((socket1 nil))
(multiple-future-bind (stream)
(das:http-request "http://www.google.com/"
:want-stream t)
(setf socket1 (das::get-underlying-socket stream))
(multiple-future-bind (stream)
(das:http-request "http://www.google.com/"
:stream stream
:want-stream t)
(let ((sockets-equal-p (equal socket1 (das::get-underlying-socket stream))))
(format t "equal? a%" sockets-equal-p))
(close stream)))))
equal? NIL
1
SIMPLE-DRAKMA-TEST>

without close same result (after little hanging)

from drakma-async.

Hmm, I'm beginning to wonder if the drakma API has changed slightly, because using :want-stream t in the latest drakma requires that :close nil be passed in order to be able to do two requests on the same stream.

I updated the want-stream test to explicitely pass :close nil when doing the requests, can you do a pull and give it another shot?

from drakma-async.

I tried with drakma from quicklisp and git. Result is the same

CL-USER> (defpackage :simple-drakma-test
(:use :cl :cl-async-future))
(in-package :simple-drakma-test)
#<PACKAGE "SIMPLE-DRAKMA-TEST">
SIMPLE-DRAKMA-TEST> (in-package :simple-drakma-test)
#<PACKAGE "SIMPLE-DRAKMA-TEST">
SIMPLE-DRAKMA-TEST> (as:with-event-loop ()
(let ((socket1 nil))
(multiple-future-bind (stream)
(das:http-request "http://www.google.com/"
:want-stream t)
(setf socket1 (das::get-underlying-socket stream))
(multiple-future-bind (stream)
(das:http-request "http://www.google.com/"
:stream stream
:want-stream t)
(let ((sockets-equal-p (equal socket1 (das::get-underlying-socket stream))))
(format t "equal? a%" sockets-equal-p))))))
equal? NIL
1

SIMPLE-DRAKMA-TEST> (as:with-event-loop ()
(let ((socket1 nil))
(multiple-future-bind (stream)
(das:http-request "http://www.google.com/"
:want-stream t)
(setf socket1 (das::get-underlying-socket stream))
(multiple-future-bind (stream)
(das:http-request "http://www.google.com/"
:stream stream
:want-stream t)
(let ((sockets-equal-p (equal socket1 (das::get-underlying-socket stream))))
(format t "equal? a%" sockets-equal-p))
(close stream)))))
equal? NIL
1

CL-USER> (drakma-async-test:run-tests)
.....f....
Did 10 checks.
Pass: 9 (90%)
Skip: 0 ( 0%)
Fail: 1 (10%)

Failure Details:

WANT-STREAM [Makes sure :want-stream returns a stream that can be operated on again, and
also makes sure that :want-stream keeps the stream open]:
SOCKET-EQUAL-P did not return a true value.

NIL

from drakma-async.

You're on SBCL 1.1.10, AMD64, what OS?

from drakma-async.

Linux/Gentoo (glibc 2.17/libevent 2.0.21)

from drakma-async.

Thanks. I'm going to try to match your environment as closely as I can (Linux64, SBCL 1.1.10, libevent 2.0.21) and see if I can reproduce. May take a few hours...

from drakma-async.

Ok, got everything running (Linux64 (glibc 2.15 is the best I can do), SBCL 1.1.10, libevent 2.0.21) and all tests are passing =[. Did you compile SBCL from source or grab the binary? I grabbed the binary from the site.

Can you try cloning cl-async, cl-libevent2, and drakma-async again and putting them in ~/quicklisp/local-projects/? I want to make sure there's nothing out of date on your machine that could be causing this.

from drakma-async.

Hm. Let's see. I remove the fasls cache and now shure that cl-async, cl-libevent2, drakma, drakma-async compiled from local-projects (git), but result is the same.

I found out problem. Looks like stream is not preserved while redirects. As i'm from ukraine, google.com redirects me to google.com.ua:

(trace drakma-async:http-request)
DRAKMA-ASYNC-TEST> (as:with-event-loop ()
  (let ((socket1 nil))
    (multiple-future-bind (stream)
        (das:http-request "http://www.google.com/"
              :close nil
              :keep-alive t
                          :want-stream t)
      (setf socket1 (das::get-underlying-socket stream))
      (multiple-future-bind (stream)
          (das:http-request "http://www.google.com/"
                            :stream stream
                :close nil
                :keep-alive t
                            :want-stream t)
        (let ((sockets-equal-p (equal socket1 (das::get-underlying-socket stream))))
      (print (slot-value socket1 'cl-async::c))
      (print (slot-value (das::get-underlying-socket stream) 'cl-async::c))
          (format t "equal? ~a~%" sockets-equal-p))
        (close stream)))))
  0: (HTTP-REQUEST "http://www.google.com/" :CLOSE NIL :KEEP-ALIVE T
                   :WANT-STREAM T)
  0: HTTP-REQUEST returned
       #<FUTURE
         callback(s): 0
         errback(s): 0
         finished: NIL
         forward: NIL {1006BF6483}>
  0: (HTTP-REQUEST #<PURI:URI http://www.google.com.ua/?gws_rd=cr> :METHOD :GET
                   :REDIRECT 4 :ADDITIONAL-HEADERS NIL :PARAMETERS NIL
                   :PRESERVE-URI T :FORCE-SSL NIL :CLOSE NIL :KEEP-ALIVE T
                   :WANT-STREAM T)
  0: HTTP-REQUEST returned
       #<FUTURE
         callback(s): 0
         errback(s): 0
         finished: NIL
         forward: NIL {1006C1DB93}>
  0: (HTTP-REQUEST "http://www.google.com/" :STREAM
                   #<FLEXI-STREAMS:FLEXI-IO-STREAM {1006C24D13}> :CLOSE NIL
                   :KEEP-ALIVE T :WANT-STREAM T)
  0: HTTP-REQUEST returned
       #<FUTURE
         callback(s): 0
         errback(s): 0
         finished: NIL
         forward: NIL {1006C961F3}>
  0: (HTTP-REQUEST #<PURI:URI http://www.google.com.ua/?gws_rd=cr> :METHOD :GET
                   :REDIRECT 4 :ADDITIONAL-HEADERS NIL :PARAMETERS NIL
                   :PRESERVE-URI T :FORCE-SSL NIL :STREAM
                   #<FLEXI-STREAMS:FLEXI-IO-STREAM {1006C24D13}> :CLOSE NIL
                   :KEEP-ALIVE T :WANT-STREAM T)
  0: HTTP-REQUEST returned
       #<FUTURE
         callback(s): 0
         errback(s): 0
         finished: NIL
         forward: NIL {1006D096B3}>

#.(SB-SYS:INT-SAP #X7FFFE80123B0) 
#.(SB-SYS:INT-SAP #X7FFFE8012EF0) equal? NIL
1

DRAKMA-ASYNC-TEST> (as:with-event-loop ()
  (let ((socket1 nil))
    (multiple-future-bind (stream)
        (das:http-request "http://www.google.com.ua/"
              :close nil
              :keep-alive t
                          :want-stream t)
      (setf socket1 (das::get-underlying-socket stream))
      (multiple-future-bind (stream)
          (das:http-request "http://www.google.com.ua/"
                            :stream stream
                :close nil
                :keep-alive t
                            :want-stream t)
        (let ((sockets-equal-p (equal socket1 (das::get-underlying-socket stream))))
      (print (slot-value socket1 'cl-async::c))
      (print (slot-value (das::get-underlying-socket stream) 'cl-async::c))
          (format t "equal? ~a~%" sockets-equal-p))
        (close stream)))))
  0: (HTTP-REQUEST "http://www.google.com.ua/" :CLOSE NIL :KEEP-ALIVE T
                   :WANT-STREAM T)
  0: HTTP-REQUEST returned
       #<FUTURE
         callback(s): 0
         errback(s): 0
         finished: NIL
         forward: NIL {1007236483}>
  0: (HTTP-REQUEST "http://www.google.com.ua/" :STREAM
                   #<FLEXI-STREAMS:FLEXI-IO-STREAM {100723F003}> :CLOSE NIL
                   :KEEP-ALIVE T :WANT-STREAM T)
  0: HTTP-REQUEST returned
       #<FUTURE
         callback(s): 0
         errback(s): 0
         finished: NIL
         forward: NIL {1007294AA3}>

#.(SB-SYS:INT-SAP #X7FFFE80123B0) 
#.(SB-SYS:INT-SAP #X7FFFE80123B0) equal? T
1
DRAKMA-ASYNC-TEST> 

from drakma-async.

What happens if you (trace drakma-async::http-request-complete-stream) instead of http-request when doing the test that redirects?

I tested the redirects on my end earlier (I also thought that was the issue) by connecting to "http://google.com" (which forwards to "http://www.google.com") but still having trouble reproducing.

from drakma-async.

Good news, finally was able to reproduce. Thanks for helping me, should have a fix shortly.

from drakma-async.

You're right, it was the redirect. I thought I had tested for this, but I was wrong.

So actually, the code in drakma/drakma-async is correct. If an address redirects to another host, the stream should be closed and a new one opened because it might need to be connected to another server.

So really, I just ended up changing the test to use "http://www.google.com.ua" since the test was the one expecting a result it shouldn't have been getting.

Thanks again for bearing through this with me. Please give the test suite another shot when you have a minute and let me know if you have any failures (the proxy tests sometimes fail randomly, so if they do, give it another run).

from drakma-async.

Sorry for delay - had a loong week. Currently all tests successfully passed for me:

CL-USER> (drakma-async-test:run-tests)
..........
Did 10 checks.
Pass: 10 (100%)
Skip: 0 ( 0%)
Fail: 0 ( 0%)

NIL

from drakma-async.

Awesome! thank you for testing again.

from drakma-async.