orthecreedence / cl-async Goto Github PK

For some reason, whenever libevent2 loads via asdf, it loads twice. Cannot figure out why.

Note that this breaks on SBCL, which bitches about redefining constants.

Some library wants to use the stream-element-type method on streams, and this isn't implemented on async-stream.

For tcp-send and http-client, you can specify an IPV4 address or a hostname, but if you pass an IPV6 address, it will work, but only by doing an unnecessary DNS lookup. For tcp-server or http-server, passing an IPV6 address will break.

Also need to add IPV6 support to dns-lookup, which only supports IPV4 lookups for now.

Need to wrap the IPV6 address classes in the bindings and update the internal ip-address-p to return whether we're looking at IPV4 or IPV6 and create the appropriate c struct to pass into the connect function.

This only applies to dns-lookup, everything else should support IPV6 automatically, to my understanding (since it's mostly abstracted away by libevent).

I get an error trying to load cl-async on Windows (sbcl-1.1.0.36 x64 from akovalenko/sbcl-win32-threads)

CL-USER> (ql:update-all-dists)
1 dist to check.
You already have the latest version of "quicklisp": 2012-10-13.
NIL
CL-USER> (ql:quickload "cl-async")
To load "cl-async":
  Load 1 ASDF system:
    cl-async
; Loading "cl-async"

Here's the output from the above session. I've tried recompiling via option 3 but just get the same error again.

Unable to load foreign library (LIBEVENT2).
  Error opening shared object "libevent.dll":
  126.
   [Condition of type CFFI:LOAD-FOREIGN-LIBRARY-ERROR]

Restarts:
 0: [RETRY] Try loading the foreign library again.
 1: [USE-VALUE] Use another library instead.
 2: [TRY-RECOMPILING] Recompile libevent2 and try loading it again
 3: [RETRY] Retry loading FASL for #<CL-SOURCE-FILE "libevent2" "libevent2">.
 4: [ACCEPT] Continue, treating loading FASL for #<CL-SOURCE-FILE "libevent2" "libevent2"> as having been successful.
 5: [ABORT] Give up on "cl-async"
 --more--

Backtrace:
  0: (CFFI::FL-ERROR "Unable to load foreign library (~A).~%  ~A" LIBEVENT2::LIBEVENT2 "Error opening shared object \"libevent.dll\":\n  126.")
  1: (CFFI::REPORT-SIMPLE-ERROR LIBEVENT2::LIBEVENT2 #<SIMPLE-ERROR "Error opening shared object ~S:~%  ~A." {1005135053}>)
  2: (CFFI::LOAD-FOREIGN-LIBRARY-PATH LIBEVENT2::LIBEVENT2 "libevent.dll" NIL)
  3: ((FLET CFFI::%DO-LOAD :IN CFFI::%DO-LOAD-FOREIGN-LIBRARY) #<CFFI:FOREIGN-LIBRARY LIBEVENT2> LIBEVENT2::LIBEVENT2 (:DEFAULT "libevent"))
  4: (CFFI:LOAD-FOREIGN-LIBRARY LIBEVENT2::LIBEVENT2 :SEARCH-PATH NIL)
  5: (SB-FASL::LOAD-FASL-GROUP ..)
  6: (SB-FASL::LOAD-AS-FASL ..)
  7: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) ..)
  8: (LOAD ..)
  9: (SB-IMPL::%MAP-FOR-EFFECT-ARITY-1 ..)
 10: ((SB-PCL::FAST-METHOD ASDF:PERFORM (ASDF:LOAD-OP ASDF:CL-SOURCE-FILE)) ..)
 11: ((SB-PCL::EMF ASDF:PERFORM) #<unavailable argument> #<unavailable argument> #<ASDF:LOAD-OP (:VERBOSE NIL) {10035911F3}> #<ASDF:CL-SOURCE-FILE "libevent2" "libevent2">)
 12: ((SB-PCL::FAST-METHOD ASDF::PERFORM-WITH-RESTARTS (ASDF:LOAD-OP ASDF:CL-SOURCE-FILE)) ..)
 13: ((SB-PCL::FAST-METHOD ASDF::PERFORM-WITH-RESTARTS :AROUND (T T)) ..)
 14: ((LAMBDA () :IN ASDF::PERFORM-PLAN))
 15: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-C::%WITH-COMPILATION-UNIT))
 16: ((FLET #:WITHOUT-INTERRUPTS-BODY-102694 :IN SB-THREAD::CALL-WITH-RECURSIVE-LOCK))
 17: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK ..)
 18: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
 19: ((SB-PCL::FAST-METHOD ASDF::PERFORM-PLAN (LIST)) ..)

It would be useful to be able to monitor an fd without having to wrap a bufferevent around it, in case you just wanted to be notified of new data, for instance if integrating cl-async with an existing driver.

It would be easy to wrap the event portion of libevent, but the tricky part would most likely be intelligently freeing the event. Right now, cl-async generally takes care of all freeing of resources depending on what events are processed, and this would introduce an API where the library user would have to be aware memory management, which I'm not too keen on.

Perhaps it's not a big deal though, as long as I make note of it in the docs.

Right now, http-client always assumes request -> response -> close. it would be nice to be able to re-use a connection if needed to keep TCP overhead down if getting multiple resources from the same server (if scraping, for instance).

I want to be able to benchmark clients/servers automatically. Would be good to have a benchmark.lisp file in the tests folder, and a (cl-async-test:run-benchmarks) function. Might make sense to multithread it to give it a tad bit more authenticity.

Libevent supports chunking in the HTTP server. If transferring large files, use NginX. Just kidding, if transferring large files, it would make sense to not have to read the entire file into memory and send it all at once when it could be streamed off the disk in chunks.

tcp-connect works great, but if you want to initialize a cl-async socket without connecting it, it makes sense to have a lower-level function that creates a socket without connecting it. tcp-connect could accept this socket via a keyword parameter (in which case it would ignore the host/port args).

This might get a bit confusing, so maybe it makes sense to create two new functions: init-tcp-socket and connect-socket. init-tcp-socket would do everything tcp-connect does except connect the socket, and connect-socket would just connect it. tcp-connect would then just be a thin layer that ties the two together. This makes tcp-connect function exactly as it does now, but offers lower-level functions for more control, all without duplicating functionality.

Note that this isn't essential, but would make the API a lot less confusing (especially as far as wrapping an existing fd that has already been connected).

I'm trying to integrate cl-postgres and cl-async together. I've opened a bug and I am working on an implementatino, see marijnh/Postmodern#28

My current problem, is that all of cl-postgres is synchronous. That means I need to be able to put the socket into synchronous mode so it can write some requests, read its expected replies, and then give back the hand to the loop.

Currently, I provide a callback function to cl-postgres for it to call when it wants a stream to communicate with PostgreSQL. The problem is that it calls it and gets a stream from tcp-connect. Then it tries to authenticate right away, and it gets "end of file" really soon. I think this is due to the fact there's nothing to read at this precise moment, but I'd prefer the read function to block, in this case.

For some reason, libevent's http server implementation does not retrieve an HTTP request body when parsing a GET request. There must be a way around this.

static int
evhttp_method_may_have_body(enum evhttp_cmd_type type)
{
    switch (type) {
    case EVHTTP_REQ_POST:
    case EVHTTP_REQ_PUT:
    case EVHTTP_REQ_PATCH:
        return 1;
    case EVHTTP_REQ_TRACE:
        return 0;
    /* XXX May any of the below methods have a body? */
    case EVHTTP_REQ_GET:
    case EVHTTP_REQ_HEAD:
    case EVHTTP_REQ_DELETE:
    case EVHTTP_REQ_OPTIONS:
    case EVHTTP_REQ_CONNECT:
        return 0;
    default:
        return 0;
    }
}

This appears to be the culprit, http.c. LAME. Need to figure out a way around this (or just dump the http server/client???)

The examples in the docs are a bit sparse.

The library is at a point now where the API is taking shape, so makes sense to write some official tests.

For some reason, setting the TCP listener's error handler up creates a segault (when it's set up via (le:evconnlistener-set-error-cb listener (cffi:callback tcp-accept-err-cb)), not when an error occurs).

Need to investigate this. For now it's disabled.

If :time is 0 in delay, then it makes sense to just make a normal user-triggered event with the given callback attached and fire it instead of using a special timer event. The interface/functionality would be the same but there might be a small performance boost.

For getting drakma-async going fully, I need an SSL socket. Libevent provides this, so I need to wrap it and make sure the stream implementation works over it.

I keep getting a socket timeout using wrap-in-ssl and I think something's wrong. Here's a pretty simple script to reproduce:

(cl-async:start-event-loop
 (lambda ()
   (let* ((socket (cl-async:tcp-connect "www.google.com" 443
                                        (lambda (sock data)
                                          (declare (ignore sock))
                                          (format t "GOT: ~a~%" data))
                                        (lambda (ev)
                                          (format t "event: ~a~%" ev))
                                        :stream t
                                        :read-timeout 3))
          ;; now wrap the normal socket in SSL, save the new socket
          (socket-ssl (cl-async-ssl:wrap-in-ssl socket)))
     (format t "socket ~a~%" socket-ssl))))

This prints

socket #<ASYNC-IO-STREAM {10069D3253}>
event: TCP connection timeout: -1: Socket timed out

And exits.

One way to trick this and make it work a bit better is to use, as in the example, a call to write-socket-data, even empty. But soon after, the socket is closed with the same error.

If that's not a bug in the code, I think something should at least be put up in the documentation, because right now I'm a bit lost with my closed socket. :-)

Get the error:

[err] http.c:2291: Assertion TAILQ_FIRST(&evcon->requests) == req failed in evhttp_send

There must be a way to gracefully exit an HTTP server, finishing the current requests without accepting new ones.

Would allow an app to get detailed information on DNS lookups.

I submitted a request to have cl-libevent2 included in Quicklisp since I think it's about stable. Once it's included, need to update cl-async.asd to use cl-libevent2 instead of libevent2 in the deps.

Also, remove the libevent2/ subfolder once this is done (and tested, of course).

When loading cl-async on Ubuntu 12.04 LTS, I get the following error:

; Loading "libevent2"
[package cffi-sys]................................
[package cffi]....................................
..................................................
[package cffi-features]...........................
[package libevent2]...............................
[package libevent2.accessors]................
; Loading "cl-async"
[package cl-async]
debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "initial thread" RUNNING {AAF28D1}>:
  Unknown CFFI type: LIBEVENT2::SOCKADDR-IN-6.

Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY ] Retry compiling #<CL-SOURCE-FILE "cl-async" "util">.
  1: [ACCEPT] Continue, treating compiling #<CL-SOURCE-FILE "cl-async" "util">
              as having been successful.
  2: [ABORT ] Give up on "cl-async"
  3:          Exit debugger, returning to top level.

(CFFI::FIND-TYPE-PARSER LIBEVENT2::SOCKADDR-IN-6)
0] 

...and make data an optional keyword, also add a tcp-send wrapper that just calls tcp-connect for backwards compatibility.

Also, add a connect-cb to tcp-connect.

This is a breaking API change (once tcp-send is fully deprecated), but makes much more sense. Sockets are connected, then data is sent (generally), not transparently used to blast data out.

It seems client-side certificates can be used for user authentication. While the server currently supports server-side certs, it doesnt support verification of client-side certs (and tcp-ssl-connect doesn't support sending them).

It would be a great exercise if both the client and server could deal with client-side certificate exchanges.

For some idiotic reason, DNS requests are not happening in parallel, but instead at most two at a time, and sometimes one at a time (serially).

I'm wondering if this has something to do with there only being one DNS base, although that seems to be the way to go according to the sparse docs on the subject.

Need to investigate further.

Create one global *dns-base* for all DNS queries. It takes a while to instantiate the evdns object, so it would be useful to have one that serves all the DNS queries. Note that since the recent update where (create-data-pointer) is used internally for all data referencing, this is now 100% possible, whereas before the DNS callbacks were referenced by DNS base (meaning 1 DNS base for 1 DNS lookup). Now one DNS base can serve unlimited requests and could be closed if there are no pending DNS lookups.

Replace all (le:evdns-base-new ...) calls with (get-dns-base). if *dns-base* is null, (get-dns-base) returns (le:evdns-base-new ...), otherwise returns *dns-base*. either way, it increments the *dns-ref-count* variable. When (free-dns-base) is called, if *dns-ref-count* is 1, it frees the dns-base and sets *dns-base* to nil, otherwise it decrements *dns-ref-count* and returns without freeing.

this way, *dns-base* will only exist if it's being used, and gets freed when there are no more references to it (so the event loop can close), but doesn't get instantiated on every DNS call.

Signal handling. Libevent supports signal handling events, but they aren't really caught in lisp land. Will probably end up implementing something like cl-signal-handler with app-fired, custom events in conjuction with libevent's signal handling. Attacking the problem from both sides should get it working.

I'm trying to incorporate a cl-postgres into an event loop, in order to receive notification.

Here's my code:

(require :cl-async)
(require :postmodern)
(require :usocket)

(postmodern:connect-toplevel "db" "user" "pass" "localhost")

(defun get-socket-fd (connection)
  "Gets the fd from a socket connection."
  (let* ((type (type-of connection))
         (stream (cond ((subtypep type 'usocket:stream-usocket)
                        (usocket:socket-stream connection))
                       ((subtypep type 'stream)
                        connection))))
    (when stream
      #+sbcl
        (sb-sys:fd-stream-fd stream)
      #+cmu
        (system:fd-stream-fd stream)
      #+ccl
        (ccl::ioblock-device (ccl::stream-ioblock stream t))
      #+clisp
        (ext:stream-handles stream))))

(defun notification-handle (socket data)
  (declare (ignore socket data))
  ;; (format t "read char from socket ~a~%" (cl-postgres::read-str (cl-postgres::connection-socket postmodern:*database*))))
  (format t "Reading notification~%")
  (multiple-value-bind (channel payload pid) (cl-postgres:wait-for-notification
                                              postmodern:*database*)
    (format t "Notification received: ~a ~a ~a~%" channel payload pid)))

(defun start ()
  (postmodern:execute "LISTEN foobar")
  (as:init-tcp-socket #'notification-handle
                      (lambda (ev)
                        (format t "Event ~a ~%" ev))
                      :fd (get-socket-fd
                           (cl-postgres::connection-socket postmodern:*database*))
                      :dont-drain-read-buffer t)
  (format t "Started~%"))

(cl-async:start-event-loop #'start)

It's actually pretty simple: listen on foobar, and as soon as the socket receive a message, we read notification from the wire and print it out.

But this doesn't work as expected. If I run the program it prints

Started

Then I send a notification from psql with:

NOTIFY foobar, 'notif1';

The program prints Reading notification… but blocks, whereas it should read notif1 right away!

So I send a second notification with:

NOTIFY foobar, 'notif2';

And now it prints:

Notification received: foobar notif2 15438

So something ate up the first notification. If I do this with a 3rd and 4th notification, the 3th notification never appears too, and I get the 4th, etc.

Any hint?

It would be nice if an app already has an existing FD (via usocket, cl+ssl, etc) to be able to wrap it in a cl-async socket.

Also need ability to wrap existing SSL fd.

Get the following:

DNS log: 0: Added nameserver 76.14.0.8:53 as 006c6708
DNS log: 0: Successfully added 76.14.0.8 as nameserver
DNS log: 0: Added nameserver 76.14.0.9:53 as 006c6928
DNS log: 0: Successfully added 76.14.0.9 as nameserver
DNS log: 0: Added nameserver 76.14.96.14:53 as 006c6a68
DNS log: 0: Successfully added 76.14.96.14 as nameserver
DNS log: 0: Could not add nameserver 76.14.0.8 to list,error: 0
DNS log: 0: Could not add nameserver 76.14.0.9 to list,error: 0
DNS log: 0: Could not add nameserver 76.14.96.14 to list,error: 0
DNS log: 0: No nameservers added.
DNS log: 0: Didn't find nameservers in nt_key/"NameServer"
DNS log: 0: Didn't find nameservers in nt_key/"DhcpNameServer"
DNS log: 0: Didn't find nameservers in interfaces_key/"NameServer"
DNS log: 0: Didn't find nameservers in interfaces_key/"DhcpNameServer"
DNS log: 1: Didn't find any nameservers.
DNS log: 0: Sending request for api.musio.dev on ipv4 as 006b9110
DNS log: 0: Resolve requested for api.musio.dev
DNS log: 0: Setting timeout for request 006c2120, sent to nameserver 006c6708
DNS log: 0: Sending request for api.musio.dev on ipv6 as 006b9118
DNS log: 0: Resolve requested for api.musio.dev
DNS log: 0: Setting timeout for request 006c1f20, sent to nameserver 006c6a68
DNS log: 0: Removing timeout for request 006c1f20
DNS log: 0: Removing timeout for request 006c2120
err: Connection DNS error: 11001, nodename nor servname provided, or not known

Seems relevant:

DNS log: 0: Could not add nameserver 76.14.0.8 to list,error: 0
DNS log: 0: Could not add nameserver 76.14.0.9 to list,error: 0
DNS log: 0: Could not add nameserver 76.14.96.14 to list,error: 0
DNS log: 0: No nameservers added.

But doesn't happen when resolving remote addresses. Why?

Inheriting fundamental-input|output-binary-stream before async-stream while definine the async-input|output-stream stream classes causes the close method to load for the fundamental-*-stream instead of the defined async-stream in SBCL.

When doing HTTP load testing, the memory seems to climb steadily and not be reclaimed once testing is over. This points to a mem leak somewhere.

A somewhat consistent segfault has shown up in windows. Not sure which test(s) are causing it, but I have an inkling it might be the init-tcp-socket/connect-tcp-socket split.

So first off, this project gets like 1000 stars for having an unprecedented amount of documentation (compared to other Lisp projects).

But, it seems a bit much for all of it to be in the README...

I'll put together a branch with a proposal for maximum impact on the front page with easy links to everything else...

If you hate this idea completely, just say the word ;-)

I do not know why. My gut says it's a type mismatch between 32bit and 64bit, but playing with the types in the bindings for the DNS functions doesn't seem to do any good.

Note that this may also be broken on Windows 64bit (I haven't tested yet).

Any ideas here would be appreciated. My CCL backtrace (slackware x64) is as follows:

*** glibc detected *** ccl: double free or corruption (!prev): 0x0000000000678c10 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76ce6)[0x7f2951491ce6]
/lib64/libc.so.6(cfree+0x73)[0x7f2951498553]
ccl(_SPffcall+0x75)[0x412855]

but I'm certain I'm not "double freeing."

This works on Windows 32bit.

It'd be cool to have a function that returns the current state: how many incoming/outgoing connections, how many open dns queries, etc.

Wrap sockets/fds in their own classes, since right now there are CFFI pointers flying around willy nilly. While this works fine if people follow the API, it can potentially be disastrous as well. Wrapping pointers in classes would also allow methods to be used, which would cut down a good number of segfaults, I'm guessing.

Hi,

I wanted to implement a repl server over tcp.

(defun repl-server ()
(format t "Starting server.~%")
(as:tcp-server nil 9003 ; nil is "0.0.0.0"
(lambda (socket stream)
(print (eval (read stream)) stream))
(lambda (err) (format t "listener event: a%" err)))
;; catch sigint
(as:signal-handler 2 (lambda (sig)
(declare (ignore sig))
(as:exit-event-loop))))

This required me to define method (stream-read-char) (stream-unread-char) (stream-write-char) in tcp-streams. However I am now facing issues with the streams being binary and not understanding #\Newline etc. I want the streams to be flexible to accept character streams as well. How do I go about doing this?

-Nix

Rename:

connection-info   ->  event-info
connection-error  ->  event-error
conn-errcode      ->  event-errcode
conn-errmsg       ->  event-errmsg

These new names make much more sense considering many events are not "connection" events, but are simply base events. If you need to test for a specific connection event, you can use the events that whatever connection type you're using (tcp, http) export: tcp-info, tcp-error, etc.

It's really annoying, i realized, to have to search down for

;; definition

in the docs to get the definition of a function when you may just want a quick reference. It doesn't parse well.

For all functions/macros, I want to move the ;; definition section to directly under the title and ALSO make sure a return value is specified (if applicable).

Hi,

I needed to add

(:darwin "libevent.dylib")

in libevent2/libevent2.lisp in order to load libevent2 correctly on OS X. Remark that this line must appear before the (:unix ...) line, as the first of these is used on OS X. I guess it may be possible to stick it in as an early alternative in (:unix ...)as well, but maybe better to keep it separate?

• SSL if stringp
• future event handler refs
• future mfb tiny wrapper note

Accepts the socket object as the only arg.

Like issue #41 but for SSL sockets (libevent also supports this, so should be a pushover).

Don't you plan migrating to https://github.com/joyent/libuv ?

After hacking on cl-event for about a week, trying to get it working, I did some more googling and found this recent and excellent work! Nice going :-)

As for critical mass: I'm with you! Having lived (and continuing to live) in Twisted land (async for Python), I desperately need an async solution for Lisp (sbcl, in my case).

I'll see what I can do about promoting this project around the interwebs...

Thanks so much!

Right now, when sending data through http-client, you must send a full request. It would be useful to be able to instantiate an http-client and stream data into it bit by bit. This would allow a lot better memory usage when sending large amounts of data.