orthecreedence / blackbird Goto Github PK

does (push :future-debug features) still work?

Check out using dissect for saving stack traces when calling attach (or maybe there's a smarter/higher-level way to get traces).

As of now, promise error callbacks are always invoked immediately and
promise success callbacks are only deferred if *promise-finish-hook*
is set, e.g.:

(setf blackbird:*promise-finish-hook*
      #'(lambda (fn)
          (as:with-delay () (funcall fn))))

That's quite unlike js promises, where both success and error
callbacks are always executed only when control reaches the event
loop.

The problem is that code that uses immediate callbacks often needs to
be written in very different style from the code that uses deferred
callbacks, and changing blackbird:*promise-finish-hook* may break
some of the code depending on which style it is using.

Although immediate callback invocation is faster, it may cause some
very weird code flow. E.g. you may have some code like

(defun talk-to-server (message)
  (bb:with-promise (resolve reject)
    (set-message-handler
     #'(lambda (message)
         (resolve message)
         (clear-message-handler)))
    ;; send-message returns immediately
    (send-mesage message)))

(defun dialogue ()
  (bb:alet ((response (talk-to-server message-1)))
    (talk-to-server (make-message-2-for-response response))))

Let's see what happens in dialogue. First talk-to-server
call is made, which returns a promise which will be resolved
sometime in future when a message arrives from the server.
Until the message arrives, there's a message handler installed
that will resolve the promise.
Then, when the message is received, the promise is resolved
and second talk-to-server call is made. It sets a new message
handler (which may override the previous one, ok), initiates
message sending and returns a promise. The problem is that
the first (resolve message) call from the first (talk-to-server)
call returns at this point, and (clear-message-handler)
is called, killing the newly installed message handler.
Such very strange "blast from the past" thing made me
spend about 3 hours trying to debug a similar problem
in my MQTT driver. Other, even more weird "unwanted reentrancy"
pattens may occur, too.

Deferring error callbacks is also important because such
callbacks may activate other long promise chains.

I'd suggest either making some bb-async ASDF system which can be added
to either cl-async or blackbird, and/or using
asdf-system-connections to set blackbird:*promise-finish-hook* to
delaying function when cl-async is available.

Perhaps error callbacks should be updated to use
blackbird:*promise-finish-hook*, too.

Oi, I'm unsure if this is a bug, I'm misunderstanding how promises are supposed to work or making a simple mistake.

When writing some convenience macros for when working with rethinkdb the &body of both macros as illustrated by the code below. Any pointers as to what could be going wrong?

(eval-when (:compile-toplevel :execute)
  (ql:quickload '(alexandria cl-rethinkdb cl-async blackbird)))

(defpackage #:foo
  (:use #:cl #:alexandria #:cl-rethinkdb #:cl-async #:blackbird))

(in-package #:foo)


(defvar *host* "127.0.0.1"
  "The address of the database")
(defvar *port* 28015)

(defmacro with-db ((socket-name &key (host *host*) (port *port*))
                   &body body)
  "Run the BODY with a socket bound to SOCKET-NAME "
  `(alet ((,socket-name (connect ,host ,port)))
     (unwind-protect (progn
                       ,@body)
       (disconnect ,socket-name))))

(defmacro with-query ((result query) &body body)
  "Run the BODY with the result of QUERY bound to RESULT "
  (with-gensyms (socket-name)
    `(with-db (,socket-name)
       (alet ((,result (run ,socket-name (r ,query))))
         (progn
           ,@body
           ,result)))))

#+works
(as:with-event-loop ()
  (alet* ((sock (connect "127.0.0.1" 28015))
          (query (r (:db-list)))
          (value (run sock query)))
    (prog1
        value
      (format t "~A~%" value)
      (disconnect sock))))

#+returns-0-does-not-print-anything
(as:with-event-loop ()
  (with-db (sock)
    (alet* ((query (r (:db-list)))
            (value (run sock query)))
      (prog1
          value
        (format t "~A~%" value)))))

#+does-not-work
(progn
  (defvar *foo*)

  (as:with-event-loop ()
    (with-query (value (:db-list))
      (setf *foo* 1)
      (format t "~A~%" value)))

  *foo*)
;; *foo* is unbound, nothing is printed

P.D. Happy Holidays and thanks for cl-async, wookie and cl-rethinkdb!

Hi.

I'd like to use 'blackbird' instead of creating my own 'future' framework.
But I'm having a few headaches with the terminologies used here.

I.e.: why is it called 'promise'?
As I understand 'futures' and 'promises' is that a 'future' represents a future computation and a 'promise' will fulfill it, or set the value of the future.
Which would mean that 'promise' in this library actually is a 'future' and the 'resolve-fn'/'reject-fn' actually represents the 'promise'.

Also things like 'attach' are quite surprising. I would think that a function name like 'on-complete' or 'on-fulfilled' would be easier to understand.

So my question here would be: what are the chances that some name changes could go into this library if I would create a PR?

I had some experience with JavaScript promises in the past and would
like to share some possible ways to improve blackbird. After all,
async is an important part of in-browser programming and they worked
out some rather convenient ways to deal with it.

Some pointers:

• Promises/A+ spec https://promisesaplus.com/
• kriskowal's Q library http://documentup.com/kriskowal/q
• MDN docs on promises
  https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Promise

If you agree with some or all of the points below, I'll be glad to
contribute some code to actually implement them. Some of the features
may affect performance and thus I think should be optional.

1. Clean separation of promise producers and consumers

A possible way to separate such code:

(let ((promise
        (make-promise
         #'(lambda (resolve reject)
             (with-delay (0.3)
               (if ok
                   (funcall resolve 42)
                   (funcall reject 'my-error :format-control "...")))))))
  ...)

Or, with convenience macro:

(let ((promise
        (with-promise (resolve reject)
          (as:with-delay (...)
            (if ok
                (resolve 42)
                (reject 'my-error :format-control "..."))))))
  ...)

Additional advantage, besides improved code structure: when code is
organized this way, it's harder to accidentally 'fulfill' a promise
multiple times.

finish and signal-error may be kept for a while for compatibility
reasons.

2. .then()-like semantics

Both attach and attach-errback should always return a new
promise that is fulfilled when the callback finishes. In both cases
callbacks should be able to return a new promise, in which case the
promise returned from attach or attach-errback is only
finished when the promise returned from the callback finishes. (As
far as I understand, currently all of this only works for callbacks).
When an error (or just a serious-condition) is signalled inside a
promise callback or errback, it should be caught causing the promise
returned from attach / attach-errback to be rejected (as with
signal-error). All this may sound as if it may make debugging
harder, but see below.

Promise should not 'accumulate' events, it should be possible to
resolve/reject the only once.

Also, maybe attach-errback should be renamed to make it look
better. Naming it after Q's .fail() may not be a good idea, need to
invent something more appropriate.

3. Debugging aid

Determining what actually happened when looking at an error in the
middle of a long promise chain is PITA. Luckily this can be fixed to
some extent.

First of all, it's possible to add an option to capture the current
stack trace in attach / attach-errback and make it possible to
access it from the callback context (joining it with any stack traces
captured by attach / attach-errback higher up the chain).
With some amount of hacking I think it should be even possible to make
SLIME actually display this stack trace tail in sldb buffers, but even
just printing it to REPL buffer may help.

Second, there's such thing as .done() in Q. Although it's described
as a 'stopgap' in the docs, it may help us, too. See:
http://documentup.com/kriskowal/q/#tutorial/the-end The idea is that
every callback chain should be topped with .done() (or wrapped in
something like (bb:end ...) in our case). This will catch any
unhandled conditions that propagated to the end of promise chain and
signal them so they can be handled by debugger or some app-level
global uncaught error handler.

Another measure is an option to use trivial-garbage to attach
finalizer to promise objects so that 'orphaned' promises with
unhandled errors are detected and their stack traces are printed
during GC. It will require a bit of trickery because promise's
finalizer cannot access the promise itself, but nevertheless it's
still not that hard to implement. There's also possibility to use repl
eval hooks to catch orphaned promises that are created during
evaluation of a form entered in REPL, such as (run-tests).

It's possible to also add 'strict' option which will require all
promise chains to be explicitly topped with (bb:end ...) and will
produce warnings with stack traces if any 'orphaned' promises are
detected during GC, with pending unhandled conditions or without.

4. Forced asynchronousness

It should be possible to make all promise-related operations forcibly
asynchronous. Although such an option may affect performance, it may
help fight unexpected control flow aberrations resulting from some
code returning promises under some circumstances and direct values
under some other circumstances. Such aberrations may be hard to detect
during testing.

This mode of operation should be used when *idle-call-function*
is non-null (a possible function to be placed in this variable can be
implemented using recently added 'idlers' from cl-async). In this
mode, callbacks and errbacks should only be invoked when the control
reaches application event loop (not necessarily cl-async event loop,
similar things can be done for iolib, CommonQt event loop and so on).

Also, there should be an easy way to convert a plain value to a
promise like Q(value) does it, e.g. a macro called (promisify ...)
which is an easier way to write

(attach
 (with-promise (accept) (accept t))
 #'(lambda (v)
     (declare (ignore v))
     ...))

which provides delayed (on-idle) execution and promise-based error
handling.

5. Error handling

I think promise-handler-case semantics should be cleaned up a bit.
Namely, I think that

(promise-handler-case (something)
  (some-error (e)
    (format t "some error: ~a~%" e))
  (another-error (e)
    (format t "another error: ~a~%" e)))

should behave like the following code which assumes the new
attach-errback behavior described below:

(attach-errback
 (bb:promisify (something))
 #'(lambda (e)
     (typecase e
       (some-error
        (format t "some error: ~a~%" e))
       (another-error
        (format t "another error: ~a~%" e))
       (t
        (error e)))))

I understand that this substantially differs from what
promise-handler-case is currently doing but as of now IMO it provides
rather easy way to shoot oneself in the foot. Maybe the new version of
error handling form should have some different name. Lexically scoped
form of error handling may be useful from performance POV but frankly
it's not overly intuitive.

Another useful construct would be something like promise-protect
which works like an asynchronous version of unwind-protect or like
Q.finally().

6. Misc

Some useful funcs can be borrowed from Q such as Q.resolve(),
Q.reject(), Q.all().

Should this fire the catcher error?

CL-USER> (as:with-event-loop (:catch-app-errors t)                                                                      
           (bb:catcher                                                                                                  
            (bb:alet ((result (carrier:request "https://www.apple.com/" :return-body t)))                               
              (error "dead")                                                                                            
              (format t "result: ~s" result))                                                                           
            (t (e) (format t "error: ~a" e))))                                                                          
0                                                                                                                       
CL-USER>

I would have expected: "error: dead", I think that's how cl-async-future did it I think?

Take this example:

(chain 4
  (:then (x) (+ x 'test))
  (:then (x) (save-to-db x))

The promise chain is broken but the error isn't caught (unless attach is called on the chain call). If we can use trivial-garbage to detect these dangling chains and run some sort of callback, it would aid in the debugging of improperly handled chains.

super easy, but dont want to do it this sec and dont want to forget

http://report.quicklisp.org/2014-12-10/failure-report/green-threads.html

I'm getting this:

Unhandled UNDEFINED-FUNCTION in thread #<SB-THREAD:THREAD "main thread" RUNNING {1004477613}>: The function BLACKBIRD::PROMISE-VALUES is undefined.

Is that a problem with the library? Or is it something to do with blackbird?

Hi,

I am trying to use the cl-async-repl library in combination with the drakma-async library. One issue I've ran into is that it seems impossible to return the result of a promise as the result of the repl. What I am looking for is a function (which I am going to call force) which takes a promise and blocks until the promise is finished, and then returns the value inside of the promise. As an example, when used within cl-async-repl

(force (drakma-async:http-request ...))

should act like ordinary drakma:http-request.

Does such a function already exist? If not, is there an easy way to emulate its behavior? The closest thing I've come up with is to add a callback to the promise that will print out the value, but I am looking for something that actually returns the value.

If you do:

(alet ((result (amap #'do-stuff my-list)))
  (format t "~&done~&"))

"done" will only print if my-list has content, if its nil it will never print anything or finish the future.

doesn't unwind the stack when handling errors

finally now forwards the value(s)/error of the promise it gets, and essentially throws out its return values.

http://report.quicklisp.org/2015-01-07/failure-report/blackbird.html has the log. I get this:

; file: /home/quicklisp/quicklisp-controller/dist/build-cache/blackbird/3bebe4cd4434384ba02ffd4bd0ba9ebb0a3fda70/blackbird-20150106-git/promise.lisp
; in: DEFUN DO-CATCH
;     (RETURN-FROM BLACKBIRD-BASE:CATCHER
;       (BLACKBIRD-BASE::REJECT BLACKBIRD-BASE::E))
; 
; caught ERROR:
;   return for unknown block: CATCHER

It fails as:

:info:build ; Compilation unit finished
:info:build ;   Caught 5 WARNING conditions
:info:build ;   Caught 2 STYLE-WARNING conditions
:info:build ;   The following functions were used but not defined:
:info:build ;     JSS::READ-SHARP-JAVA-EXPRESSION
:info:build Caught ASDF/FIND-SYSTEM:LOAD-SYSTEM-DEFINITION-ERROR while processing --eval option "(asdf:operate (quote asdf:build-op) (quote blackbird-test))":
:info:build   #<LOAD-SYSTEM-DEFINITION-ERROR {43E86854}>
:info:build Command failed: env XDG_CACHE_HOME=$HOME/.cache /opt/local/bin/abcl --noinit --batch --eval '(require "asdf")' --eval '(setf asdf:*central-registry* (list* (quote *default-pathname-defaults*) #p"/opt/local/var/macports/build/_Users_catap_src_macports-ports_lisp_cl-blackbird/cl-blackbird/work/build/system/" #p"/opt/local/share/common-lisp/system/" asdf:*central-registry*))' --eval '(asdf:operate (quote asdf:build-op) (quote blackbird-test))' 2>&1

SBCL, ECL, CLisp and CCL works.