joaotavora / fiasco Goto Github PK

View Code? Open in Web Editor NEW

55.0 55.0 11.0 322 KB

A test framework for Common Lisp

License: Other

Common Lisp 100.00%

fiasco's People

Contributors

Stargazers

Watchers

Forkers

puercopop fstamour luismbo stjordanis notmgsk tarnhelm hineios rudolfochrist ashok-khanna appleby jbouwman

fiasco's Issues

Document return value of `RUN-PACKAGE-TESTS`

The return value is useful and meaningful. Document it in the docstring.

feature: tailor/advertise test assert macros as a replacement for cl:assert

copy-pasting an older feature idea here:

maybe it would be worth a few thoughts to consider supporting/encouraging the usage of stefil's asserts in the project code itself (as opposed to only in its unit tests).

e.g. when the asserts in the project code are run without a dynamic test environment, then stefil asserts can behave differently in case of a failed assertion. currently they print stuff to stderr and signal an error, which is not too bad, but to be honest i've never considered thoroughly this use-case, nor have i used stefil asserts outside of the unit tests.

the current behavior was implemented with REPL usage in mind, mostly to enable copy-pasting parts of the test code to the REPL without any hassle.

Quicklisp integration

What should be done for this package to be included into quicklisp? If needed, I will PR required changes myself. October quicklisp release passed without fiasco in it.

DEFINE-TEST-PACKAGE should export RUN-PACKAGE-TESTS

As it is one of the entry point to call the tests or it meant only for interactive testing? I am there are other entry points such us being fiasco:run-suite-tests

Lack of fixture functionality

In the original stefil, suites also did double duty as fixtures, routines to set up and tear down the global environment. This was done by defining the suite with a body in which run-child-tests was called. The body argument was omitted on the implementation rewrite, but with no replacement functionality.

There's a number of options to support setup and teardown for tests, and I don't know which one is right for fiasco. It's tough having none of them, though. I'd rather not wrap deftest in my own macro.

Remove playful language

Sorry if I sound like a buzzkill with this issue, but can we remove the playful language (e.g., Fiasco!) and just keep it to informative, boring output?

The reason I'm asking is because this is being considered to be used within a professional environment. Nothing is "wrong" or "politically incorrect" with the language, but it just (in the eyes of some) makes it seem less serious.

Provide mechanism to skip a test

It would be good if Fiasco had a way that the test opted to skip itself. Example:

(fiasco:deftest test-socket ()
  (when os-is-windows
    (fiasco:skip-test))
  ...)

Prefer not to use #-windows as I think that's bad practice.

CHECK-REQUIRED is undefined + other warnings

; file: /fiasco-20150923-git/src/infrastructure.lisp
; in: DEFMACRO CHECK-REQUIRED
;     (DEFMACRO FIASCO::CHECK-REQUIRED (FIASCO::SYM)
;       `(ERROR "Must provide ~a" ,FIASCO::SYM))
; --> PROGN EVAL-WHEN 
; ==>
;   (SB-C::%DEFMACRO 'FIASCO::CHECK-REQUIRED
;                    (SB-INT:NAMED-LAMBDA (MACRO-FUNCTION FIASCO::CHECK-REQUIRED)
;                        (#:EXPR #:ENV)
;                      (DECLARE (SB-C::LAMBDA-LIST (FIASCO::SYM)))
;                      (DECLARE (IGNORE #:ENV))
;                      (SB-INT:NAMED-DS-BIND (:MACRO FIASCO::CHECK-REQUIRED
;                                             . DEFMACRO)
;                          (FIASCO::SYM)
;                          (CDR #:EXPR)
;                        (BLOCK FIASCO::CHECK-REQUIRED
;                          `(ERROR "Must provide ~a" ,FIASCO::SYM))))
;                    (SB-C:SOURCE-LOCATION))
; 
; caught STYLE-WARNING:
;   CHECK-REQUIRED is being redefined as a macro when it was previously assumed to be a function.

; compilation finished in 0:00:00.334
WARNING:
   Lisp compilation had style-warnings while
   compiling #<CL-SOURCE-FILE "fiasco" "src" "infrastructure">
WARNING: Implicitly creating new generic function FIASCO::FAILURES-OF.
WARNING: Implicitly creating new generic function FIASCO::ASSERTIONS-OF.

fiasco-20150923-git/src/infrastructure.lisp
; in: DEFCLASS FIASCO::CONTEXT
;     (FIASCO::CHECK-REQUIRED 'FIASCO::ACTUAL-TEST-ARGUMENTS)
; 
; caught STYLE-WARNING:
;   undefined function: FIASCO::CHECK-REQUIRED
; 
; compilation unit finished
;   Undefined function:
;     FIASCO::CHECK-REQUIRED
;   caught 2 STYLE-WARNING conditions

Can't format custom message for failed-assertion form

Hi there, I am executing some test in fiasco and it seems to fail properly.

The problem is that I get something like:

Can't format custom message for failed-assertion form (is
                                                       (equal
                                                        (list
                                                         '(foo
                                                           bar))
                                                        (compute-graph
                                                         '(foo/bar))))

It is a bit hard for me to understand if I have a problem in my setup (being a super newbie) or it is a problem in the library.

Thank you for any feedback/answer and for maintaining this library!

Broken on CCL

Trying to run the README example with (or any fiasco test suite) with CCL results in the following error:

> Error: There is no next method for #<STANDARD-METHOD CCL:STREAM-LINE-COLUMN (FIASCO::COLUMN-COUNTING-OUTPUT-STREAM)>
>        args: (#<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D>)
> While executing: #<CCL::STANDARD-KERNEL-METHOD NO-NEXT-METHOD (STANDARD-GENERIC-FUNCTION STANDARD-METHOD)>, in process listener(1).

The full backtrace

1 > B
 (15193D0) : 0 (FUNCALL #'#<#<CCL::STANDARD-KERNEL-METHOD NO-NEXT-METHOD (STANDARD-GENERIC-FUNCTION STANDARD-METHOD)>> #<STANDARD-GENERIC-FUNCTION CCL:STREAM-LINE-COLUMN #x30200006906F> #<STANDARD-METHOD CCL:STREAM-LINE-COLUMN (FIASCO::COLUMN-COUNTING-OUTPUT-STREAM)> #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D>) 101
 (15193F8) : 1 (%CALL-NEXT-METHOD (NIL #<STANDARD-METHOD CCL:STREAM-LINE-COLUMN #> . 2765461)) 253
 (1519428) : 2 (FUNCALL #'#<#<STANDARD-METHOD CCL:STREAM-LINE-COLUMN (FIASCO::COLUMN-COUNTING-OUTPUT-STREAM)>> #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D>) 61
 (1519450) : 3 (%%STANDARD-COMBINED-METHOD-DCODE (#<STANDARD-METHOD CCL:STREAM-LINE-COLUMN #>) 2765461) 701
 (15194B8) : 4 (INITIALIZE-XP #<CCL::XP-STRUCTURE #x3020065077DD> #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D>) 381
 (15194E0) : 5 (GET-PRETTY-PRINT-STREAM #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D>) 221
 (1519500) : 6 (DO-XP-PRINTING #<Compiled-function (:INTERNAL FORMAT) (Non-Global)  #x300000A091CF> #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D> ((0 #\ ))) 45
 (1519528) : 7 (FUNCALL #'#<(:INTERNAL CCL::DO-IT CCL::XP-PRINT)> #<Compiled-function (:INTERNAL FORMAT) (Non-Global)  #x300000A091CF> #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D> ((0 #\ ))) 53
 (1519550) : 8 (MAYBE-INITIATE-XP-PRINTING #<Compiled-function (:INTERNAL FORMAT) (Non-Global)  #x300000A091CF> #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D> (0 #\ )) 533
 (15195B8) : 9 (FORMAT #<FIASCO::COLUMN-COUNTING-OUTPUT-STREAM #x30200669BD6D> "~&~v@{~C~:*~}" 0 #\ ) 1781
 (1519648) : 10 (FUNCALL #'#<(:INTERNAL FIASCO::PP FIASCO::PRETTY-RUN-TEST)> "~A (Suite)" FIASCO-SUITES::FIASCO-EXAMPLES) 213
 (1519668) : 11 (PRETTY-RUN-TEST #<test FIASCO-SUITES::FIASCO-EXAMPLES :tests 3> #<Compiled-function (:INTERNAL FIASCO-SUITES::FIASCO-EXAMPLES FIASCO-SUITES::FIASCO-EXAMPLES) (Non-Global)  #x3020065567DF>) 205
 (1519688) : 12 (FUNCALL #'#<(:INTERNAL #:BODY-SYM77146 FIASCO-SUITES::FIASCO-EXAMPLES)>) 453
 (15196D8) : 13 (FIASCO-EXAMPLES) 1885
 (1519848) : 14 (RUN-SUITE-TESTS #<test FIASCO-SUITES::FIASCO-EXAMPLES :tests 3> :VERBOSE NIL :STREAM #<SYNONYM-STREAM to *TERMINAL-IO* #x3020003C57AD> :INTERACTIVE NIL) 637
 (1519978) : 15 (RUN-TESTS #<Package "FIASCO-EXAMPLES"> :DESCRIBE-FAILURES T :VERBOSE NIL :STREAM #<SYNONYM-STREAM to *TERMINAL-IO* #x3020003C57AD> :INTERACTIVE NIL) 805
 (1519A08) : 16 (RUN-PACKAGE-TESTS :PACKAGE #<Package "FIASCO-EXAMPLES"> :PACKAGES (#<Package "FIASCO-EXAMPLES">) :DESCRIBE-FAILURES T :VERBOSE NIL :STREAM #<SYNONYM-STREAM to *TERMINAL-IO* #x3020003C57AD> :INTERACTIVE NIL) 389
 (1519A80) : 17 (CALL-CHECK-REGS RUN-PACKAGE-TESTS) 229
 (1519AB8) : 18 (TOPLEVEL-EVAL (RUN-PACKAGE-TESTS) NIL) 789
 (1519B30) : 19 (READ-LOOP :INPUT-STREAM #<SYNONYM-STREAM to *TERMINAL-IO* #x3020003C590D> :OUTPUT-STREAM #<SYNONYM-STREAM to *TERMINAL-IO* #x3020003C57AD> :BREAK-LEVEL 0 :PROMPT-FUNCTION #<Compiled-function (:INTERNAL CCL::READ-LOOP) (Non-Global)  #x30000058862F>) 2341
 (1519D78) : 20 (RUN-READ-LOOP :BREAK-LEVEL 0) 157
 (1519DA0) : 21 (TOPLEVEL-LOOP) 93
 (1519DB0) : 22 (FUNCALL #'#<(:INTERNAL (CCL:TOPLEVEL-FUNCTION (CCL::LISP-DEVELOPMENT-SYSTEM T)))>) 109
 (1519DD0) : 23 (FUNCALL #'#<(:INTERNAL CCL::MAKE-MCL-LISTENER-PROCESS)>) 661
 (1519E68) : 24 (RUN-PROCESS-INITIAL-FORM #<TTY-LISTENER listener(1) [Active] #x3020003C470D> (#<CCL:COMPILED-LEXICAL-CLOSURE # #x3020003C423F>)) 669
 (1519EF0) : 25 (FUNCALL #'#<(:INTERNAL (CCL::%PROCESS-PRESET-INTERNAL (CCL:PROCESS)))> #<TTY-LISTENER listener(1) [Active] #x3020003C470D> (#<CCL:COMPILED-LEXICAL-CLOSURE # #x3020003C423F>)) 581
 (1519F98) : 26 (FUNCALL #'#<(:INTERNAL CCL::THREAD-MAKE-STARTUP-FUNCTION)>) 277

Testing in a threads

I have a test structured as follows:

start a server
run client in a separate thread (bt:make-thread...)
do fiasco tests in the (client) thread
before terminating the client thread, client lets server know it should close.

Unfortunately, at least in sbcl, the tests in the client/other thread are not "counted" in reports - (0 assertions, 0 failed, 0 errors, 0 expected). Instead, I get few "WARNING: Condition FIASCO::IS-ASSERTION was signalled." warnings.

I guess that this might be due to conditions based implementation, and sbcl not sharing restarts/handlers/whatever with the child threads.

I am not sure if I do something wrong, if it can be easily fixed or if it is one of "do not do it" item - any suggestion?

Are the TravisCI test run exit codes correct?

So maybe I don't understand the test suite, but it seems like the tests are failing despite being green: https://travis-ci.org/github/joaotavora/fiasco

But since the exit code is 0 TravisCI doesn't know any better.

infinite recursion in recent synonym-stream introduction

Commit fe89c0e introduced unstable behavior in SBCL (and probably all Lisp implementations). Running (some?) tests cause infinite recursion.

The reduced test case is:

CL-USER> (let ((*standard-output* fiasco:*test-run-standard-output*))
           (format t "x"))
Control stack guard page temporarily disabled: proceed with caution

which more or less appears in the deftest macro. This is equivalent to

CL-USER> (let ((*standard-output* (make-synonym-stream '*standard-output*)))
           (format t "x"))

I think the infinite recursion makes sense because the symbol is looked up dynamically.