ag91 / doctest Goto Github PK

To reproduce, add this test to a file a few times:

>> (sleep-for 1.0)
=> t

and run it (noninteractively). A user would expect that approximately every second the framework would print information about the next failure (sleep-for returns nil, so all these tests will fail). Instead, after several seconds the framework prints all the information at once. Interactive behavior appears to be similar.

While probably not important in most cases, it can be annoying if you have a lot of (moderately) slow tests, as then you have no idea about the progress.

Also, as far as progress notification is concerned, it would be nice for the framework to optionally print something for passing tests too. E.g. there could be a variable doctest-verbose or something like that to control the behavior.

Currently (this is still unpublished work) when I run testing from Eldev on a project with two files I get this:

1 passed, 0 failed
1 passed, 0 failed

This means that test in each of the two files are run separately, with counts being reset after each file and summary being printed. This is problematic for two reasons: 1) this is unlike every other framework I know; 2) the summary doesn't even mention which file was executed, so splitting it is kind of pointless, since you don't know which "subsummary" corresponds to which file anyway.

I'm not opening a PR here, because I can see several different ways of solving this, but neither is a clear winner, so I want to discuss first.

Currently the "highest level" entry point is function doctest. However, by design it runs on a single buffer/file, resets counters and prints summary itself. At least for backward compatibility I propose to keep this behavior.

However, there should be a way to mark a testsuite that consists of multiple buffers somehow.

Some possibilities:

• Have another function doctest-<NAME-MISSING> that accepts multiple files and considers them a single suite. Existing doctest would then just call this function with a single file.

• Have a global variable, e.g. sth. like doctest-inside-suite, initially nil. If bound to non-nil, function doctest doesn't reset the counters and doesn't print the summary, i.e. considers itself a part of ongoing testsuite. However, we then also need to make public the counter-resetting and summary-printing functions public. Usage would look something like this:

  (let ((doctest-inside-suite t))
    (doctest-reset-state)
    (dolist (file all-files)
      (doctest file))
    (doctest-print-summary))
  

• Optionally extending the above: add a function like e.g. doctest-composite-suite that "hides" as internals first and last steps (advantages: no need to make a lot of things public, they can stay package-private). Usage would look like:

  (doctest-composite-suite (lambda ()
                             (dolist (file all-files)
                               (doctest file)))
  

Or would you prefer some other way? I can create a PR, just pick some approach.

To reproduce, for example, edit doctest.el and replace ">> (stringp doctest-input)" with ">> (stringp doctest-input", i.e. delete the closing paren. Then run doctest on the file:

$ emacs --batch -L . --eval "(progn (require 'doctest) (doctest \"doctest.el\"))"
Warning (emacs): Test cases in comments can cause byte-compile warnings, please add ’(not docstrings) to ‘byte-compile-warnings’

Error: end-of-file nil
  mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode 0x1ce8b97441ae86be>))
  [...]
  normal-top-level()
End of file during parsing

In comparison, when the test is correct:

$ emacs --batch -L . --eval "(progn (require 'doctest) (doctest \"doctest.el\"))"
Warning (emacs): Test cases in comments can cause byte-compile warnings, please add ’(not docstrings) to ‘byte-compile-warnings’
7 passed, 0 failed

The problem can be seen not only if a test expression is malformed, but also if it somehow signals anything, e.g. if there is a test like this:

>> (/ 1 0)
=> 2

test run dies in this confusing stacktrace.

In comparison, if the test is

>> (/ 1 1)
=> 2

it fails "in a normal way", with the framework reporting the problem. But if any signal happens, doctest run is aborted (further tests are not executed) and there is no indication which test caused the problem. In fact, there is no good way to understand that the problem is in a test, not the framework itself, unless you know the internals well enough.

All other frameworks I know of (e.g. Buttercup, ERT, but also those not for Elisp, e.g. Java's JUnit) treat exceptions/signals in tests as test errors and report them unambigously, so that you can see that the exception happened during a test (and which test). They also continue execution normally, so that e.g. you can eventually see the summary that mentions at least one test failing.