zunit-zsh / zunit Goto Github PK

It's sometimes very difficult to figure out why something has failed from within zunit because there is (that I can tell) no way of displaying the text written to stdout and stderr when running commands.

E.g. if I have assert $state equals 0 and it fails, it would be nice to see why.

This is especially important if the test is failing on a test CI such as travis but not locally.

The assertion library is rather robust, and growing, but it would be handy to provide access to pass, fail or error a test manually and end its execution, like we already can with the skip helper.

Example:

@test 'test some-other-command' {
  if some-other-command; then
    pass
  do
  
  [[ $? -eq 127 ]] && error 'some-other-command not found'
  
  fail 'some-other-command failed'
}

The CI checks currently fail with error:

✘ Test _zunit_assert_contains match

  'Assertion world does not exist' is not empty

The test that it refers is:

@test 'Test _zunit_assert_contains match' {
  run assert 'hello world' contains 'lo wo'

  assert "$output" is_empty
  assert $state equals 0
}

It seems that somehow the second word of the string "hello world" is taken for a name of the assertion to run. What could be causing this?

At the moment if a test is skipped, the overall run returns a failed exit status as the total count does not match the passed count. This should be changed to compare against passed + skipped so that the overall run results in success.

Need some kind of reporting system, and options to specify which ones to use.

TAP is probably good enough as a starting point. (TAP generation added in #10, just need to redirect it to a file).

HTML would be nice-to-have.

Affects ZSH 5.0. From ZUnit's own tests:

✘ Test _zunit_assert_in failure
    ''a' is not in (x)' is not the same as ''a' is not in (x ; y ; z)'

This was in the README prior to v0.6.0, but due to the new compilation step on install the method that was in there would not work, so I removed it. If anyone uses zgen and knows how to install ZUnit in a way that would get it working please feel free to submit a PR to add it back in.

Hello,
I'm submitting a Zsh gitignore template to github and to toptal. I'm wondering if any of the projects like zunit, revolver, zulu, etc. creates:

• any files in a projects directory,
• or any files in the directory in which it's being run?

Such files could be then added to the template.

Hello,
I did a quick glance at zunit's source and saw many sed and other external programs invocation. An example found by ack:

src/commands/run.zsh
221:# Encode test name into a value which can be used as a hash key
228:            | sed 's/\- /-/' \
229:            | sed 's/ \-/-/' \

The project might already be very important, I often think about using it after positive experience with Bash BATS, however such coding style is IMO a problem, I wouldn't introduce forking-style tool to my projects. BATS probably does the same, but on Bash it is different, there rather is no choice except forking for certain features. On Zsh, one can code like in Ruby and other "real" programming languages. I've made a handbook explaining the no-fork way: Zsh Native Scripting Handbook.

The project seems to be inactive – I see it has been implemented, works, so no new commits appear. I'm thinking in engageing into zunit, but would you accept no-fork-replacements versions of existing code in pull requests?

There are many instances where all tests within a project need the same setup code (e.g. autoloading modules etc.). Including this same code in the @setup method for each file would be tedious. It would be far easier to have a project-level bootstrap script which could handle setup for the whole suite.

• Check for a project-level bootstrap script (Probably in tests/_support/bootstrap) prior to tests being loaded, and execute it.
• Create empty bootstrap script during init.

For some reason, even though test files are parsed line-by-line, the tests in each file don't always run in the correct order. They should.

The project seems to be interesting and enough-invested (>190 commits). But README doesn't provide examples. Can some examples be added?

BTW, maybe you will be interested – there's a tool that parses scripts using Zshell's (z) flag and extracts functions. I use it in one project – no need to load whole crasis script, instead the extracted functions are autoloaded. The extraction is zsd-transform, autoloads are in runtest.zsh: https://github.com/zdharma/zplugin-crasis/tree/master/test/bin

Currently, zunit init -[-t]ravis fails if tests and/or .zunit.yml already exist…
It should probably just always generate the .travis.yml unless [only] it already exists.

If something fails in @setup or @teardown, its often hard to debug what went wrong even with the new --verbose option. The work around I am currently using is to move the failing code to a test so that I can see what the failing message is.

At the moment assertion failures, skipped tests etc. are all handled using a range of exit codes. It would be nice to switch this over to use crash to handle these different states, and then each can be given a custom handler.

The basics are included, but the more the merrier. Any ideas for new assertion methods are welcome.

Hello!

I was hoping you could consider adding installation methods 🙂

Currently, the only available installation methods are Zulu, homebrew and manual installation.

While I like the prompt Zulu set for me, I'm not a fan of programs that change my shell environment without asking. Using homebrew on Linux, even if possible, does not seem so straight-forward.

I know nothing about these, but what about Zplug, Zgen or Antigen? Do they fill the same purpose as Zulu? Would they be good candidate as installation methods for ZUnit?

Also, coming from Bash, I started to use Basher for other projects, and like its cleanness and simplicity (it git clones shell projects from providers like github, gitlab, etc.). Basher supports Zsh (and Fish) as well, and I would love to be able to install ZUnit through it. The only problem I see is that ZUnit's installation seems to be in 2 steps: build, then install, which Basher would not support (it only clones stuff, and doesn't execute any custom code). Is the build step mandatory? Couldn't you directly provide the zunit executable script in the repo?

Installing the latest version of zunit seems to be producing a dependency error:

$ zunit       
Missing required dependency: Revolver - https://github.com/molovo/revolver

However this is installed:

$ zplug list | grep -i revolver
molovo/revolver

We have CLI options for these (--fail-fast and --allow-risky), but it would be nice if they could be defined in .zunit.yml as well.

Currently, every file in the tests directory is treated as a zunit test file, and an error is thrown if the file does not start with the zunit shebang:

Is there a reason for not filtering out files that lack the .zunit extension? I have experienced a number of situations where zunit fails due to trying to read a macOS .DS_Store or vim .swp file as a test file.

We can specify the time_limit key in .zunit.yml, but it would be handy to be able to override this by passing a --time-limit CLI option.

When tests are passed through eval they are parsed twice, once to catch and output errors, and then again the load the function into the current scope. We should do the same with @setup and @teardown methods to prevent fatal errors.

ZUnit originally worked without the color and revolver dependencies, however as it has moved on they have become required.

We should throw an error message when the user runs ZUnit without these dependencies, and point them towards the correct repositories to go an install them.

Simplest way to explain this is with a test example:

@test 'This test will pass no matter what' {
  assert "$output" contains "hello"
  assert $state equals 0
}

This also works with any other empty variable being tested - but output is usually the most likely case.

This is likely to do with the way assert has been written so that the assert operator is after the first operand. So zunit treats the the assert as assert contains "hello" since "$output" is empty.

Could use syntax like:

zunit /path/to/file@"The name of a test"

Would load that file, but only parse and load that particular test. Particularly useful when writing slow running tests to avoid running the whole file.

Could possibly use zprof to work out which lines are covered during the course of running a test, but this may be tricky. Something to look into.

Will probably rely on elements of #8 for reporting coverage data.

At the moment, execution halts and results are not printed to screen. They should be.

Depends on #24

Having this option would be very useful for being able to integrate nicely with CI tools like CircleCI :)

There is also no option to disable it since the opposite is meant to be the default behaviour

Tests should be independent from what's contained within the users zsh configuration as it can interfere with the test process. In short, tests should not load the user's zshrc profile when running tests.

(PS: Only just realised I had an invitation pending for me to join this project, sorry about that!)

When set, will stop the execution of tests at the first sign of failure.

It would be nice to be able to simply set the .zunit.yml config options with environment variables, and avoid having to manage another file when it only contains a few lines… options like ZUNIT_DIR_TESTS and ZUNIT_DIR_ SUPPORT for example.

I made an @setup test but some of the things never get parsed...

@setup {
  zinc_left=( zincs_userhost )

  prompt_zinc_setup
  echo $zinc_left
  this is something that gets ignored by zunit but shouldnt!
  literally anything past that prompt_zinc_setup gets ignored
}

It goes on to the tests later on in the file and they all fail since it's as if prompt_zinc_setup never got executed, and as the example says, the echo doesn't get parsed or executed either.

Trying to use Zunit here: https://github.com/robobenklein/zinc/blob/261cf980b813ea90424e49a3b5cba554072df42f/tests/zinc-base.zsh

In ZSH 5.0 the $path variable is already typecast as an array, and assigning a string to it throws the error path: can't assign initial value for array.

Renaming the variable should fix the issue.

I've registered the zunit-zsh Github organisation, and will eventually move this repository there in the hope that it will encourage community contributions. As part of that, I'll be building a website for ZUnit and fleshing out the documentation a bit.

When this is done, it would be nice to have a simple logo and some branding ideas to build into the website. Any ideas are welcome.

If we integrate crash as described in #50, we could allow users to mark tests as expecting a certain exception or exit code, and mark the test as passed if the exception occurs.

Syntax:

@test 'My awesome test' {
  @expect AnException
  
  # Test code here...
}

At the moment if --fail-fast is enabled the script exits immediately. We should call a proper shutdown handler here which can kill the revolver process rather than this being repeated throughout the program, and print the remaining HTML output (as the report stops being written when the exit is triggered).

Currently the automated deployment uploads 6(!) times, because it deploys at the end of each test run. Now that Travis have released support for build stages, we should put the deploy step in it's own build stage so it only runs once, and after all tests have passed for all ZSH versions.

It seems as though when a line in @setup or @teardown returns with a non-zero exit code, all tests will fail. This seems like it might be a bug, but even if it isn't, it would be helpful to get more diagnostic information from zunit when this is the case.

By way of example, here's a zunit script:

#!/usr/bin/env zunit

@teardown {
  # When this line is used, the test fails:
  [ -f tests/_support/out.dmg ] && rm tests/_support/out.dmg;

  # When this line is used, everything works:
  #if [ -f tests/_support/out.dmg ]; then rm tests/_support/out.dmg; fi
}

@test 'Test that should pass' {
  assert 1 equals 1
}

When run with zunit --verbose, you just get a fail on "Test that should pass".

To be fair, I wasn't aware that the short-circuit evaluation would return a non-zero exit code when the condition is false, but this definitely had me scratching my head for a good while since nothing in zunit's verbose output was pointing me in the direction of the problem.

If you are trying to debug a single test case amongst a bunch of other tests, this is extremely useful

Mod the init command to generate a .travis.yml file when initialising a project. Will probably ask the user whether they want this first, and add a CLI option which answers the question in case it is used in a script.

Hopefully this isnt another non-issue, but using run doesnt seem to maintain it's environment between subsequent calls in the same test.

e.g.

@test 'something' {
    echo "hello" > "foo/bar"

    run cd foo
    run cat bar
}

the second run will fail because the cd to foo is not maintained

If I set the var in the .yml file it causes all of my tests to pass immediately (when they shouldn't) and when I set it on the cli it runs as if no time limit was set. I know it's not working as expected because one of my tests goes into an infinite loop and the zunit process never times out, and must be manually killed.

Here is my use-case:

I wrote a shell script to compute code coverage. It uses the xtrace/debug mode of Zsh/Bash to know which lines of which files have been executed.

Initially I wrote it for Bash only, but Bash's LINENO variable is... not always useful. Zsh's LINENO is much better, so I added support for Zsh.

Now, this script will generally be used to run test suites. Typically, with Bats for Bash and ZUnit for Zsh. It works wonderfully with Bats (except for the LINENO detail), because I can use Bash's BASH_XTRACEFD variable which redirect the xtrace output to another file descriptor than stderr.

Unfortunately, such a variable does not exist in Zsh: the xtrace output is always sent to stderr. This is a problem because the xtrace otuput is then captured by Zunit (in run statements), and therefore made unavailable to my coverage script.

This is where my feature request comes to play: while waiting for Zsh to implement such a ZSH_XTRACEFD variable, or similar (I sent a mail to the zsh-workers mailing list asking for it), I would like Zunit to provide an option to copy the stderr of a run command to another file or file descriptor of my choice. Some kind of tee on stderr, something like 2> >(tee >&1 "$zunit_copy_stderr_file").

This example (which is not tested and maybe not working) is explained as:

• send stderr to a new file descriptor
• which uses tee to send it back
  • to stdout: this is what Zunit currently does:

    zunit/src/commands/run.zsh

    Line 185 in 7677348

    output="$(__zunit_tmp_test_function 2>&1)"

    or the async version:

    zunit/src/commands/run.zsh

    Line 182 in 7677348

    output="$(__zunit_async_test_wrapper 2>&1)"

  • and to a file whose path is contained in the zunit_copy_stderr_file variable.

This feature would allow me to copy the stderr, and therefore xtrace output of Zsh into a file of my choice to further process it later.

Would you be willing to integrate such an option in Zunit? If yes, I can try and open a PR in a few days.

P.S.: I saw earlier that you have a "coverage" branch, but that it was not updated since a relatively long time. It would be great if Zunit had integrated code coverage, but I think an independent script is still a good idea, not being tied to a particular testing framework. This is why I'm opening this ticket 🙂

There are occasions when tests can get stuck in an infinite loop. We should allow for a time limit for tests to be specified.

When the time limit is reached, the subprocess for the test will be killed, and the test marked as errored.

We can also specify an overall time limit for all tests, and cease execution when it is reached.

The only thing that currently generates warnings are risky tests (without any assertions) but when these occur, the test still counts as passed. We should make warnings represent their own state, and not increment the passed count, otherwise the --allow-risky option provides no benefit.