zunit-zsh / zunit Goto Github PK
View Code? Open in Web Editor NEWA powerful testing framework for ZSH projects
Home Page: https://zunit.xyz
License: MIT License
A powerful testing framework for ZSH projects
Home Page: https://zunit.xyz
License: MIT License
Could possibly use zprof to work out which lines are covered during the course of running a test, but this may be tricky. Something to look into.
Will probably rely on elements of #8 for reporting coverage data.
At the moment assertion failures, skipped tests etc. are all handled using a range of exit codes. It would be nice to switch this over to use crash to handle these different states, and then each can be given a custom handler.
Hopefully this isnt another non-issue, but using run
doesnt seem to maintain it's environment between subsequent calls in the same test.
e.g.
@test 'something' {
echo "hello" > "foo/bar"
run cd foo
run cat bar
}
the second run will fail because the cd to foo is not maintained
It would be nice to be able to simply set the .zunit.yml
config options with environment variables, and avoid having to manage another file when it only contains a few lines⦠options like ZUNIT_DIR_TESTS
and ZUNIT_DIR_ SUPPORT
for example.
If you are trying to debug a single test case amongst a bunch of other tests, this is extremely useful
Installing the latest version of zunit seems to be producing a dependency error:
$ zunit
Missing required dependency: Revolver - https://github.com/molovo/revolver
However this is installed:
$ zplug list | grep -i revolver
molovo/revolver
Hello!
I was hoping you could consider adding installation methods π
Currently, the only available installation methods are Zulu, homebrew and manual installation.
While I like the prompt Zulu set for me, I'm not a fan of programs that change my shell environment without asking. Using homebrew on Linux, even if possible, does not seem so straight-forward.
I know nothing about these, but what about Zplug, Zgen or Antigen? Do they fill the same purpose as Zulu? Would they be good candidate as installation methods for ZUnit?
Also, coming from Bash, I started to use Basher for other projects, and like its cleanness and simplicity (it git clones shell projects from providers like github, gitlab, etc.). Basher supports Zsh (and Fish) as well, and I would love to be able to install ZUnit through it. The only problem I see is that ZUnit's installation seems to be in 2 steps: build, then install, which Basher would not support (it only clones stuff, and doesn't execute any custom code). Is the build step mandatory? Couldn't you directly provide the zunit
executable script in the repo?
I've registered the zunit-zsh Github organisation, and will eventually move this repository there in the hope that it will encourage community contributions. As part of that, I'll be building a website for ZUnit and fleshing out the documentation a bit.
When this is done, it would be nice to have a simple logo and some branding ideas to build into the website. Any ideas are welcome.
Here is my use-case:
I wrote a shell script to compute code coverage. It uses the xtrace/debug mode of Zsh/Bash to know which lines of which files have been executed.
Initially I wrote it for Bash only, but Bash's LINENO
variable is... not always useful. Zsh's LINENO
is much better, so I added support for Zsh.
Now, this script will generally be used to run test suites. Typically, with Bats for Bash and ZUnit for Zsh. It works wonderfully with Bats (except for the LINENO
detail), because I can use Bash's BASH_XTRACEFD
variable which redirect the xtrace output to another file descriptor than stderr.
Unfortunately, such a variable does not exist in Zsh: the xtrace output is always sent to stderr. This is a problem because the xtrace otuput is then captured by Zunit (in run
statements), and therefore made unavailable to my coverage script.
This is where my feature request comes to play: while waiting for Zsh to implement such a ZSH_XTRACEFD
variable, or similar (I sent a mail to the zsh-workers mailing list asking for it), I would like Zunit to provide an option to copy the stderr of a run
command to another file or file descriptor of my choice. Some kind of tee
on stderr, something like 2> >(tee >&1 "$zunit_copy_stderr_file")
.
This example (which is not tested and maybe not working) is explained as:
This feature would allow me to copy the stderr, and therefore xtrace output of Zsh into a file of my choice to further process it later.
Would you be willing to integrate such an option in Zunit? If yes, I can try and open a PR in a few days.
P.S.: I saw earlier that you have a "coverage" branch, but that it was not updated since a relatively long time. It would be great if Zunit had integrated code coverage, but I think an independent script is still a good idea, not being tied to a particular testing framework. This is why I'm opening this ticket π
There is also no option to disable it since the opposite is meant to be the default behaviour
Affects ZSH 5.0. From ZUnit's own tests:
β Test _zunit_assert_in failure
''a' is not in (x)' is not the same as ''a' is not in (x ; y ; z)'
If I set the var in the .yml file it causes all of my tests to pass immediately (when they shouldn't) and when I set it on the cli it runs as if no time limit was set. I know it's not working as expected because one of my tests goes into an infinite loop and the zunit process never times out, and must be manually killed.
At the moment if --fail-fast
is enabled the script exits immediately. We should call a proper shutdown handler here which can kill the revolver process rather than this being repeated throughout the program, and print the remaining HTML output (as the report stops being written when the exit is triggered).
For some reason, even though test files are parsed line-by-line, the tests in each file don't always run in the correct order. They should.
Currently, every file in the tests directory is treated as a zunit test file, and an error is thrown if the file does not start with the zunit shebang:
Lines 396 to 425 in 8048638
Is there a reason for not filtering out files that lack the .zunit extension? I have experienced a number of situations where zunit fails due to trying to read a macOS .DS_Store or vim .swp file as a test file.
Having this option would be very useful for being able to integrate nicely with CI tools like CircleCI :)
In ZSH 5.0 the $path
variable is already typecast as an array, and assigning a string to it throws the error path: can't assign initial value for array
.
Renaming the variable should fix the issue.
I made an @setup
test but some of the things never get parsed...
@setup {
zinc_left=( zincs_userhost )
prompt_zinc_setup
echo $zinc_left
this is something that gets ignored by zunit but shouldnt!
literally anything past that prompt_zinc_setup gets ignored
}
It goes on to the tests later on in the file and they all fail since it's as if prompt_zinc_setup
never got executed, and as the example says, the echo doesn't get parsed or executed either.
Trying to use Zunit here: https://github.com/robobenklein/zinc/blob/261cf980b813ea90424e49a3b5cba554072df42f/tests/zinc-base.zsh
Tests should be independent from what's contained within the users zsh configuration as it can interfere with the test process. In short, tests should not load the user's zshrc profile when running tests.
(PS: Only just realised I had an invitation pending for me to join this project, sorry about that!)
The only thing that currently generates warnings are risky tests (without any assertions) but when these occur, the test still counts as passed. We should make warnings represent their own state, and not increment the passed count, otherwise the --allow-risky
option provides no benefit.
Mod the init
command to generate a .travis.yml
file when initialising a project. Will probably ask the user whether they want this first, and add a CLI option which answers the question in case it is used in a script.
We have CLI options for these (--fail-fast
and --allow-risky
), but it would be nice if they could be defined in .zunit.yml
as well.
Currently the automated deployment uploads 6(!) times, because it deploys at the end of each test run. Now that Travis have released support for build stages, we should put the deploy step in it's own build stage so it only runs once, and after all tests have passed for all ZSH versions.
At the moment, execution halts and results are not printed to screen. They should be.
Depends on #24
Need some kind of reporting system, and options to specify which ones to use.
TAP is probably good enough as a starting point. (TAP generation added in #10, just need to redirect it to a file).
HTML would be nice-to-have.
Hello,
I did a quick glance at zunit's source and saw many sed
and other external programs invocation. An example found by ack
:
src/commands/run.zsh
221:# Encode test name into a value which can be used as a hash key
228: | sed 's/\- /-/' \
229: | sed 's/ \-/-/' \
The project might already be very important, I often think about using it after positive experience with Bash BATS
, however such coding style is IMO a problem, I wouldn't introduce forking-style tool to my projects. BATS probably does the same, but on Bash it is different, there rather is no choice except forking for certain features. On Zsh, one can code like in Ruby and other "real" programming languages. I've made a handbook explaining the no-fork way: Zsh Native Scripting Handbook.
The project seems to be inactive β I see it has been implemented, works, so no new commits appear. I'm thinking in engageing into zunit, but would you accept no-fork-replacements versions of existing code in pull requests?
When set, will stop the execution of tests at the first sign of failure.
The assertion library is rather robust, and growing, but it would be handy to provide access to pass
, fail
or error
a test manually and end its execution, like we already can with the skip
helper.
Example:
@test 'test some-other-command' {
if some-other-command; then
pass
do
[[ $? -eq 127 ]] && error 'some-other-command not found'
fail 'some-other-command failed'
}
This was in the README prior to v0.6.0, but due to the new compilation step on install the method that was in there would not work, so I removed it. If anyone uses zgen and knows how to install ZUnit in a way that would get it working please feel free to submit a PR to add it back in.
There are occasions when tests can get stuck in an infinite loop. We should allow for a time limit for tests to be specified.
When the time limit is reached, the subprocess for the test will be killed, and the test marked as errored.
We can also specify an overall time limit for all tests, and cease execution when it is reached.
At the moment if a test is skipped, the overall run returns a failed exit status as the total count does not match the passed count. This should be changed to compare against passed + skipped
so that the overall run results in success.
The project seems to be interesting and enough-invested (>190 commits). But README doesn't provide examples. Can some examples be added?
BTW, maybe you will be interested β there's a tool that parses scripts using Zshell's (z)
flag and extracts functions. I use it in one project β no need to load whole crasis
script, instead the extracted functions are autoloaded. The extraction is zsd-transform
, autoloads are in runtest.zsh
: https://github.com/zdharma/zplugin-crasis/tree/master/test/bin
$ cat tests/test_cover.zunit
#!/usr/bin/env zunit
@test "cover prints help with -h option" {
run cover -h
assert $state equals 0
}
$ zunit tests/test_cover.zunit
Launching ZUnit ZUnit: 0.8.2
ZSH: zsh 5.3.1 (x86_64-debian-linux-gnu)
_zunit_run_testfile:21: tests: assignment to invalid subscript range
The basics are included, but the more the merrier. Any ideas for new assertion methods are welcome.
It seems as though when a line in @setup
or @teardown
returns with a non-zero exit code, all tests will fail. This seems like it might be a bug, but even if it isn't, it would be helpful to get more diagnostic information from zunit when this is the case.
By way of example, here's a zunit script:
#!/usr/bin/env zunit
@teardown {
# When this line is used, the test fails:
[ -f tests/_support/out.dmg ] && rm tests/_support/out.dmg;
# When this line is used, everything works:
#if [ -f tests/_support/out.dmg ]; then rm tests/_support/out.dmg; fi
}
@test 'Test that should pass' {
assert 1 equals 1
}
When run with zunit --verbose
, you just get a fail on "Test that should pass".
To be fair, I wasn't aware that the short-circuit evaluation would return a non-zero exit code when the condition is false, but this definitely had me scratching my head for a good while since nothing in zunit's verbose output was pointing me in the direction of the problem.
The CI checks currently fail with error:
β Test _zunit_assert_contains match
'Assertion world does not exist' is not empty
The test that it refers is:
@test 'Test _zunit_assert_contains match' {
run assert 'hello world' contains 'lo wo'
assert "$output" is_empty
assert $state equals 0
}
It seems that somehow the second word of the string "hello world" is taken for a name of the assertion to run. What could be causing this?
Simplest way to explain this is with a test example:
@test 'This test will pass no matter what' {
assert "$output" contains "hello"
assert $state equals 0
}
This also works with any other empty variable being tested - but output is usually the most likely case.
This is likely to do with the way assert has been written so that the assert operator is after the first operand. So zunit treats the the assert as assert contains "hello"
since "$output" is empty.
There are many instances where all tests within a project need the same setup code (e.g. autoloading modules etc.). Including this same code in the @setup
method for each file would be tedious. It would be far easier to have a project-level bootstrap script which could handle setup for the whole suite.
tests/_support/bootstrap
) prior to tests being loaded, and execute it.init
.We can specify the time_limit
key in .zunit.yml
, but it would be handy to be able to override this by passing a --time-limit
CLI option.
It's sometimes very difficult to figure out why something has failed from within zunit because there is (that I can tell) no way of displaying the text written to stdout and stderr when running commands.
E.g. if I have assert $state equals 0
and it fails, it would be nice to see why.
This is especially important if the test is failing on a test CI such as travis but not locally.
Could use syntax like:
zunit /path/to/file@"The name of a test"
Would load that file, but only parse and load that particular test. Particularly useful when writing slow running tests to avoid running the whole file.
Currently, zunit init -[-t]ravis
fails if tests
and/or .zunit.yml
already existβ¦
It should probably just always generate the .travis.yml
unless [only] it already exists.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.