filmil / bazel-bats Goto Github PK

I'm seeing the following output:

$ bazel test
...
//ops/releng/scripts:releng_test                                         PASSED in 2.0s
$ for b in *.bats; do echo $b; bats $b; done
...
rollback-pods.bats
 ✓ should error on too few arguments
 ✓ should error on too many arguments
 ✓ happy path
 ✓ happy path with absolute revision
   happy path with relative revision                                                                                                        5/5
   bats warning: Executed 4 instead of expected 5 tests

What can cause something like this?

Currently in _bats_test_impl(), environment variables (from the env attribute) are expanded in a two step process. First, any $(location ...) (or similar) are expanded (using ctx.expand_location()), then any variables are expanded (using ctx.expand_make_variables()).

This covers many situations. However, if any variables expand to $(location ...) (or similar), an unrecoverable error is thrown (exception from Java). This level of recursive expansion is supported by built-in rules. It is unfortunate that those APIs are not exposed to Skylark at all.

For bats_tests targets to have the same env support that built-in rules have, proper expansion logic must be used. This can be done in 2 ways (this GH issue is to discuss which is preferred).

Use sh_test internally

This option is done by using sh_test as the main internal target for bats_tests. The existing custom rules (_bats_test_attrs and _bats_with_bats_assert_test_attrs) would still be generated, but the output executable is then wrapped in a sh_test. The env attribute would be passed to the sh_test -- and not to _bats_test_impl(). This allows for the environment variables to be properly expanded by the built-in (Java) implementation for built-in rules. Other environment variables which are manually set in _bats_test_impl() (e.g. TMPDIR and PATH) would remain expanding with their existing logic.

Use fancier bzl expansion logic

This option is to have better expansion logic in Skylark which mimics (and optionally improves upon) the built-in expansion logic. For more reuse, I put up a PR to Skylib (bazelbuild/bazel-skylib#486). This PR has more much detail about expansion logic in Bazel in the description. The logic in the PR also adds extra functionality beyond built-in expansion logic. This logic could either be referenced directly (adding Skylib as dep for bazel-bats) or added to a bzl file here to avoid extra deps.

I've implemented both of these approaches, and either way works fine. Let me know which you think would be better.

Hi @filmil, is there a way to output the junit formatted report in a directory with this rule, e.g., using bats --formatter and --output?

At the moment, README.md contains only the instructions on how to use top-of-tree code.

We should add instructions for using specific source releases too.

First off, thank you for creating this repo.

The last release 0.29.1 was released on Sep 26, 2019 . Since then there have been a number of great additions; namely, usage of http_archive() over new_git_repository() (which is more secure).

This issue is to simply request a newly published release that includes these changes (and others).

Thanks!

P.S. The WORKSPACE sample in the README is not consistent. It targets 0.29.1 (latest release), which does not support the (commented) sha256 parameter. Once a new release were to be published, and the README references it, it would work fine. Thanks again.

Issue #9 has the history. It seems that the images github uses to run the tests contains an incompatible version of parallel, which confuses the test runner.

It doesn't seem to be possible to pin the container to a particular version to avoid surprises.