rachitnigam / runt Goto Github PK

It would be nice to have a field in the config to specify how to build a binary. It would look something like this:

[[tests]]
...
build = “cargo build”
...

Runt would run this command once before running all the file tests.

Allow specification of tests where a non-zero error code is an error.

There is no reason for runt to tie together the frontend, which is responsible for figuring out which tests need to run, where the expect files are, etc. to the executor, which actually runs the commands in parallel and reports back results. We can use a generic intermediate form to specify which commands need to be executed for which tests:

[
  { "name": "compile:foo/one.txt", "cmd": "cc foo/one.txt", "expect": "foo/one.expect" }
]

The frontend is then responsible for computing these while the executor takes these as input and produces output results.

This will also allow us to connect to other testing frameworks like turnt.

I have two test suites using Verilator. Turns out that there is a race condition caused by two threads trying to read the same file at the same time causing one of them to fail.

make: execvp: obj_dir/Vmain: Text file busy
make: *** [Makefile:19: tests/verilog-primitives/register.vcd] Error 127

I don't know what to do here. The thing I'll do for now is just make a copy of the contended file so that these can be run in parallel. But maybe there should be an option to run a test suite after another finishes?

It would be nice if within a runt.toml there was an option to designate an output filename different than the basename of the input file.

As an example
Something like this

[[tests]]
name = "[core] parsing"
# Round-tripping from the compiler should not change anything.
paths = ["./tests/parsing/file1.futil"]
cmd = """
./target/debug/calyx {} -m file -p none -l .
"""
output = "output.futil"

Would output to a file called output.futil instead of file1.expect. The thought is that this might be able to assist in testing multiple steps in a compilation cycle generated from a single source file, as opposed to needing duplicate source files for each step.

For example we could have multiple tests starting from file1.futil.
The first test would check the first step against output.futil
The second test would run the first step then check the output of the second step against output2.futil and so on.

When running runt -s with expect_dir set to a directory that does not exist, the saving will fail and you have to manually create the directory. It would be nice if that was done automatically

I'd be happy to work on this if it is something you're interested in

Maybe use the --only flag to provide a regex and suppress non-matching strings. This would be enough to allow for:

runt --only tests/cat-test/a.txt

and have it run just display the output for that test.

Use the -o flag to provide a string that paths need to match to be run.

For tests where you want to run multiple commands to generate the test output, it would be very convenient to have a way to get the filename without the file extension. I don't exactly know what the syntax would be it would look something like this:

[[tests]]
name = "Verilog tests"
paths = [ "tests/verilog/*.futil" ]
cmd = "./target/debug/futil {}.futil -l ./primitives/std.lib -b verilog | {}.v"

where the command would be replaced with

cmd = "./target/debug/futil tests/verilog/seq.futil -l ./primitives/std.lib -b verilog | tests/verilog/seq.v"

Right now, it seems like only failing tests get a non-zero exit code. However, when running in pre-commit hooks or CI, I would like missing tests to also give a non-zero exit code since it likely means I either forgot to add an expect file, or forgot to check if the missing tests are fine

If you want to run a makefile to generate the file to do diff testing on, you are forced to make the command look like:

cmd = "make {} 1>/dev/null 2>/dev/null; cat {}; rm {}"

It would be nice if you could instead do:

setup = "make {}"
cmd = "cat {}"
cleanup = "rm {}"

Eventually, you could imagine automating the cleanup to remove any files generated during the setup command.

With the latest version, at most two suites can run in parallel. Let suites grab locks on the stdout and start dumping out results as they stream in.

When running a runt test which diffs from the previous version, it is a bit annoying having to re-run the whole test to save the result with the -s flag

https://crates.io/crates/insta solves this by creating .new files for tests that differ, and has a separate command insta accept to replace the old files with the new ones without re-running the test suite, perhaps that could be copied here?

When a path is a concrete path, don't run glob on it.