openwdl / wdl-tests Goto Github PK

To get around #1, I'm running extract_tests.py without specifying an output folder:

python extract_tests.py -i ~/Documents/wdl/wdl/SPEC.md -d ~/Documents/wdl/wdl/tests/data/

I also changed this line to get around the data and test directory mismatching:

output_data_dir = output_dir

This lets me run run_tests_miniwdl.py, but for pretty much every test, I get an error similar to this:

declarations.wdl: ERROR
-----------------------
Return code:
Invalid output(s):
  dir: /home/heaucques/Documents/wdl/wdl-tests/scripts/tests/20240529_125428_declarations != None
  outputs: {'declarations.pi': 3.14} != None

I assume it's because of these lines, where the condition seems to be always true:

This seems to be consistent with every test, as MiniWDL always has a consistent output. For example:

(venv3.12) heaucques@pop-os:~/Documents/wdl/wdl-tests/scripts/tests$ /home/heaucques/Documents/toil/venv3.12/bin/miniwdl run -p . -i {} test_sub.wdl
2024-05-29 13:01:44.709 wdl.w:test_sub workflow start :: name: "test_sub", source: "test_sub.wdl", line: 3, column: 1, dir: "/home/heaucques/Documents/wdl/wdl-tests/scripts/tests/20240529_130144_test_sub"
2024-05-29 13:01:44.711 wdl.w:test_sub miniwdl :: version: "v1.12.0", uname: "Linux pop-os 6.8.0-76060800daily20240311-generic #202403110203~1714077665~22.04~4c8e9a0 SMP PREEMPT_DYNAMIC Thu A x86_64"
2024-05-29 13:01:44.728 wdl.w:test_sub done
{
  "dir": "/home/heaucques/Documents/wdl/wdl-tests/scripts/tests/20240529_130144_test_sub",
  "outputs": {
    "test_sub.choco4": "I like chocolate when\nit's late",
    "test_sub.chocoearly": "I like chocoearly when\nit's early",
    "test_sub.chocoearlylate": "I like chocearly when\nit's late",
    "test_sub.chocolate": "I like chocolate when\nit's early",
    "test_sub.chocolove": "I love chocolate when\nit's late",
    "test_sub.no_newline": "I like chocolate when it's late"
  }
}

For example, in the hello.wdl test, when the subprocess output is parsed out into key value pairs, key = outputs and value = {'hello.matches': ['hello world', 'hello nurse']} but the key outputs is compared to {'hello.matches': ['hello world', 'hello nurse']} from the expected value supplied by test_config.json. So the condition key not in config["output"] will always be true, leading to an expected actual output of None. Maybe it should be the value from MiniWDL that is compared?

I'm having trouble running the run_tests_miniwdl.py script after extracting the test files out of the spec with extract_tests.py.

I'm running this command to extract the tests:

python extract_tests.py -i path/to/SPEC.md -d path/to/data/ -O tests

But when I run the run_tests_miniwdl.py script with:

python run_tests_miniwdl.py -T tests

I get a CalledProcessError. It looks like this line changes the running directory for miniwdl, but the WDL paths specified in test_config.json do not reflect this change. For example:

...
    "id": "hello",
    "path": "tests/hello.wdl",
...

Thus, the value passed into the cwd is tests/data, which eventually leads to the script being unable to find the file:

[Errno 2] No such file or directory: 'tests/hello.wdl'

I think this issue will occur whenever a data directory is specified and when an output directory is given.