t4im / shakedown Goto Github PK

To aid #5, an intuitive way to specify test tags/categories in a test-specification is needed.

For example with travis-ci.

Use the feature proposed in minetest/minetest#1587 after it was implemented to color failed vs passed tests in the console to ease recognition of each type.

... to handle cases, in which the declaration order needs to be different to its execution.

• There's already support for different formats, but no way to decide yet, where it should be formatted to (file? ingame console? formspec? terminal? default:sign?)
• Many report-event handlers still don't know the test outcome, which makes it hard to e.g. implement a TAP formatter for CI systems.
• Formatters could be given the chance to be configurable in appearance
• Despite skipping test steps on success, theres still a lack of "verbosity" control

blocks #13

Using conjunctions as steps makes the code harder to reuse and is seen as an bdt antipattern.
A test for common conjunctions (and, but) should be added, that warn the test authors about this.
Best time might be after minetest/minetest#2263 is resolved, so we can log the warnings properly.

Allowing to substitute selected variables in descriptions will make descriptions easier to maintain (instead of having to use concatenation for example) but also help removing specifics to help with #10

Fixtures, that are run after/before every contained testcase should be added.

On a testing server, testers might be provided more tools for testing than on a production server, but some tools might not be as "save" to give out (like running a complete test suite, that will clog up everything)
There should be an easy but sufficient permission model.

Asserts define hard expectations that have to be met, or the test will fail.

However not all expectations are valid all the time.
A conditional expression will skip the complete definition with unmet expectations.

Assumptions solve this, by treating the test as "skipped" (see #2 ) if failing instead. This way the test remains defined, and can e.g. be rerun, once the expectations are met.

The player mocking object still lacks a proper implementation for inventories.
Options are:

• Use detached inventories (but these lack a way of deleting these again)
• Mock inventories, too

Currently tests can either fail or pass. But sometimes tests should be neutrally skipped instead.

allow for step definitions to be abstracted and be reused as a whole

...at the very least for what is being set after a simple construction.
Formspecs have some deprecated fields to look for and should match the provided inventory size (as long as it is matchable: Some complicated formspec/inventory setups might not be as easy to test)

https://github.com/minetest/minetest/blob/master/doc/lua_api.txt#L2126

Multi line string descriptions should be properly trimmed and formatted.
On the other hand, they also should be discouraged.

Find a good way to run tests with provided parameters, and to provide these parameters.

Currently tests can only be selected by setting a config variable in minetest.conf to a filter that is matched against the spec-description followed by (re)starting the game. There is also an ingame command that runs based on the same filters. But since the world is already loaded by then, some tests might not be runnable anymore. Others run very slow, due to worldupdates.

Generally it is very inconvenient to run selected tests during test authoring or mod testing.

There needs to be a better way to provide match criteria and a better selection mechanism.