Sometimes a test breaks and an error is produced. This also means the test never got to run so it has completed 0 of 0 test methods. Should add in a default fail state to handle this.

Description

When filesystem._list is called, the paths that it returns are like res://path/to//filename.gd. This caused a few problems when I tried to load the templates to copy them.

Possible Fix

In filesystem.gd at line 27 -

var absolute_path: String = "%s/%s" % [path, name]

here, the / between the two %s could be removed, but when I tried that, WAT.tscn kept causing the editor to crash and the error message indicated cyclic reference.
The crashes happened if the addon was enabled, if the scene was opened or if the scene was run (in this case, only the preview window crashed).

I don't know why this is happening but it seems when methods get tokenized, there is an additional parameter being added which causes issues later with the rewriter.

Some scripts being tested may be waiting for a yield, some of these yields may never be called so we need a way to cancel them after an expected time.

This would be quite handy when having many tests.

When you create a subfolder in tests and select it in the option button, then try to delete that subfolder via Godot. Godot will instead reimport your entire project (ie "res://") into that folder instead.

1. Create any folder in tests
2. Select that folder in the Select Directory Button
3. Now try to delete it (with or without it currently selected in the Select Directory button)

Currently when you double a scene, it is only stored in the variable and not saved. This was a half-measure because when saving it, it would use the wrong scripts.

• Save the scene
• Make sure the scripts are correct

Basically sometimes you need to create an object to suit a purpose and that object needs data. We take in the object script/class, read what it needs, populate it, then return it back up to the test so it would read:

var object = _create_junk_object(object.klass)

Not an issue for me personally but it seems to spam the output of anyone using. Need to ignore everything with WAT itself manually with the #warning_ignore comments.

When pressing Default Settings prefixes get cleared.

The log should be a tab alongside the other logs like the debugger and the game output IMO.

I have an Autoload Script that does not appear to be loading before the Tests begin to run. I am trying to create a sample project for you but I thought I would report it anyway first.

I will update this with more information when my sample project is ready for you.

We should be able to save test results in a resource script (if toggled as an option). We can then further use these to track any differences between changes by comparing new results against previous ones. Possibly even adding a counter along with them.

Shouldn't be that way imo.

Asserts, unlike expectations, will be used in fixtures (start, pre, post, end) to help ensure certain things are in place for all of the tests. If an assert fails, it will crash/skip the test script/method because the prerequisties are no longer in order.

https://github.com/CodeDarigan/WAT/blob/a573911ab7692b3d557daf821bd2e2ff4c9188d6/addons/WAT/utils/input_output.gd#L4
This script doesn't exist. Blank.gd should be renamed to blank.gd.

WAT had a to_string method which conflicts with the new to_string method in Godot. Rename for a fix.

When I mess with the UI elements of WAT (or sometimes testing servers) WAT goes crazy so I need to fix it via unload - reload. We should add a button to handle it automatically.

(I don't know how to do this easily!)

I've just downloaded version 3.0 and got an error message while trying to run a test. It turns out that the script runner.gd had at line 7 the following code:

const COLLECT = preload("res://addons/WAT/runner/collect.gd")

I had to change it to:
const COLLECT = preload("res://addons/WAT/Runner/collect.gd")

Because of Linux. I see that in the newest commit code there is the same problem of capitalized paths.

This code is messy and unclear but I haven't figured a better way to name it yet due to its recursive nature. We could possibly add more implementation details to the result object instead of display.

Problem:

Trying to activate the Plugin without a tests folder in your project results in Godot crashing.

Recreate Problem:

• Create a new Project
• Import the Plugin using GIT (not asset library)
• Activate the Plugin
• You will receive the Error "res://tests missing"
• Godot will Close

Workaround:

• create the folder "tests" in the root of your project

• activate the Plugin

• create

The idea here is that some preconditions must be met before our primary assertion is. However we don't want to write:

asserts.thing(precondition)
execute_test():
asserts.thing(postcondition)

The typical way around this is to store the precondition state in a variable like this:

var precondition = object.method()
execute_test()
asserts.thing(precondition)
asserts.thing(postcondition)

What I could do is create a precondition interface. This would essentially show up similar to asserts but it would specifically only show on failure to avoid any cognitive overload.

check(precondition, "X must be true")
execute_test()
asserts(condition)

Alternatively we could have precondition crash the test there if it is required.

Add a comprehensive optional logging system that logs everything correctly, outputs to the console and has different levels of output.

WAT looks a bit out-of-the-place at the moment.

I'd imagine since 3.2 shouldn't be a breaking change, that master should just default to it instead of a separate branch. Keeping this here as a reminder.

Problem: Hitting a breakpoint in a unit-test is currently not possible through the WAT interface.

Potential solution: I suggest adding a "Debug selected X" button somewhere in the interface that would launch a godot scene running the selected tests (either script or directory, like the current interface)

This would allow the use of in-built godot breakpoints without the users having to create custom debugging scenes.

CLI requires a proper exit code to be set with OS.exit_code = ? so CI (and other) platforms can take advantage of it.

Had a bug where my tests hung because I forgot to call "resume" after the yielded test method. Most obvious solution to this is to make a yield manager that stores yields, pop them on finish (and frees them) while also resuming if no yields left on test.

The CLI doesn't wait for yields in tests. Requires fix.

See #32 and #42 for related discussions.

The primary issue with Autoloads is that they don't seem to be instanced during tests (and or on the same tree?) However we seem to be able to call:

EditorPlugin.new().add_autoload_singleton("AlreadyExistingName", "res://AutoLoad.gd")

With no issues and it works inside tests? On that note however, "remove_autoload" will take it away from the current test and the entire project too so avoid that.

Godot crashed after I pressed "Run All Tests" after renaming a test script.

So I don't have to look up everything retroactively.

I've created a stable branch which does work but I don't consider it a true stable branch. This is just some prep for the next release. Once I have my first true release, I'll close this issue (which is only acting to serve as a reminder).

This was too complicated to handle for the Alpha. It requires quite a bit of instancing/re-instancing and complicated parsing.

Currently using a Linux-Godot3.2 Image on CIrcleCI. Should add a windows as well.

When running tests, you will see a line go up and down in 10 dots. This works fine except for when we're yielding in code. One way is to maybe consider running it in a different thread but maybe other alternatives also exist.

Apologies if this exists somewhere but I did not see it, nor did I find on the Wiki.

Please add the ability to specify the root folder for test scripts. I have several project where the tests are located in folders called

my_project_tests
another_project_tests

this lets me keep things separated and organized.

Ideally this would be a modifiable somewhere in the Project Settings.

Add a CLI so people can run and see tests from their CL. Make sure to allow for windows, linux & mac.

I'm freeing memory in a for loop. I'm a fucking idiot, no wonder its been crashing. Will be fixing that tomorrow.

Trying to set up Docker/CI for WAT I ran into the issue of WATemp not existing. This is because it is only created when you load the plugin, not when you run the tests. Which causes problems with automated git hooked tests.

I get the error

res://addons/WAT/test/expectations.gd:112 - Invalid get index 'NOT_IN' (on base: 'GDScript').

because const NOT_IN = "not in" is missing:
https://github.com/CodeDarigan/WAT/blob/58fe802fe6f23736a51c6ae118447b0f25f61df9/addons/WAT/constants/operators.gd#L1-L15

Sometimes when running the bootstrapped test scene_director.test.gd (either with other tests, its own directory or itself).

It seems this could be caused by trying to free an already freed reference OR by trying to remove a file too quickly/too many times.

It is a pretty specific issue so it probably isn't worth worrying too much overall so I'm just writing this issue to keep myself reminded of it in future.

Create a system to push errors and warnings to the Player. We should prefix the error with the source script + line along with the fact that it comes from WAT.

When you run the CLI and use yields, it may complain about memory not being freed. This is likely because we are quitting once we're finished which isn't what the GUI does. Might require a simple check on PREDELETE NOTIFICATIONs.

Right now we calculate a number of things about result lists in the display script itself which leads to an odd, somewhat recursive-lite style code that can be confusing.

Instead we could maybe create a middle-man script that stores all of that data in a struct.

Add an option for additional subdirectories being tested. This allows for different series of tests to be run for different circumstances. The more obvious example would be integration, end-to-end style of tests.

Also need to consider directories within directories, if the first dir is for the test type, the sub-subdirs will be for different configurations.

The first time you load the editor, the run all or run dir buttons seem to freeze (This is probably due to loading resources). They also seem to intermittently freeze up occasionally.

This is animation only. Their functions don't freeze (thankfully).

Ran into this issue when I needed to set up a connection before hand. Had to resort to sprinking yields throughout the script.

Basically change expectations that exist now to fluent assertions.

ie

asserts.has(value, container, context)

we would do

asserts.collection(collection).has(item).give_context("string")

or something like that.