qa-clean-test.py assumes simulators.sim and config_files.txt are in the qa-toolbox dir and does not accept arguments that the user can input into the Makefile.

• Create a new qa_test/regression directory and place tests in there.
• Add regression keyword in .cfg file.
• Add flag to test pointing to regression info
• Add code to compare the .stat.gold files at the bottom of routine test.run()

E.g. https://github.com/TDycores-Project/qa-toolbox/blob/master/qa_solution_comparison.py#L279

place in qa_common

2D plotting/error metrics assumes that x and y are equal, update error metric and plotting to account for different x and y lengths

Stat files are all showing up as same file in documentation

Improve documentation to make clearer:

-what need for solution writer in time slice vs observation
-what to input for solution writer functions
-running steady state simulations in python

For 1D plot comparison have each solution plot a different lifestyle to allow for easier comparison

In qa_test_doc.py, we need to swap the path to figures in description_text variable if the figure filename uses a relatively path (no "/" at the beginning of the path). May need to make description_text an array of lines (description_text = file.readlines()).

We cannot store absolute paths as they will differ across machines.

Can we use absolute paths in the Sphinx index.rst or underlying toctree files? If not, we need an alternate approach to mapping results to the Sphinx doc directory, because qa-toolbox/* may not lie below Sphinx/doc.

Truncating floating numbers for files comparing with regression tests

Do we create a log file that holds all the tests and generate index.rst at the end?
Add to index.rst incrementally?

E.g. fe4b4dc#diff-6c38eb1c2b96625155ae1f12b4a08554R239

Or we need to develop a more general approach to pre/post-processing that all simulators can leverage if need be.

Instead of relying on the makefile to clean all tests and files, we need a script that does so to avoid (1) -R issue on Mac, and (2) to leverage the paths to the tests in the config_files.txt (as these paths may not lie below the qa-toolbox makefile.

Be able to add one problem description for multiple qa tests

Qa-toolbox will check and see if observation or time slice files are available, change to have it only check if time slice or observation is specified in the input deck

Currently using description_ROOT.txt where where ROOT is the root name of the test case. We need to support two alternatives. Assuming the following:

abc.cfg with test case xyz.pflotran

abc.description is used to describe the test problem at directory level
xyz.description is used to describe the test problem at the test case level

In other words, key off whether these files exist and insert them accordingly.

Change up documentation to take in list of requirements

Add in correct mapping for liquid saturation in PFLOTRAN simulator

The ability to add tests or fix tests without re-running the entire suite.

• Add the logic to run all tests regardless of failures.
• Add the logic to re-run the tests that failed. (something along the lines of make try-again)
• Add make clean

The script that builds the regression gold files fails to run properly:

https://github.com/TDycores-Project/qa-toolbox/blob/master/regression_tests/create_gold_file.py

We need a directory hierarchy in the .rst files written for documentation in order to organize the tests. The hierarchy will be based on the path structure of the test repository. The qa-toolbox will use this path hierarchy when writing .rst files to TDYCORE_DOC_DIR/docs/source/*.

In Tdycore we only look for Liquid_Pressure (which needs to be changed to Liquid Pressure) in the vtk file, should this be changed?

Make clean currently removes all .png files. But what if we have a .png that is a image used in the problem description. We need to develop a naming convention for automatically-generated images. These images will be deleted by 'make clean_tests' while the other images will persists.

We need documentation explaining what supporting .rst files are needed for each test in order to properly document the test.

PFLOTRAN has updated to use .pft instead of .tec files, need to change this in pflotran.py in the simulator modules when reading an observation file

Improve capabilities within toolbox for 3D tests

For each variable, we need to store the overall max error across time slices and observations for each run. These should be stored in the QATestDocRun class.

Adjust documentation when documenting a steady state problem

SolutionWriter puts certain variables into groups instead of datasets if units have spaces between the brackets. This was seen in crunchflow h5 files.

• Give full path to failed simulator output

• Timeout limit, user edited? Better messaging when fails because of timeout

Have better error messaging when certain packages numpy or scipy not installed.

Update documentation to include what should be installed with python

• Rename regression tests to: QA Toolbox Regression Tests
• Move to bottom
• Put under horizontal line

Error will occur if there is a space between a comma and a variable in the options file which will be confusing to users, fix this so spaces are removed when processing opt file

Devise a method for error messaging with multiple simulators or comparison with an analytical solution

If we divide by zero in relative error it is set to 1e22 but perhaps we should just include an offset instead

Remove time column

When incremental tests are being run, directory name is not being stored

Right now, we use "Liquid_Saturation". We want to use "Liquid Saturation".

make clean_tests cleans up files that are specific to a simulator. We should develop simulator-specific scripts that perform the cleanup only if the simulator is present. In other words, we should not be traversing the test suite removing PFLOTRAN output:

-find . -type f -name 'pflotran_run.h5' -print0 | xargs -0 -r rm

if PFLOTRAN is not used.

In other words, the qa-toolbox makefile should only clean up files generated by qa-toolbox scripts, not the simulator output.