Probtest is a suite of python scripts to test various aspects of weather and climate models. Even though it is mainly developed for the ICON model, it does not contain ICON-specific definitions in the source code but allows to model specifics in a configuration script.

1. Commands
2. Quick start guide
3. Developing in probtest

The suite consists out of the following scripts:

This script is the entry point to the suite that manages all the other scripts and sets up command line arguments with click.

To work with probtest, a configuration file is needed. This file contains all the options that are used for the various commands of probtest and can be generated with this command.

Perturbs netcdf files that can be used as input by the model.

Reads and modifies model namelists to run a perturbed model ensemble.

Generates a csv file containing the min, max and mean values for each of the selected fields on each model level and for each time step.

Computes the maximum spread in each of the selected variables for each time step within a perturbed model ensemble.

Compares two files generated with stats under the tolerance ranges generated by tolerance.

Uses all given stats files generated for a model ensemble with stats. From those stats files, randomly selects a specified number of members to generate the tolerances with tolerance. Repeats this process by iteratively increasing the number of selected members and the tolerance factor until finding a selection for which all other members pass the tolerance check.

Creates a plot visualizing the results of the comparison conducted with check.

Often people like to compare model output using cdo diff. This tool compares each point of each variable and time step and reports the largest relative error for each. This is different from stats which computes differences on statistics (min, max, mean) of each model level and time step. However, the perturbed ensemble can be used to estimate expected errors with cdo diff by applying the same algorithm and storing the output in a database.

The output of cdo-table is written in a Pandas dataframe. To ease reading, cdo-table-reader allows to filter the database by variable name, output file and time step.

Note: This script is ICON-specific because it parses the ICON logfile with hard-coded regex. Read the timing output from the model. These timings are stored in a tree format in the case of nested timings. It can either create a new database or append to an existing one. It produces three databases:

• tree: the relationship between the individual timers
• meta: some metadata storing information about the model run
• data: the actual timer data

Compares two performance databases generated with performance and checks if the current one is too slow compared to the reference.

Reads some data from a series of performance databases generated with performance and produces a graph showing the performance over time.

Visualize the performance database generated with performance.

Even though probtest is used exclusively with ICON at the moment, it does not contain any information about the model or its directory structure. This makes it very flexible and applicable to any circumstance (e.g. usable by Buildbot, Jenkins and human users alike). However, it also requires a lot of information about the model and the data to be processed upon invocation. Since a typical probtest usage involves multiple commands (e.g. run-ensemble -> stats -> tolerance -> check) this leads to a lot of redundancy in the invocation. Therefore, probtest can read commonly used input variables (e.g. the model output directory, the experiment name, the name of the submit script, ...) from a configuration file in json format. To further ease the process, these configuration files can be created from templates using the init command. A template for ICON is contained in this repository in the templates subdirectory.

This command sets up the configuration file. For more help on the command line arguments for init, see

python probtest.py init --help

The --template-name argument can be used to specify the template from which the configuration file is created. One of the templates provided by probtest is templates/ICON.jinja which is used as the default in case no other template name is provided. The init command replaces all placeholder values in the template with the values given as command line arguments. All other probtest commands can then read from the configuration file. The name of the configuration file to use is read from the PROBTEST_CONFIG environment variable. If this is not set explicitly, probtest will look for a file called probtest.json in the current directory.

Setting up the configuration file with init may not be fitted perfectly to where you want your probtest files to be. In that case, you can manually edit the file after creation. Alternatively, you can add arguments for your probtest commands on the command line which take precedence over the configuration file defaults. For more help on the options on a specific command, see

python probtest.py {command} --help

Objective: Run the mch_opr_r04b07 ICON experiment and check if the output of the run is ok. Probtest requires some additional python packages. On Piz Daint, there is a pre-installed python environment which can be loaded with:

source /project/g110/icon/probtest/conda/miniconda/bin/activate probtest

Alternatively, all requirements can be easily installed with conda:

./setup_miniconda.sh
./setup_env.sh -n probtest -d -u

Once set up, probtest can generate the config file according to your needs:

python probtest.py init --codebase-install /path/to/the/ICON/Installation/ --experiment-name mch_opr_r04b07 --reference /path/to/icon-test-references/daint_cpu_pgi/ --file-id NetCDF "*atm_3d_ml*" --file-id NetCDF "*atm_3d_hl*"

This will create a probtest.json file in the current directory. This file contains all information needed by probtest to process the ICON experiment.

With everything set up properly, the chain of commands can be invoked to run the CPU reference binary (run-ensemble), generate the statistics files used for probtest comparisons (stats) and generate tolerances from these files (tolerance).

python probtest.py run-ensemble
python probtest.py stats --ensemble
python probtest.py tolerance

Note the --ensemble option which is set to take precedence over the default False from the configuration and make probtest process the model output from each ensemble generated by run-ensemble. These commands will generate a number of files:

• stats_ref.csv: contains the post-processed output from the unperturbed reference run
• stats_{member_num}.csv: contain the post-processed output from the perturbed reference runs (only needed temporarily to generate the tolerance file)
• mch_opr_r04b07_tolerance.csv: contains tolerance ranges computed from the stats-files

These can then be used to compare against the output of a test binary (usually a GPU binary). For that, manually run the exp.mch_opr_r04b07.run experiment with the test binary to produce the test output. Then use probtest to generate the stats file for this output:

python probtest.py stats --model-output-dir /path/to/test-icon/experiments/mch_opr_r04b07 --stats-file-name stats_cur.csv

Note how --model-output-dir is set to take precedence over the default which points to the reference binary output to now point to the test binary output as well as the name of the generated file with --stats-file-name to avoid name clash with the stats file from the reference. This command will generate the following file:

• stats_cur.csv: contains the post-processed output of the test binary model output.

Now all files needed to perform a probtest check are available; the reference file stats_ref.csv, the test file stats_cur.csv as well as the tolerance range mch_opr_r04b07_tolerance.csv. Providing these files to check will perform the check:

python probtest.py check --input-file-ref stats_ref.csv --input-file-cur stats_cur.csv

Note that the reference --input-file-ref and test stats files --input-file-cur need to be set by command line arguments. This is because the default stored in the ICON.jinja template is pointing to two files from the ensemble as a sanity check.

Our tests are executed using pytest, ensuring a consistent and efficient testing process. Each test dynamically generates its necessary test data, allowing for flexible and isolated testing scenarios.

Simply run

pytest -s -v tests/*

in order to run all tests.

To run only a subset of test run

pytest -s -v path/to/your/test.py

Reference data, crucial for validating the outcomes of our tests and detecting any deviations in probtests results, is maintained in the tests/data directory. This approach guarantees that our tests are both comprehensive and reliable, safeguarding the integrity of our codebase.

Code is formatted using black and isort. Please install the pre-commit hooks (after installing all Python requirements including the pre-commit package):

pre-commit install

This hook will be executed automatically whenever you commit. It will check your files and format them according to its rules. If files have to be formatted, committing will fail. Just commit again to finalize the commit. You can also run the following command, to trigger the pre-commit action without actually committing:

pre-commit run --all-files

If you are using VSCode with the settings provided by this repository in .vscode/settings.json formatting is already enabled on save.