divya063 / testingframework Goto Github PK

This is high priority.

You should start from the assumption that the yaml file will only contain tests that we want to run, and so their required parameters. If a test exists in the framework but it is not listed in the yaml, it will not be executed.

As discussed at the very beginning, keep a clear structure between the test names in the yaml file and the class names in the codebase. This allows you to dynamically try to import the class with that name and get the results of the test.

From a previous Diogo's comment, some pseudo-code:

    from test_name import test_name.upper_case() as test
    testing = test(tasks[test_name])
    testing.exit_code()

This makes very easy to run the tests we want and add new ones.
In the second case, we should just create the .py file with the matching class.

There might be cases where a given parameter is repeated over and over in the yaml file, e.g., the filename to read/write/delete or the port number to reach JupyterHub.
One could deduplicate these parameters by declaring common parameters in the yaml and referencing those in the specific configurations of each test.

At https://github.com/Divya063/TestingFramework/blob/master/test.yaml#L40 you used mode=1 (which means 'active').
Could you use active and removed instead of 1 and 2, please?

These seem to be everywhere but they do not do much...
Could we fold them into the main test function?

It would be good to list in the documentation the packages required to run the framework.
E.g.,

• docker
• grafana
• sqlite3
• argparse
• yaml
• ...

The list is not exhaustive. To be checked.

I see some duplicated code in the storage tests.
For example, write, throughtput, and checksum test need to write to a file. A similar comment is applicable to the read operation.

Consider changing this and have minimal functions accepting the relevant parameters that are doing the job for everyone. IOUtils looks like the logical place to put this.

In the test test_mount_sanity.py, the timeout is a good idea but the test should spawn a subprocess and the timeout counts against this whole subprocess.

We need to rethink the tests requiring access to the cvmfs container and issuing commands like cvmfs_config probe

Looking for a match with 'eos' seems a bit weak here.
A longer pattern with a regular expression would be preferrable.