This document provides step-by-step instructions to reproduce the analyses, tables, and figures reported in the paper. In addition, the Listings.md file provides some examples of QMutPy's quantum mutation operators to showcase their functionalities.

Note: the following bash commands have been tested and used on a Unix-based machine and therefore might not work on other operating systems, e.g., Windows.

The subsequent analyses require, at least, the following tools to be installed and available on your machine:

• GIT to clone any required repository.
• GNU Parallel to parallelize experiments.
• Python (any v2.X.X would work)
• R

In the tools directory, run the following command:

./get-tools.sh

to automatically

• setup a Simple Python Version Management: pyenv and a Virtualenv.
• install the required Python version for the experiments.
• get Qiskit-Aqua v0.9, install its dependencies, and compile it.
• install the required R packages to run the data analysis.

Once this command has finished all required tools, Python environments, and subjects to evaluated should be properly configured.

• qmutpy-support/mutation-operators.csv lists all mutation operators supported by QMutPy, including classic and quantum.

• qmutpy-support/mutpy-time-values.md provides information on how time is computed by MutPy.

• qmutpy-support/utils/yaml2csv.py converts a YAML file generated by MutPy, and therefore by QMutPy, into a CSV file. The conversion of YAML files into CSV ease the data analysis in R.

• qiskit-aqua-support/equivalent-gates.csv lists all equivalent gates used in the empirical study.

• qiskit-aqua-support/make-equivalent-gates-plot.R automatically generates Figure 1 in the paper.

To identify the programs available in the Qiskit-Aqua v0.9 for which there is a correspondent test suite, in the qiskit-aqua-support directory execute the following command:

./get-subjects.sh

This script produces a CSV file named qiskit-aqua-support/subjects.csv with the following information:

• algorithm_name
• algorithm_full_name
• test_suite_full_name

For example:

grover,qiskit.aqua.algorithms.amplitude_amplifiers.grover,test.aqua.test_grover

Once the set of subjects has been identified, one could run, in the qiskit-aqua-support directory

./get-tests-coverage.sh

to collect code coverage (at line level) of each test suite. For each test suite this script generates a JSON report under tests-coverage/, e.g., tests-coverage/qiskit.aqua.algorithms.amplitude_amplifiers.grover.json. Once all test suites have been executed, this scripts collects all coverage data in a single CSV file (qiskit-aqua-support/tests-coverage.csv) which follows the following format:

• algorithm_full_name: the algorithm's canonical name as, e.g., qiskit.aqua.algorithms.amplitude_amplifiers.grover
• test_suite_full_name: the algorithm's test suite canonical name as, e.g., test.aqua.test_grover
• number_of_tests: the number of test cases executed during code coverage analysis
• number_of_tests_skipped: represents the number of test cases excluded by the test framework
• file: represents the file for which coverage was collected, e.g., qiskit/aqua/algorithms/amplitude_amplifiers/grover.py
• statement: statement's line number which might include several lines of code
• line: line's number of a line of code
• covered: either 1 if a line was exercised by the test suite, 0 otherwise
• excluded: either 1 if a line was excluded from coverage report, 0 otherwise

For example,

qiskit.aqua.algorithms.amplitude_amplifiers.grover,test.aqua.test_grover,5,0,qiskit/aqua/algorithms/amplitude_amplifiers/grover.py,12,12,1,0
qiskit.aqua.algorithms.amplitude_amplifiers.grover,test.aqua.test_grover,5,0,qiskit/aqua/algorithms/amplitude_amplifiers/grover.py,12,13,1,0

Similar to the qmutpy-support/utils/yaml2csv.py script, the qiskit-aqua-support/utils/json2csv.py script converts a JSON file generated by the Python coverage tool (Coverage.py) into a CSV file.

One could then run, in the analysis directory,

Rscript subjects_as_table.R subjects_as_table.tex

to generate a latex-based that summarizes subjects' data, e.g., number of tests, coverage, etc.

To run QMutPy on the 24 quantum programs the scripting in the experiments directory should be used as follows:

# Create a list of jobs (where each job is a repair attempt)
jobs_file="qiskit-aqua-jobs.txt"
./create-qmutpy-jobs.sh
  --quantum_framework_name <name, e.g., qiskit-aqua>
  --quantum_framework_root_path <path, e.g., $(pwd)/../tools/qiskit-aqua/>
  --quantum_subjects_file_path <name, e.g., $(pwd)/../qiskit-aqua-support/subjects.csv>
  --quantum_mutation_operators_file_path <name, e.g., $(pwd)/../qmutpy-support/mutation-operators.csv>
  --report_output_dir <path, e.g., $(pwd)/../data/qiskit-aqua-mutation-testing-data/>
  --jobs_file_path "$jobs_file"

The experiments/create-qmutpy-jobs.sh creates a jobs file with as many calls to QMutPy as number of programs x mutation operators to evaluate. Each job is basically a call to the experiments/run-qmutpy.sh script which runs QMutPy on a single quantum program on a single mutation operator. Data generated by each job is written to the provided report_output_dir argument.

The jobs file can then be used with GNU-Parallel, e.g.,

parallel --progress -j 8 -a  "$jobs_file"

to run all jobs in parallel.

Once all jobs have finished, all mutation testing data can be collected in a single CSV (to ease further analyzes) by running the following command in the data directory

./get-data.sh
  --output_file <path, e.g., qiskit-aqua-all-mutation-operators.csv>
  --exps_dirs <path, e.g., $(pwd)/qiskit-aqua-mutation-testing-data>

The scripting in the analysis automatically performs data analysis, generation of tables and figures.

To generate Figure 2, run

Rscript time_values_as_plots.R time_values_as_plots.pdf

To generate Table 2 and 3, run

mkdir summary_data/
Rscript summary_data_as_table.R summary_data/

To generate Figure 3, run

Rscript num_tests_to_kill_a_mutant_as_plot.R num_tests_to_kill_a_mutant_as_plot.pdf

To generate Figure 4, run

Rscript kill_reason_as_plot.R kill_reason_as_plot.pdf

The exploratory-experiments directory provides support for Section 5 in the paper. In detail, it provides

• The changes we made to each quantum program's test suite are available as git patches. For instance, exploratory-experiments/coverage/non-covered-to-covered-code.patch lists the changes we made to investigate the hypothesis described in Section 5.1, and exploratory-experiments/assertions/not-killed-to-killed.patch lists the changes we made to investigate the hypothesis described in Section 5.2.

• Scripts to re-run mutation analysis on the augmented test suites.

  • exploratory-experiments/coverage/run-exp.sh for Section 5.1
  • exploratory-experiments/assertions/run-exp.sh for Section 5.2

To generate Figure 5, run (in the [analysis][analysis] directory)

Rscript var_X_vs_var_Y.R var_X_vs_var_Y.pdf