Improving Qiskit Aer benchmarks about quantum-benchmarks HOT 3 CLOSED

Here is the updated circuit benchmark for qiskit run on a server with a P100 GPU. Note I didn't re-run any of the other simulator benchmarks, just used the existing data in the repo.

from quantum-benchmarks.

Hi @chriseclectic Thanks for your comment! It'd be nice if you could include the suggested changes in a PR, so I could help review it (since I'm still not sure which kind of option you would like to put into the qiskit benchmark), and I think we could also include multiple benchmarks for qiskit too. Please feel free to open a PR first, then I could also help you edit it.

In principle, as a demonstration of actual running time for simulation in practice, I think we should use user interface as much as possible. It's fair to benchmark through C++ if the C++ interface is an official API for users. But if the interface is in Python, we should in principle benchmark the python interface (through the standard python benchmark framework pytest-benchmark). For single gate benchmarks, yes, this is a test for the implementation of each instruction, it shows if certain acceleration tricks e.g SIMD is applied, or if the simulation algorithm is correct or not. But we currently do not have a C++ benchmark setup. This was also discussed in QuEST's benchmark review: #5

This is what we do for other frameworks, but qiskit is the only exception at the moment. I had to write a custom execute function since I was not familiar with the Qiskit simulation backend, and I found the user interface will spawn a task that gives constant time in pybenchmark measurements.

Regarding to stabilizer simulator, is this a fair comparison for other simulators? since the benchmark was mainly made for variational circuits (at least at the moment) and all other frameworks are benchmarking the full amplitude simulation. Maybe we should let the stabilizer simulators benchmark with stabilizer simulators?

I'd love to have more professional benchmark scripts from developers themselves for sure (which was what we did for other frameworks). Thanks!

from quantum-benchmarks.

I'm getting a PR ready that will update to the correct simulator backend, and will leave the timing as you have it currently setup with the native_execute function. We don't expose our C++ API directly so going through Python is fine. The Python overhead will just appear as a constant run-time for low qubit numbers. I think that for how Pytest works the native_execute function you have is the best approach since it bypasses the async job model for Qiskit providers.

As part of the PR I also enabled the QCBM circuit benchmarks since these are supported. The "native" gate set of the simulator doesn't matter since the Qiskit compiler handles conversion to supported basis gates (eg Rx -> u3, and Rz->u1). Another point is we just released an update to Qiskit Aer (version 0.4) a few days ago, which means the current native_execute function will no longer work (the internal method was renamed from _format_qobj_str to _format_qobj). This new release also included our first version of a GPU enabled simulator. Currently the GPU enabled simulator is only available for Linux and can be installed separately with pip install qiskit-aer-gpu. I can also add this to the benchmark scripts for the QCBM circuit.

One question with the benchmarks, and in particular GPU benchmarks, are you running them on the other configurations as single-precision or double-precision? We support both options for both CPU and GPU (default is double-precision).

With regards to the Stabilizer simulator, I agree it's not fair compare it to a statevector simulator because it can only simulate Clifford circuits, however since our simulator will choose it automatically if the input circuit if Clifford that is why you need to explicitly specify running on the statevector method.

from quantum-benchmarks.