Memory leak when symbolic circuits used with scipy.minimize about pytket-docs HOT 6 CLOSED

Hi, thank you for reporting this. We have identified a few memory leaks in our codebase and are continuing to hunt for others. Fixes will appear in the next release. We cannot guarantee that we will be able to eliminate all leaks in a high-level procedure like this since it is also dependent on the implementation of the simulator and minimiser which are beyond our control.

from pytket-docs.

Thank you! For your info:

• the backend I used was ProjectQ.
• the minimiser was scipy optimise, L-BFGS-B. The few similar incidents reported tend to suggest its not the minimiser e.g. https://stackoverflow.com/questions/35637270/why-does-scipy-optimize-fmin-l-bfgs-b-memory-usage-increase-with-iterations

In my previous analogous implementations, where I basically generated a new circuit each time I update the parameters, and did not use parameterised symbolic circuits and substituting symbols, there was no memory leak. I also used ProjectQ then. Which may suggest that symbol_substitution is responsible perhaps? Thanks a lot!

from pytket-docs.

After some investigations, we found the following were causing memory bloating in typical variational experiments:

• The backends typically cache the results of every circuit run through them during that python session, which can end up getting pretty big. We are fixing this by making the get_<>_expectation_value methods in pytket/utils prune the cache as they go. You can mitigate this in your own code by running backend.empty_cache() at the end of the objective function. However, I suspect this shouldn't apply here since your code above is using the native support for expectation values within ProjectQ which doesn't add to the cache.
• A handful of memory leaks in our C++ code whereby gate definitions would stay in memory even if there are no longer circuits that use them (which gets pretty bad for variational stuff when we are creating new gates with different angles each time). This could be the cause, but it doesn't explain why this was only observed when using symbolic circuit definitions.

Both of these fixes will be included in our next release.

We could not find any leaks around our usage of symbolic expressions, but there is a small chance that something like this could be system dependent. Could I ask what OS and python version you are running on?

from pytket-docs.

Thank you very much! I will verify the above and report any findings in the comments below, as the next release gets ready.

OS: pop_os 20.04 (basically Ubuntu 20.04)
Python: 3.8.3

from pytket-docs.

I went back to some of my old code and it looks like running variational calculations where a new circuit is generated for each measurement (without symbolics) the problem persists! I will investigate more but I think the second fix may address it.

from pytket-docs.

pytket 0.6.0 has just been released with the memory leak fixes and automatic pruning of the backend caches within our utility methods to prevent memory blowing up. If you encounter this issue with the new version, please let us know

from pytket-docs.