Memory Leak both on CPU and GPU about denoising_dihard18 HOT 3 CLOSED

@staplesinLA, I've replicated this issue myself using both CNTK 2.6 and the latest nightly build running on Ubuntu 18.04 LTS. It looks to be a known issue since at least 2018:

• microsoft/CNTK#2834
• microsoft/CNTK#3578

The Keras team proposed the subprocess workaround, which seems to work, but also degrade speed: keras-team/keras#9022

from denoising_dihard18.

Sorry for the late reply. How much memory of your GPU? It will take less than 4GB when processing a 5-minute segment.

from denoising_dihard18.

Thanks for the reply,

My GPU has 6 GB GDDR5 and actually it doesn't matter how i set the truncate_minutes parameter it will always go OOM at some point.

It is CNTK fault which does not seems to release the memory at each iteration.
I've been able, thanks to a suggestion, to go around this issue using the multiprocessing python package and calling the decode_model function inside a process.
Unfortunately this slows down inference considerably like 2 times slower.

from denoising_dihard18.