Out of memory exceptions cause next model to fail about devol HOT 4 CLOSED

@joeddav

Thanks for the quick response - I'll pull down your changes and try it later in the week!

Regards,

Alex

from devol.

@joeddav

I'm running into these problems trying to replicate your mnist results (in the "Running max of MNIST accuracies across 20 generations" figure).

I get

"Resource exhausted: OOM when allocating tensor with shape[1024,256]"

after running for a while (this time on the 811th candidate solution).

I am running on a Titan X with 12gb VRAM. Do you think that it's possible tensorflow / keras is having a problem with freeing memory? If so, do you know of a way of explicitly forcing memory to be freed after the candidate solution evaluation?

Regards,

Alex

from devol.

Yes, I think that after the error that memory is not being immediately deallocated. The puzzling thing is that the error happens when the next model is being compiled, not trained, so it's got to be some kind of allocation issue. We could try doing a gc.collect() after catching the oom error, but I'm not sure if that will actually make any impact in this case.

from devol.

Calling gc.collect() after the first OOM exception seems to solve it.

from devol.