Is accumulating adaption loss then backward the same effect with accumulating gradients about learn2learn HOT 2 CLOSED

Hello @NookLook2014,

Yes, the two are the same but accumulating gradients is much cheaper than accumulating loss because you can free the activations.

One issue in your code: you accumulate evaluation_error.item() but should accumulate evaluation_error.

from learn2learn.

Hello @NookLook2014,

Yes, the two are the same but accumulating gradients is much cheaper than accumulating loss because you can free the activations.

One issue in your code: you accumulate evaluation_error.item() but should accumulate evaluation_error.

Thanks for your confirm. As to the issue, I used the evaluation_error but faced with the CUDA_OUT_OF_MEMORY issue even the meta_bsz is quite small.

from learn2learn.