Experimenting with finetuning GPT2Large on Colab's V100 about ru-gpts HOT 4 CLOSED

try to decrease fp16_opt_level

from ru-gpts.

try to decrease fp16_opt_level

Yeah, the problem here is that with O2 it still doesn't want to run training and goes OOM, even at lesser block sizes. Weird thing here is that gradient checkpointing is present in your model implementation, but somehow it doesn't optimize model enough for greater order models to be ran on less VRAM hardware, which is really weird... Still trying out different things to make it work.

from ru-gpts.

Hey guys,
@fen0s I'm trying to do the same thing - fine-tune GPT2 Large model in Colab, but I run out of RAM as well as you do. I tried to resolve this issue with a loop while True, but it doesn't seem to work either. Did you manage to find something to make it work? 🤞

from ru-gpts.

try use deepspeed version of script

from ru-gpts.