Why is it impossible to finetune GPT-2 Large on V100?.. about ru-gpts HOT 8 CLOSED

There are two variants of V100 - 16Gb and 32Gb. You need 32Gb. I can recommend https://immers.cloud service for that (I'm not affiliated).

from ru-gpts.

I just don't really understand why GPT-2 Large is not runnable while original GPT-2 774M runs fine?.. Maybe I'm doing something wrong or there is a way to compress the model for 16 GB VRAM?

from ru-gpts.

Runs or trains? You need way less memory to run a model.

from ru-gpts.

Train, I meant. GPT-2 large trains fine and actually fast enough on V100 with gpt2-simple library, but the model of same size from sberbank isn't available for finetuning, weirdly enough. With same weights... What's the difference if architecture is basically identical and the only difference is training sets?

from ru-gpts.

I've taken a quick look at gpt2-simple. It seems they use SGD and gradient checkpoints to fit the large model. I'd suggest you to set the batch_size to 1, optim to SGD and fp16 with O2 and see how it goes.

from ru-gpts.

UPDATE: It did work! O2 arg for fpt16 finally fixed the not enough memory error. But now there's another issue that I have no idea how arises. Honestly, the whole dataset loading seems faulty, for some reason, considering it's second error with it.

@king-menin, mind helping to figure out what it is?
UPDATE 2: The features file generated seems to contain something really weird with seemingly broken encoding. Maybe that's the problem?

from ru-gpts.

Update #3: The line_by_line allows you to start finetuning for a few iterations, but apparently, on some line it stops. Does it mean that some lines from the dataset just can't be loaded?

Update #4: Yeah, seems like it. Dataset with russian horror stories worked, while dataset of articles from foundation didn't. Seems like we'll need a function to preproccess dataset, apparently...

from ru-gpts.

PROBLEM SOLVED: GPT-2 model needs formatting. You can't just feed it raw text. You need to use <|n|> delimiter for your dataset, else it doesn't recognize it.. Honestly, you should update the pretraining section of readme, because this thing cost me 2 days of headache...

from ru-gpts.