Memory requirements for ContraCode about contracode HOT 3 CLOSED

Copying from #6

Pretraining is memory hungry as contrastive learning benefits from large batch sizes (see https://arxiv.org/abs/2002.05709). Moreover, the transformer backbone we leverage uses significantly more memory than typical image classification architectures.

We generally performed pretraining over 2-4 16GB V100 GPUs. We provide pretrained checkpoints due to the large cost of pretraining. Finetuning is very cheap and was performed on 1 V100 GPU.

Some recommendations to reduce memory consumption:
(1) reducing the sequence length for the Transformer encoder
(2) decreasing the hidden dimension size of our model
(3) adding checkpoint annotations for gradient checkpointing (e.g. PyTorch gradient checkpointing)

from contracode.

Thanks for your reply.
Your idea is very novel,and I will try it on your advice.
Thanks again.

from contracode.

Thanks! Feel free to reopen the issue if you have any further questions.

from contracode.