effect of reducing batch_size and forward_batch_size about gdc HOT 2 CLOSED

On Sat, Jul 17, 2021 at 6:15 PM Hady Elsahar ***@***.***> wrote: Hey thanks for your interest in GDC, batch_size is the effective batchsize that should affect the results of your experiments. The way we implement it is by accumulating gradients from smaller forward_batch_size , (good blogpost on that <https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255>) so reducing the forward_batch_size wouldn't affect the results of your experiments it will just slow your experiments a bit. Now considering using smaller batch_size than 2048, in fact going as low as 512 might still work on easy constraints such as single words and some wordlists. Keep an eye on the "KL(p,q)" and "KL(p,pi)" metrics logged in wandb or neptune if they are decreasing then your experiments are going fine. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBQSIVJO2LUBLB6SPVJJ3LTYIFHJANCNFSM5ARO7NFQ> .