izhx / uni-rep Goto Github PK

@izhx Could you kindly share the training scripts for models of varying scales, or alternatively, the complete set of training parameters used for models at different scales?
I saw the following parameters in your paper (AdamW learning rate=4e-4, warmup period=0.25 epochs, batch size=1024, 8 negative examples, 8 A100-80GB GPUs, TF32). And I'd like to know the values of other parameters (whether all scales of the bloom model use epoch=3, chunksize=64, debias_batch, freezenonbias, max_seq_length=default 300 as written in train.py, and if there are any additional parameters included).