Minibatch sequence length about protein_bert HOT 2 CLOSED

@r-kellerm

The sentence you quote refers to the pretraining of ProteinBERT. For fine-tuning, we did filter sequences by length:

Throughout the entire fine-tuning process we used a sequence length of 512 tokens, except for a final epoch of 1,024 tokens which was introduced to encourage the model to generalize to different sequence lengths. For all but the last epoch, we filtered out all the proteins larger than 512 tokens from the training and validation set, and trained the model on the remaining records. For the final epoch, we trained the model on records from the training and validation set of size up to 1,024 tokens.

(see "Fine-tuning and evaluation" in the Supplementary Methods for more details)

Since ProteinBERT is pretrained on all sequence lengths and includes different sequence lengths in fine-tuning, we show in our paper that it can generalize to sequence lengths that aren't encountered during fine-tuning (see Figure 4 in the paper).

In all cases, whether or not sequences are filtered by length, START and END tokens must be included prior to sub-sequencing.

from protein_bert.

Thank you very much for the clarification!

from protein_bert.