Can transformers sort lists of numbers? Is there a limit to how long these lists can be?

We're trying to gain an intuition and surface-level understanding of what Transformer models can do. Open questions:

• What are they good at? What are they bad at?
• Can they operate with small vocabularies?
• How resistant to noise are they?
• How do they work with out-of-sample input sizes?

Things We've Learned

• Use RAdam not Adam. Or use learning rate warmup.

Reverse a Sequence

• We need to use RAdam
• We can reverse short sequences
• We can reverse long sequences
• We can reverse short sequences with large vocabularies
• We can reverse large sequences with large vocabularies

Finding Adjacent Duplicates

• We need ~10,000+ examples to generalize on a sequence length of 10 (Vocab of 10,000)
• It's hard to train long sequences with sparse outputs directly
  • We tried FocalLoss but couldn't get it to work
• We can train on short sequences and scale them up until we have a model that can predict on long sequences:
  1. Train on sequence length 10
  2. Train on sequence length 50
  3. Train on sequence length 100

Finding Adjacent Repeats

• We used 10,000+ training examples based off our previous investigation
• It's also hard to train long sequence with sparse outputs directly
• Since the outputs aren't "quite" as sparse, we can get away with:
  1. Train on sequence of length 10
  2. Train on sequence of length 100