relative bias addition is row additions of permutations of a subset of bias vector ,
need to find way to get rid of for loop and code this in one. definitely parralizable.

logsumexp use torch's Cython

For each input sequence make relevant mask sub Matrix of initial register, so not instantiating again and again

really not clear from paper: 'computed as cosine similarity
with annealing between the encodings hx and
hy. It starts at 1 and ends atp
d, linearly increasing
over the first 10K training batches.'

Seems to interleave the two??

currently using torch gelu. fast gelu in paper

probs want no biases, stop model bloating

implement BPE from scratch with unk tokens hashed (although may achieve worse results on downstream tasks) as # perhaps not as general as bpemb's 25000.model

need to check ive implemented label smoothing with how authors how they label smoothed their objective sampling as objective fn includes negative sampling.