about start tokens about spacetimeformer HOT 1 CLOSED

The start token is inherited from Informer. You're right that there's no new information there... it is mostly included as a trick to improve decoding performance. We are generating entire sequences of predictions in one forward pass instead of the autoregressive format you'd see in LSTM time series models (and generative language Transformers). Without a start token, the decoder sequence would be empty values and there wouldn't be much information for the early self-attention layers to work with. The start token lets information from the recent encoder sequence values go straight to the decoder self-attention. That info probably becomes redundant after the first cross-attention layer can access it from the end of the encoder sequence. This is just a guess but I think if we switched to autoregressive decoding we could drop the start token sequence and just use a single "start token" embedding like in NLP.

from spacetimeformer.