Hey,

thank you so much for providing both a video and presentation slides for everyone!

I'm currently delving deep into the workings of transformers, and your materials have been incredibly helpful to me!

At first, I thought I had understood everything by now, but as I was reading some other literature, a few minor questions popped up, and I would greatly appreciate it if you could provide some insight into them.

Here my questions:

1. Did you invent the numbers in your visualizations, or did you create a notebook for each step and then visualize the values based on the model's output? (I ask because, for the purpose of understanding, I'm trying to create some of my own visualizations and I've set up a separate notebook for that. This also brings me to my second question.)

2. I was a bit confused because in your explanation of the input embedding, you exclusively use "regular" words, but later in your overall visualizations of training or inference, not only the words are present but also the tokens SOS and EOS. Did you omit these tokens for the sake of clarity in explaining the input embedding, even though they should actually be included and, just like the regular words, receive an input ID and an embedding vector? (And this question is also related to the next one.)

3. I'm a bit confused because you use both the SOS and EOS tokens for the input, but for the output, only the EOS token. I've been experimenting a bit with the T5 model from Hugging Face, and there it seems to be the opposite (you have only an EOS token for the input, and in return, you get a pad token at the beginning of the output, which can presumably be interpreted as an SOS token here, along with an EOS token). Is this then dependent on the library used and essentially doesn't matter?

4. This question is related to the second question. You use 'seq = 6' as the length of the input sequence "YOUR CAT IS A LOVELY CAT". Wouldn't the actual input length be 8 after tokenization (assuming both an SOS and EOS token are needed)?

I'm really sorry that I'm asking such meticulous questions, but I'm always very precise, and these questions are driving me crazy right now in my attempt to truly understand the intricacies of the Transformers' steps in detail.

Thank you very much once again
Sarah