rahulschand / llama2.c-for-dummies Goto Github PK

As a newcomer to transformers, there are some questions about the matrix multiplication operator in run.c, can the author answer them.

In the forward function, it can be seen that the matmul function is called when calculating qkv, but the actual matmul is done with the calculation of a one-dimensional vector multiplied matrix, so the resulting qkv is also a 1*dim vector. But shouldn't qkv be a two-dimensional matrix in the transformer I'm exposed to? There is a little doubt here, I hope the author can answer it!! thanks！

I think its practical if you can include an example about a more detailed approach to training a model, because the examples in the original tinystories doesn't make sense to most of us.

If you can replicate a conversational chat from a custom dataset, just like what karpathy did, it would be really helpful, and his research could be passed on to a wider community.

Thanks.

karpathy/llama2.c#151

probably the comments most valuable - could ask chatgpt to not do intents if you choose this approach again.