question about the memory calculation about llm-analysis HOT 2 CLOSED

Thank you for your clarification.

from llm-analysis.

Hi @ShouyangDong, thanks for bringing up the questions/issues

1. there is a *2 when calculating weight_memory_layernorm_per_layer, https://github.com/cli99/llm-analysis/blob/main/llm_analysis/analysis.py#L306, we count the weight of each layernorm as hidden_dim
2. layernorm is not tensor parallelized, thanks for catching this, I removed the dividing by tp_size.
3. the llama2 model uses gated linear units and I recently added a field in ModelConfig to specify mlp_gated_linear_units to true in model json files. If the model config is pulled from HF by name, there is no equivalent information to use, so I added https://github.com/cli99/llm-analysis/blob/main/llm_analysis/config.py#L218-L223 and make the assumption that if the expansion ratio (intermidiate dim / model dim) is 3.5, the model uses gated linear units. Now the params estimation for llama2 shall be correct.

from llm-analysis.