thu-pacman / fastermoe Goto Github PK

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

比较好奇这个仓库在千卡集群上，dense 20b，16 expert top2 的情况下，mfu 能到多少

Switch Transformer is probably the only MoE architecture that currently has open-source pretrained checkpoints.

Recent work shows that MoE like SwitchTransfomer has extremely huge potential for scaling and instruction finetuning. Actually, after instruction finetuning, MoE architectures like SwitchTransformers strictly outperform their compute-matching dense counterparts.

Huggingface Transformers is the most popular transformers library. However, their implementation of SwitchTransformers is very vanilla and does not support MP well.

It would be awesome that FasterMoE could support SwitchTransformer. I believe this feature will make a huge impact in the large language model community.