关于在香橙派上部署的一些问题 about rwkv_pytorch HOT 2 OPEN

可以看一下这里https://gitee.com/guizhiyu/rwkv_ascend

from rwkv_pytorch.

For RWKV, the time state is very sensitive, so fp16 can cause numerical overflow. Your approach is correct, but it will make the inference results different from what is expected.

For time-related calculations, due to exponential decay, precision is lost as values approach 0, rather than overflowing. Therefore we need to use fp32 for Time-related calculations.

FP16 overflow is addressed by x/2 every 6 layers.

However, this may have huge differences from original, therefore we have no plan to support it now.

from rwkv_pytorch.