How to calculate Bert FLOPs about fastbert HOT 4 OPEN

There is no a general code that can test the FLOPs of all models, because different models are implemented differently.

If you want to get the FLOPs of a model, you should add the following code to the models' script:

flops, params = profile(model, inputs, verbose=False)

the model is a torch instance, the inputs is the input tensor for this model.

from fastbert.

Thanks! It worked!

from fastbert.

There is no a general code that can test the FLOPs of all models, because different models are implemented differently.

If you want to get the FLOPs of a model, you should add the following code to the models' script:

flops, params = profile(model, inputs, verbose=False)

the model is a torch instance, the inputs is the input tensor for this model.

Hi,

In your paper, is the total FLOPS of BERT 21785M? It looks very small.
Is thop capable of calculating FC layers?
I see that FLOPs depend on input sequence length. What is the FLOPs for one token?

Thanks,
ZK

from fastbert.

https://arxiv.org/pdf/2003.10555.pdf
It seems that this paper also represents the inference FLOPs of BERT-BASE as 29 BFLOPs.
so 21785M looked quite affordable for me.

from fastbert.