Fastformers/Transformers question about fastformers HOT 4 CLOSED

Hi @jmwoloso.

Thanks for your interest in FastFormers! There's one assumption that is not valid for head pruned attention modules. (hidden_size != head_size * num_heads, because head_size is fixed and some heads are pruned. So, num_heads is now smaller than the original value.). There seem to be a few places in HF transformers which rely on this assumption.

If you could cherry-pick the changes in modeling_bert.py and configuration_bert.py from the PR in HR repo, you might be able to execute the fastformers example script.

Hope this answers your question. And, please let me know if you have any other questions. Thanks!

from fastformers.

Thanks for the reply @ykim362! That's pretty interesting. I'm still reading through the paper, but is that the only real hangup preventing your work from being merged to Transformers?

from fastformers.

@jmwoloso that was a major blocking issue. The other things are more about pipelining of datasets based on applications which also need some work. But, those are more related to the applications.

from fastformers.

ok, thank you for the detail @ykim362!

from fastformers.