[Question] Map attention weights to original input text instead of tokenized input about transformers-interpret HOT 1 CLOSED

Hi @hardianlawi, this is an interesting question. The short answer is that I think what you would like to do is possible. While the tokenized text is not super UX friendly it's important to remember that that tokenized text IS what the model is fed as input so attribution with respect to the tokens is for me the best way to try and represent the model's behaviour.

Bert like models tend to use the wordpiece method for tokenization and there are times in the explainer when you can see a model giving a negative attribution to the starting word piece and positive attribution to the negative wordpiece. For researchers and ML users interpreting their model's this could be quite useful information. To get rid of this information would be detrimental to the true interpretation of what is happening with the model.

I don't personally think I will implement a feature for mapping the attributions back to the exact text, or at least I'll have to think on it for a bit, but if you were to do this yourself you would just need to write some logic that could identify certain when a wordpiece occurs e.g. ["gp", "##u"] and then you could average the two raw attributions for gp and ##u together for a single attribution score, this will then correspond back to the original text.

Hope this helps.

from transformers-interpret.