Hi, first of all thanks to <a class="user-mention notranslate" data-hovercard-type

Extract the attention weights about neuralcodesum HOT 3 CLOSED

wasiahmad commented on July 16, 2024

Extract the attention weights

from neuralcodesum.

Comments (3)

wasiahmad commented on July 16, 2024

QUESTION A

It is either 1 or 2. If you are using copy attention, then it is (1), otherwise, it is (2). You can check here.

QUESTION B

Yes, you can. You just need to update the code snippet as I referenced above.

from neuralcodesum.

MattePalte commented on July 16, 2024

Thanks a lot for the fast reply!

I checked and my model has copy_attn set to true, and I was extracting beam_attn, so I was getting the 'copy' attention, right?

I already performed the required modification in order to extract both attentions:

if self.copy_attn:
      out = copy_generator.forward(dec_out, attn["copy"], src_map)
      out = out.squeeze(1)
      # beam x batch_size x tgt_vocab
      out = unbottle(out.data)
      for b in range(out.size(0)):
          for bx in range(out.size(1)):
              if blank[bx]:
                  blank_b = torch.Tensor(blank[bx]).to(code_word_rep)
                  fill_b = torch.Tensor(fill[bx]).to(code_word_rep)
                  out[b, bx].index_add_(0, fill_b,
                                        out[b, bx].index_select(0, blank_b))
                  out[b, bx].index_fill_(0, blank_b, 1e-10)
      transformer_attention = unbottle(attn["std"].squeeze(1)) # CHANGE - CONSIDER THIS TRANSFORMER "REGULAR" ATTENTION (?)
      beam_attn = unbottle(attn["copy"].squeeze(1))  # CONSIDER THIS COPY ATTENTION (?)
  else:
      out = generator.forward(dec_out.squeeze(1))
      # beam x batch_size x tgt_vocab
      out = unbottle(f.softmax(out, dim=1))
      # beam x batch_size x tgt_vocab
      transformer_attention = None
      beam_attn = unbottle(attn["std"].squeeze(1))

I want to know if I can now consider the "std" that I extract as correct "regular" transformer attention and the beam attention (that I was already extracting before) as copy attention, correct?

Thanks in advance.

from neuralcodesum.

wasiahmad commented on July 16, 2024

According to my understanding, yes, you are right.

from neuralcodesum.

Recommend Projects