The pytorch-plato from hwwancient

pytorch-plato's People

Contributors

Stargazers

Watchers

pytorch-plato's Issues

你好，plato.pt下载不了，可以在提供一个下载链接吗？谢谢

Questions about the calculation of the attention mask.

The calculation of the joint attention mask of source and target text seems have some mistakes.

    def _join_mask(self, mask1, mask2):
        """
        Merge source attention mask and target attention mask.
        合并后的整个mask矩阵可以分为四个部分：左上lu/右上ru/左下lb/右下rb

        @param : mask1 : source attention mask
        @type : Variable(shape: [batch_size, max_src_len, max_src_len])

        @param : mask1 : target attention mask
        @type : Variable(shape: [batch_size, max_tgt_len, max_tgt_len])
        """
        batch_size = mask1.shape[0]
        seq_len1 = mask1.shape[1]
        seq_len2 = mask2.shape[1]
        seq_len = seq_len1 + seq_len2

        mask_lu = mask1
        mask_ru = torch.ones(batch_size, seq_len1, seq_len2)

        if self.use_gpu:
            mask_ru = mask_ru.cuda()
        mask3 = mask2[:, :, :1].repeat(1, 1, seq_len1)
        mask4 = mask1[:, :1, :].repeat(1, seq_len2, 1)
        mask_lb = mask3 + mask4 - mask3 * mask4
        mask_rb = mask2

        mask_u = torch.cat([mask_lu, mask_ru], dim=2)
        mask_b = torch.cat([mask_lb, mask_rb], dim=2)
        mask = torch.cat([mask_u, mask_b], dim=1)
        return mask

I think the right upper part of this matrix should be zeros. but is using torch.ones().

and the calculation of the left bottom part should be mask_lb = mask3 * mask4, i suppose, to set all attention mask of [PAD] zeros.

By the way, thanks for your code. It's really structured clearly and helps me a lot :)