"key_padding_mask" in attention mechanism not be implemented? about akt HOT 2 CLOSED

Hi huajian123. Thank you for your interest. In L204, i defined an upper triangular matrix of ones, in L205 mask is inverted (so lower triangular matrix of ones). in L305, I set values to -inf to all future items (including padded indices). However, for computing prediction at those padded time indices, it computes attention over possibly padded indices. But it does not matter since, prediction loss does not flow through those padded timesteps. Hope my explanation helps.

from akt.

Okay, arghosh, thank you for your detailed explanation. It may because that the padded values is in the right of the sequence, and use the upper triangular matrix could ignore the padded index. Well, thanks again. : )

from akt.