does the key masking work? about transformer HOT 6 CLOSED

I have the same question, I don't think the mask work!.

from transformer.

I tested the model, you are right, all of masking are not work! @Kyubyong

from transformer.

i found that there are two method implement position embedding，key_mask can work on PE which params are training among model，but not work on PE which the paper says，because the padding embeddings are not 0 ！who can fix it？ @Kyubyong

from transformer.

I alos found the same question. if the raw code are used, the mask doesn't work at all. I verify the code like this:
self.length_mask = tf.cast(tf.sequence_mask(length_batch, maxlen), tf.int32)
length_embedding = tf.Variable(tf.concat([tf.zeros(shape=(1, num_units)), tf.ones(shape=(maxlen-1, num_units))], 0), trainable=False)
self.length_mask_embedding = tf.nn.embedding_lookup(length_embedding, self.length_mask)
self.dec_position_embedding *= self.length_mask_embedding
self.dec += self.dec_position_embedding

where length_batch is the length of each sentence in the batch

from transformer.

I recently ran this model and found that it has not been running.
tqdm progressbar no running!

from transformer.

I alos found the same question. if the raw code are used, the mask doesn't work at all. I verify the code like this:
self.length_mask = tf.cast(tf.sequence_mask(length_batch, maxlen), tf.int32)
length_embedding = tf.Variable(tf.concat([tf.zeros(shape=(1, num_units)), tf.ones(shape=(maxlen-1, num_units))], 0), trainable=False)
self.length_mask_embedding = tf.nn.embedding_lookup(length_embedding, self.length_mask)
self.dec_position_embedding *= self.length_mask_embedding
self.dec += self.dec_position_embedding

where length_batch is the length of each sentence in the batch

in the length_embedding, the second embedding of one should be a shape[1, num_units]? not the max_len - 1

from transformer.