Split decoder about neuralcodesum HOT 1 CLOSED

Yes, the split decoder mechanism is implemented, however, I found a mistake that is causing the error. The error is due to copier_out being a list of Tensor, instead of a Tensor. As you can see here, TransformerDecoder returns a list of representations where each item is the output from each layer of the TransformerDecoder. So, to make the split decoder mechanism work, you need to make 2 changes.

First, in the following line,

f_t = self.fusion_sigmoid(torch.cat([copier_out, dec_out], dim=-1))

copier_out and dec_out are both lists. So, modify the line as follows.

f_t = self.fusion_sigmoid(torch.cat([copier_out[-1], dec_out[-1]], dim=-1))            

Second,

When split decoder mechanism is enabled, decoder_outputs [as in here] is no longer a list of Tensor, it is only a Tensor. Therefore, to cope with the statements 1 and 2, you can simply do:

decoder_outputs = self.fusion_gate(gate_input)
decoder_outputs = [decoder_outputs]

Hopefully, this would work but please verify it. A pull request is welcomed.

from neuralcodesum.