label smoothing inf err about annotated-transformer HOT 2 OPEN

Hi jerett - we need the inputs of KLDivLoss to be in log space. Hence we need to apply log() - The -inf issue is because we have zeros in the tensor. So the log() applied to the predict tensor is creating issue with LabelSmoothing().

Hence I propose to use softmax_log() instead of log()

I also raised a pr.

Thanks
Satya

from annotated-transformer.

Same problem here, but i don't think we should use softmax_log instead of log because the predict is already defined as probabilities.

Rather, I changed the predict tensor to:

predict = torch.FloatTensor([[1e-9, x/d - 1e-9, 1/d, 1/d, 1/d]])

to avoid the inf.

The result I get is the same to the example provided:

from annotated-transformer.