NaN with modified attention about dl4mt-tutorial HOT 5 CLOSED

@sebastien-j i dont think this issue is related with #29 (that is just a problem for out-of-sync nmt.py after pr #15 - i've synced them here but not yet made a pr).
Not in this repo, but i've adapted this modified GRU to multi-way translation and it was performing slightly better. From what i've experimented, NaNs arise with Adam having a learning rate bigger than 1e-3.

from dl4mt-tutorial.

Thanks @orhanf.

I'm using adadelta. After all, the NaN issue may not be related to the different GRU variant as it just crashed with the previous version. I'll try different datasets/preprocessing to try to understand what is going on.

from dl4mt-tutorial.

One more thing that comes to my mind is stabilizing the attention softmax, this enhancement. We were getting NaNs because of that instability (in blocks implementation tho)

from dl4mt-tutorial.

This is strange. Out of 2 computers I tried, the bug only occurs on one. The Theano versions are identical, and I added some code to allow choosing a RNG.

In any case, it appears that the problem is not directly related to the dl4mt code.

from dl4mt-tutorial.

I have a same problem with NANs. Using smaller batch_size(=32) solved it.

from dl4mt-tutorial.