"reload = true" behavior ? about dl4mt-tutorial HOT 4 CLOSED

Just to illustrate what I wrote above.
I measure the BLEU score with a test set belonging to the training data (working on a task specific model). After the first 3 days, BLEU was 60 then continued to increase to 67 on day 6.
I stopped it. restarted it with "reload". 3 hours after the restart BLEU went down to 41.

from dl4mt-tutorial.

@vince62s the current checkpointing is rather primitive, in the sense that we only reload the model parameters. But they are not the only elements that define the state of training. We also need to reload (hence save) the accumulators/auxiliaries of the step rule that we are using (e.g. moments, iteration number for adam). Another point is the iterator state, everytime you restart training, data iterator restarts from the beginning of the dataset, introducing some bias for those examples.

We will probably work on it soon, or point a reference implementation but please feel free to make a PR about it which will be appreciated.

from dl4mt-tutorial.

ok, so just to make it clear, the reload will only teka the parameters from the model.npz.pkl file but the model itself model.npz from the previous job is not reloaded, hence rewritten with the new job from scratch ?

from dl4mt-tutorial.

What i meant by the model is model.npz and it is loaded when you set reload=True, model.npz.pkl is just for the options.

from dl4mt-tutorial.