start decay after resume without "-continue" about opennmt HOT 6 CLOSED

The learning rate is only updated at the end of an epoch. So it will complete epoch 6 first, then a first decay will be applied.

from opennmt.

well if I specify start_epoch 6 end_epoch 9 start_decay 5 decay_rate 0.5
without continue it should decay right away, no ?
Otherwise we have to manually adapt the initial learning rate.

from opennmt.

Why are you not using -continue in this case?

I believe the current behavior is correct. For example if -start_decay 2 -start_epoch 6 are set, do you expect the code to replay to entire decay history?

from opennmt.

then to avoid confusion, if continue is not set and if you load from an existing model, I f would suggest we throw an error if start_decay_at if less or equal to start_epoch.
if you reset the lr and decay, I don't exactly undertstand the possibility to start from epoch X when load an existing model. do you see my point?

from opennmt.

Unless I'm missing something, that is reason we introduced the -continue option, to continue exactly where a checkpoint left off.

When you don't use -continue, it is actually a new training which uses the parameters from a checkpoint independently from their optimization history. It has other use cases like if you want to change the data and set a higher learning rate or change the optimization method.

from opennmt.

ok best to discuss this on the forum.

from opennmt.