Cannot resume training without quality loss about mellotron HOT 5 CLOSED

@AndroYD84 try changing the hyperparameter before resuming from checkpoint: ignore_layers=[] and use_saved_learning_rate=True

from mellotron.

By default a checkpoint is saved every 500 iterations.

from mellotron.

Yes, but that's not the issue, imagine I trained for weeks up to 300.400 iterations and a black out happened, I'd lose only 400 iterations of progress but still have a "checkpoint_300000" file, is it possible to resume training from this checkpoint? Any attempt I made to resume from a checkpoint have generated a model that sounded much worse than its predecessor (checkpoint_300000" file), I know sometimes resuming a training requires some warming up before returning to its original state, but this isn't happening after a week, the results are not even close to its predecessor, if I had a time machine and could have prevented the blackout now the new checkpoint (ie. checkpoint_400000) would have sounded better not worse than before, do I have to start over again from scratch and lose weeks of training or I did something wrong? Thanks for your patience.

from mellotron.

Note that when using --warm_start does not include the optimizer.
When resuming from your own model. you should not include --warm_start.

from mellotron.

Thank you for your help, I resumed training without the "warm_start" option and I confirm that so far I haven't noticed any quality loss, I haven't tried texpomru13 solution as the results were already improving without the need to change anything.
However, if at some point I notice that the model is not improving any further then I plan to test the other solution as well, but right now I didn't want to jinx it.

from mellotron.