MMTL training accuracies swinging between two datasets about metal HOT 3 CLOSED

Hmm, hard to say without more information. It's possible that your batches are not randomly shuffled somehow (by default they should be). But if your model was seeing all batches of one task followed by all of the other, that could explain the oscillating effect you described. Have you also tried lowering the learning rate to see if that calms things down?

from metal.

So I managed to fix this problem by changing the optimiser config to this:

optimizer_config = {
        "optimizer": "sgd",
        "optimizer_common": {"lr": 0.005},
        "sgd_config": {"momentum": 0.01},
    }

Which has meant that my datasets' dev accuracies both peak within a few epochs at a level that I would expect.
I am currently under the assumption that this was a problem with my datasets, instead of with the implementation of the default Adam optimiser. I would recommend this fix to anyone else with this problem working on obscure datasets as it has worked well for me in this case.

from metal.

Thanks for sharing! Glad you got it working for your problem.

from metal.