some parameters appear in more than one parameter group about finetuning-scheduler HOT 6 CLOSED

A new FTS patch release (0.2.1) is out now including several of the potential improvements you helped elucidate, thanks @quancs! I think the new release (specifically 0a3b2b8, 3b33c7f, aa6399e) should close this issue now but feel free to open other issues as necessary or reach out anytime. Thanks again for your valuable feedback and good luck with your future research!

from finetuning-scheduler.

Glad to see you trying out Fine-Tuning Scheduler!

The root of the issue you describe is that all parameters are being passed in the creation of the optimizer, including those that were set to be frozen by FTS in accordance with the schedule (i.e. 'layer2' params were configured to be frozen but still passed to the initial optimizer). While instantiation of the optimizer is left the user, your issue has led me realize FTS could be improved in by issuing a warning to the user in cases like this where parameters with requires_grad=False are included in the initial optimizer instantiation but also referenced in subsequent training phases. In the meantime, the issue above could be addressed with a requires_grad filter applied to the optimizer instantiation like so:

    def configure_optimizers(self):
        # optimizer = torch.optim.Adam(self.parameters(), lr=0.1)
        parameters = filter(lambda x: x.requires_grad, self.parameters())
        optimizer = torch.optim.Adam(parameters, lr=0.1)
        ...

BTW, I also noticed in the example above you're using a scheduler that's not a subclass of torch.optim._LRScheduler. Though the current release of FTS indicates this scheduler requirement in the lr scheduler reinitialization documentation section, the check for this requirement is only done for schedules using lr reinitialization. For the next FTS patch release, I'll include a fix to mention the scheduler requirement in the primary FTS documentation (not just the lr reinitialization section) and to check for supported scheduler use in the on_fit_start callback hook. In the meantime, you can use any of the other lr schedulers that are supported by FTS, so for instance, this will work:

        ...
        # case 1: with scheduler
        # self.scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, factor=0.5, patience=5)
        self.scheduler = torch.optim.lr_scheduler.StepLR(optimizer=optimizer, step_size=1, gamma=0.7)
        ...

Thanks so much for the valuable feedback! Your usage elucidated two ways in which FTS could be improved and I think commits addressing both will be included in the next patch release.

I hope you find FTS somewhat helpful in your research and I'm always happy to help answer any questions... love the increasingly collaborative and open-source nature of the ML research community! Feel free to reach out anytime, I love hearing if and how researchers are using FTS in their experimentation. Wish you all the best with your future work!

from finetuning-scheduler.

Thank you for your replies.

According to you advice, I changed the configure_optimizers to:

    def configure_optimizers(self):
        parameters = filter(lambda x: x.requires_grad, self.parameters())
        optimizer = torch.optim.Adam(parameters, lr=0.1)
        # case 1: with scheduler
        self.scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, factor=0.5, patience=5)
        self.scheduler.base_lrs = []
        return {
            'optimizer': optimizer,
            'lr_scheduler': {
                'scheduler': self.scheduler,
                'monitor': 'valid_loss',
            }
        }

where, self.scheduler.base_lrs = [] is to used to suppress the error 'ReduceLROnPlateau' object has no attribute 'base_lrs' . I know it must have some side effects like the lrs specified in the config will not work.

The reason I want to use ReduceLROnPlateau is that the paper I'm trying to compare with (on a new dataset which is not the one used in that paper) uses this lr_scheduler. Changing ReduceLROnPlateau to other lr_schedulers will cause the reproducing imperfect. So, is there any way to support ReduceLROnPlateau and any other user-defined lr_schedulers? For example, give the user a way to know the current phase number, and then let the user to adjust their lr_scheduler by themself. Or, let the user provide a function which accepts the lr specified in config and their lr_scheduler instance, and executing the function will set the lr_scheduler properly.

It would be much better to have a way to support that. I willing to help if there is anything I can do.

from finetuning-scheduler.

Totally understand the reproduciblity requirement and agree 100% on the need for ReduceLROnPlateau support.

ReduceLROnPlateau wasn't supported in the first minor release of FTS because of the non-trivial incremental complexity in supporting its divergent configuration in the context of both implicit and explicit mode fine-tuning (it's different from the _LRSchedulers in some important ways). While one could subclass _LRScheduler (like is done for a custom LambdaLR example here) and replicate the functionality of ReduceLROnPlateau that's not the right solution and I wouldn't ask you to do that 🤣

Instead, I've experimented today with implementing support for ReduceLROnPlateau directly and I think it'll be doable without adding too much complexity. I've got a few things going in parallel but I'll try to bang out beta support ASAP and let you know as soon as it's pushed (it should land in the next patch release not too long after that).

As far as granting the user the flexibility to set lrs arbitrarily in different phases, that should be achievable in both implicit and explicit fine-tuning scheduler modes using one or more LambdaLR schedulers. You can always both specify the desired scheduler for a given phase and the initial lr (or initial lr for each individual parameter group in explicit fine-tuning mode).

Thanks again for your valuable feedback, I'm definitely open and eager to hear suggestions on improvements and PRs are always welcome! Thanks again for your thoughts, your feedback is already improving FTS in a couple ways @quancs!

from finetuning-scheduler.

Great! I will try the next release as soon as it comes.

from finetuning-scheduler.

Nice, I will try it.

from finetuning-scheduler.