Different learning rate of different layers about transfer_learning_tutorial HOT 5 CLOSED

1. Do you mean the 'excluded' layers not restored from the checkpoint would have a faster learning rate? I don't think there will be a faster learning rate for these layers, since restoring from checkpoint mean there is simply a nicer set of weights that perform better than random weights. Otherwise, the variables are either trainable or non-trainable, and if they're trainable I guess they take the same amount of time. I'm not very sure if it is possible to set a different learning for different layers, or if there's a good rationale behind doing this.

2. The arg_scope is useful when you want to set certain parameters consistently throughout the model without having to repeat writing the same code again. The arg_scope is found here: https://github.com/tensorflow/models/blob/master/slim/nets/inception_resnet_v2.py

from transfer_learning_tutorial.

Hello,

Thank you very much for your responds. I agree with you. The reason I am wondering whether we can set different learning rate is because the layers restored from pre-trained model already has a nice set of parameters while the layers excluded (not restored from the checkpoint) do not. Thus, I suspect that the "excluded layers" need more training than the restored layers.

from transfer_learning_tutorial.

Yes indeed. The excluded layers will require some training before you can customize the model to your own use. However, I'm not sure if the excluded layers themselves each require a different learning rate.

from transfer_learning_tutorial.

Thanks a lot!

from transfer_learning_tutorial.

No problem :D

from transfer_learning_tutorial.