How to pruning the residual block without the 1x1 conv projection shortcut? about dnn-pruning HOT 12 CLOSED

As you may know, Mr. Li mentions in his paper that he determines the pruning ratio (the number of filters to prune from each layer) by simple experimentation. However the ratio he defines might not optimal for new models or slighly different version of the same model. So I defined a simple schema to estimate how many filters can your prune from each layer without braking the model. This the summary of the models I have managed to prune.

The paper is not out yet T.T
I willl try upload an updated version of the code.

from dnn-pruning.

To the best of my understanding, is simply not possible to prune those residual blocks. The reason is exactly what you mention:

"The channels of the residual branch and the main branch might be different, and thus F(x) and the residual x cannot be added directly"

I this is a limitation that residual models have.

from dnn-pruning.

@slothkong
It would be a little disappointing that we cannot apply this excellent idea to prune ResNet.
Anyway, thanks very much for your valuable opinion and discussion.

from dnn-pruning.

Hi, @ganji15

I have the same problem when I try to prune the ResNet. I was wondering did you prune the ResNet successfully?

Thanks.

from dnn-pruning.

Hi, @ZhuweiQin

In the paper "Pruning Filters for Efficient ConvNets", the author mentioned that

"Since there is no projection mapping for choosing the identity feature maps, we only consider pruning the first layer of the residual block."

However, this compromise limits the prune effecitivness for ResNet.

from dnn-pruning.

@ganji15 Yeah, I saw that. So, did you try to prune the first layer of the residual block?

from dnn-pruning.

@ZhuweiQin, here is a summary of how I pruned resnet56. To the left is the pruning ratio defined in Mr. Li's paper and to the right, the one that I used.

from dnn-pruning.

@slothkong Thanks for sharing your experiment result. That‘s amazing. Your work prune more filters than Mr. Li's paper.
How about the accuracy drop under this pruning ratio? Does this figure come from your published paper? Would mind share with us?
I will try to implement the ResNet pruning, and really hope you could keep updating this repository. It's really helpful.
Thanks~

from dnn-pruning.

@ZhuweiQin, I have pruned ResNet18 but the results are not satisfactory compared with VGG16.

@slothkong, I still think it is not a good idea to prune filters of the ResNet due to its complex structure. On the other hand, it would be greatly helpful if you would like to share your implementations.

from dnn-pruning.

@slothkong, really impressive results! I am waiting for your good news! ;-)

from dnn-pruning.

Hi, @slothkong How is your paper? I was wondering if you could share your implementations.

from dnn-pruning.

sorry for the late replay. I'm afraid the paper will be coming any time soon (maybe never). But I plan to open source the full code with which I obtained those results in the table. Unfortunately I no longer have the .caffemodel files but I think can provide the .prototxt and the training configurations for you guys to train the models on your own. Please give me at least one week. I need to write the documentation

from dnn-pruning.