filters after training about filter-pruning-geometric-median HOT 4 CLOSED

Your observation is right. The operation of the gradient of FPGM is different from that of SFP.
For FPGM, the gradients of the pruned filters are zero, and for SFP, they are non-zero, refer to this link.

Several reasons for this:

1. The nonzero gradient of SFP would enlarge the model capacity, which is the main point of SFP. But the contribution of FPGM is about the filter selection criterion, so we would like the eliminate the effect of model capacity to compare fairly.

2. This kind of operation is also different from HFP (Please refer to HFP in my IJCAI paper). Although the selected filters are pruned and kept as zero, the connections of them remain.

Imagine the first case, we use 500 neurons to learn a training set for one epoch (This is HFP).
In the second case, we use 1000 neurons to learn and delete 500 of them. For every epoch, 500 neurons are different (This is SFP).
The third case, we use 1000 neurons to learn and delete 500 of them. For every epoch, 500 neurons are the same (This is something between SFP and HFP).

from filter-pruning-geometric-median.

Got it! Thank you~

from filter-pruning-geometric-median.

@he-y What if only change the filter selection method and make the selected filters different for every epoch?In other words, maintaining the advantages of both the model capability enlarge of SFP and the filter selection of FPGM?

from filter-pruning-geometric-median.

@he-y What if only change the filter selection method and make the selected filters different for every epoch?In other words, maintaining the advantages of both the model capability enlarge of SFP and the filter selection of FPGM?

I think the performance would be slightly better.
As discussed above, the original FPGM is "FPGM + half SFP". So the improvement of "FPGM + SFP" over "FPGM + half SFP" may be less than that of "SFP" over "HFP".

from filter-pruning-geometric-median.