why loss_clf = F.cross_entropy(logits[:, self._known_classes:], fake_targets)? about pycil HOT 3 CLOSED

Typical finetuning loss should be the latter one you mentioned. We make some modifications to improve its performance.

But I think this does not influence much of the performance since finetuning is always the worst choice of incremental learning. Just replace it and have a try.

from pycil.

Good question. Just as we claimed in the README file

By default, weights corresponding to the outputs of previous classes are not updated.

This implementation is to avoid updating the prototypes of old categories. And therefore, the training of new classes will only corrupt the feature extractor while the prototypes are preserved.

from pycil.

But here is indeed a bug caused by weight decay that I ignored before. Since even the old prototypes are not used to calculate the final loss, the old prototypes will still be slightly changed by the weight decay. To understand this problem, run the following python script. I will correct the bug later. And if you have any insights into this bug, welcome to contribute to our repo.

a=nn.Linear(10,2)
optimizer=torch.optim.SGD(a.parameters(),lr=0.1,weight_decay=5e-4)
for i in range(1000):
    optimizer.zero_grad()
    output=a(torch.randn(10))
    output[0].backward()
    optimizer.step()
    print(a.bias.data)

from pycil.