A concern about the loss function. about im-net-pytorch HOT 6 CLOSED

The leaky clamp operation looks like this. As you see, the problem you described does not exist.

Personally, I prefer piece-wise linear activation functions, which is why I usually use leaky ReLU or leaky clamp. Since I use leaky clamp in the output layer (instead of sigmoid), there will be negative values in the outputs, therefore BCE cannot be applied.

from im-net-pytorch.

Oh, sorry~ I just confused loss and gradient.
The gradient may be greater when the prediction is closer to 1. Emm...It seems not a big problem, right?

Thanks for your reply.

from im-net-pytorch.

When computing the gradient, you need to consider the ground truth. The gradients you showed on the figure seem to assume the ground truth is 1.

It is actually better to consider the gradient as the product of the gradient from MSE and the constant gradient (1 or 0.01) from this piece-wise linear function.

from im-net-pytorch.

Yes, you'r right. Because negative samples are much more than the positive samples, I guess the network would struggle to learn the positive samples. So I discussed about the positive samples. And I wonder whether the imbalance between positive and negative samples is the main reason for which we need train the network so long.

from im-net-pytorch.

I would consider the network structure as the main reason, i.e., the continuity/smoothness of the functions represented by MLPs.

from im-net-pytorch.

Ok. Thanks for your patient :)

from im-net-pytorch.