Cost going negative after epoch 12 about returnn HOT 5 CLOSED

Hi,

I think that the cost is not supposed to become negative, not sure why this happens.
It could be that this points to a problem, or maybe it's just some unimportant thing like a wrong scale of the loss values.
Considering that the accuracy still increases, it might be that it will just work fine. It would be good if you can try to make a recognition with this model, and see if it works as usual.

from returnn.

What loss is that? Depending on the loss, it sure can become negative. For example, I think the CTC loss can become negative.

from returnn.

Its CTC loss that I am using. I have not made any changes to the loss function (using the same as given with returnn library). I will update the final results once the training stops...So far accuracy improving continuously(surprising!) and loss going more negative.

from returnn.

The CTC loss (Eq. 12 in ftp://ftp.idsia.ch/pub/juergen/icml2006.pdf ) cannot become negative: each summand is ln(p), where 0 <= p <= 1 which means that -infty <= ln(p) <= 0 which means the negative sum cannot become negative in theory. Don't know why it's happening here anyway, could be a bug. First let's see if you get good results anyway.

from returnn.

Hello, Thanks for the reply. I figures out that there was one character with single instance in the dataset. I guess, this was causing trouble. Once I remove that line image and took the revised character set, it is giving the desired behavior. I didnt expect the super-high imbalance in classes can result in such a disaster :(

Thanks!

from returnn.