Question for CTC decoding about crnn HOT 8 CLOSED

@Qhuangcn: see my repo CTCDecoder.

from crnn.

@chengzhanzhan I have also observed the same phenomenon. Perhaps it's not a bug, but a strategy learned by LSTM. Notice that the CTC loss function does not care the position of individual letters, but only the final recognition results.

from crnn.

@bgshih I test 38k images for this phenomenon. Because of the convolution operation, a section of continuous pixels are represented by a shorter interval, which may results in the first position including some information of the first character.

Then I conduct experiments based on the
setting 1: 3 * 256 * 32 (channel * width * height) image ==> convolution layers ==> 128 * 65* 1 ==> LSTM layers ==> CTC. Here, each column (of 65) represent several columns of origin image.

setting 2: 3 * 256 * 32 (channel * width * height) image ==> convolution layers ==> 128 * 225* 1 ==> LSTM layers ==> CTC to make sure that each step (of 225) represents each column of the image.

A example is as follow:

given image:

setting 1 result:
3-----------22----6----22-----b----------------------------------
shows that the first position is character 3.

setting 2 result:
-----33------------------2----------------------6-------------------22-------------------------b---------------------------------------------------------------------------------------------------------------------------------
shows that the first several positions are blanks.

In CTC, each peak represents corresponding character, otherwise represent blank. Therefore, I think that the convolution strategy results in the phenomenon and CTC can index the probable position of each character.

from crnn.

@chengzhanzhan just want to let you know that I've observed the same phenomena with a different model, different framework (TensorFlow) and different input data (IAM). The last character is always predicted at the last time-step of the RNN output sequence. All other character-predictions are roughly aligned with the characters in the image. Look at the attached image, where the probability of the characters over time is plotted in the bottom-most graph: the "i", the "l"s and "t"s are aligned with the image while the final "e" is not.

from crnn.

@githubharald
Thankyou for your experimtent, do you figure out the reason caused this phenomena?

from crnn.

no.

from crnn.

@githubharald Hello, can you tell me how to calculate the character-level probability in ctc-beam search? Is there a tutorial of implementing that in tensorflow?

thank you

from crnn.

does anyone research on ctc of this phenomenon, for my model it shows the same problem as this. for example a blank image input to this, it sometime decode something instead of blank only output...... This is a serious problem on OCR application.
Let say I have 4 fields have to recognition, and 1 field is optional to fill in on the document, it should be able to recognition as blank image as blank output. if not the recognition accuracy within a document will be affected

from crnn.