Training, validation, and testing of models, and Predicting using a model about returnn HOT 13 CLOSED

1. during training, after each epoch, in the log file you will find a line similar to this:
   epoch 1 score: 2.65181904453 elapsed: 1:33:08 dev: score 1.39069169463 error 0.369806780381
   the number after error shows the label error rate on the validation ("dev") set. The train error is currently not printed I think

the config_fwd is not about "validating the model", but it will use the model to produce outputs for the validation data and write it to a file (in this case to "output_file": "mdlstm_real_valid.h5" ).
You can then view the posteriors by using
h5dump mdlstm_real_valid.h5

Using config_fwd, you can produce posteriors for the specified input file (here "eval": "features/raw/valid.h5") as a hdf file. This hdf file can then be decoded e.g. by RASR, Kaldi or a simple self written decoder

from returnn.

I did find a line similar to that in log file.

When do I get to use the test set (test.h5)? :O

from returnn.

the usual procedure is to forward on the testset (using config_fwd and changing the "eval" entry to "eval": "features/raw/valid.h5" and the "output_file" entry to, say "output_file": "mdlstm_real_test.h5")
Then you can use the posteriors from mdlstm_real_test.h5 for decoding

from returnn.

Do the scores mean "accuracy" of the model? :O

from returnn.

Score in this case is the CTC loss function

from returnn.

How do I use the model to recognize the handwriting of a single image that contains the handwritten text? :O

from returnn.

first you create a hdf 5 file containing the data for the image. For this, you can adapt the script https://github.com/rwth-i6/returnn/blob/master/demos/mdlstm/IAM/create_IAM_dataset.py or you might also start from here https://github.com/rwth-i6/returnn/blob/master/demos/mdlstm/artificial/create_test_h5.py and replace the artificial data with your image.

Afterwards you forward your trained model on this data, which gives you a hdf5 file with posteriors for the image.

Then you run a decoder (e.g. Kaldi, RASR).

It might also be possible in a easier way to run the best path decoder using "task": "daemon". For a rough explanation, please view this issue: #3

from returnn.

Thanks for the reply. So far here are the steps I've done.

1. I've created an hdf 5 file containing the data for the image using this code I've adapted from create_IAM_dataset.py.

2. I created a config file to be used in the decoding using "task": "daemon"

and I got this on the log when I ran the script python ... : httpd listening on port 3333

3. I opened a terminal and tried out this command curl -X POST http://localhost:3333/classify -H "Content-Type: application/json" -d '{"data":[[-0.7, 0.98],[0.62, 1.3]], "classes" : [0,0]}'

and what I got is this:

httpd listening on port 3333 classifier started: 27a40758fcbf5efd7418bfedb60fc4b8bd5b9d80 starting task extract running 1 sequence slices (2 nts) of batch 0 on device gpu0 Device gpuX proc exception: total size of new array must be unchanged Apply node that caused the error: Reshape{2}(y_sizes, MakeVector{dtype='int64'}.0) Toposort index: 111 Inputs types: [TensorType(int32, matrix), TensorType(int64, vector)] Inputs shapes: [(1, 1), (2,)] Inputs strides: [(4, 4), (8,)] Inputs values: [array([[-1]], dtype=int32), array([2, 0])] Outputs clients: [[InplaceDimShuffle{1,0}(Reshape{2}.0)]]

I have a few questions on "task": "daemon"

1. Can the classification be done without having to connect to a localhost?
2. What do I do with the hash?
3. What should I expect when I run the task?

Please let me know if I'm not being clear enough with my questions.

from returnn.

I'm not very familiar with the daemon.
@doetsch can you comment on this?

from returnn.

Let me just clarify what I understood about the steps you mention in recognizing the handwriting in a single image.

1. I "convert" the image first into an hdf5 file to serve as the input for the model.
2. I perform forwarding using config_fwd with task forward on this data to produce the posteriors.
3. I decode these posteriors using Kaldi or RASR thus giving me the recognized text.

or
2. I perform forwarding using config_fwd with task daemon on this data to produce the posteriors which are going to be decoded. This will give me the recognized text.

And here's how I see how it goes:

1. An image containing the words handwritten words "I like trains" is to be converted into an hdf5 to serve as an input for the model.
2. Perform forwarding on this input with task forward to produce the posteriors.
3. I decode these posteriors using Kaldi or RASR thus giving me the recognized text "I like trains".

or
2. I perform forwarding using config_fwd with task daemon on this data to produce the posteriors which are going to be decoded. This will give me the recognized text "I like trains".

Did I get it right? :O

from returnn.

Yes, looks right to me.

One thing to keep in mind is that the daemon is an easy solution to quickly get a result, but decoding like this is not as powerful as using RASR or Kaldi, since the daemon will just do a best path decoding without lexicon and language model.

from returnn.

Am I correct in saying that the BestPathDecoder returns the recognized text? If so, is it possible to just use the decoder without having to use daemon? :o

from returnn.

@pvoigtlaender thank you for your explanation.

"
Afterwards you forward your trained model on this data, which gives you a hdf5 file with posteriors for the image.

Then you run a decoder (e.g. Kaldi, RASR).

It might also be possible in a easier way to run the best path decoder using "task": "daemon".

"

what do you mean by using a decoder Kaldi, RASR, is there any example code to decode the posterior file test.hd5?
BTW, using "task":"daemon" is a kind of decoder for decoding the posterior file?

Thank you again~

from returnn.