The error of training an RNNsearch model about thumt HOT 6 CLOSED

The num_hidden_layers is a hyper-parameter specific to seq2seq architecture. This error should not be happened unless you have renamed seq2seq.json to rnnsearch.json. I think you should delete the train directory and try the command again.

from thumt.

I just downloaded all of the THUMT packages from GitHub and then tried to reproduce your experiments in accordance with the user manual. The training files were generated automatically after running the code in the user manual 3.2.1. I did not rename any files. Just after running the code in 3.2.2 and then running the code in 3.2.2, there was an error. I use python2.7.0, Tensorflow1.6.0, is the version I use wrong?

from thumt.

That's weird. I have tested the latest commit of THUMT and do not found this problem. Have you tried deleting train directory and re-run the command?

from thumt.

Hi, thanks for your reply!

I tried deleting train directory and re-run the command, but now I get this error:

2018-05-01 15:30:16.636478: W tensorflow/core/common_runtime/bfc_allocator.cc:279] **************************************************************************************************xx 2018-05-01 15:30:16.647710: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS 2018-05-01 15:30:16.647734: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1 2018-05-01 15:30:16.647724: E tensorflow/core/common_runtime/bfc_allocator.cc:381] tried to deallocate nullptr 2018-05-01 15:30:16.647755: E tensorflow/core/common_runtime/bfc_allocator.cc:381] tried to deallocate nullptr Aborted (core dumped)
nvidia driver version: 390.30
CUDA Version 9.0.176
cudnn7_7.1.3.16
gcc version 5.4.0

Should i ignore this？

from thumt.

It seems that you have run out of GPU memory. Try to reduce the batch_size hyper-parameter.

from thumt.

Thanks very much， it finally works！

from thumt.