searobbersduck / elmo_chin Goto Github PK

Why do I get an error when performing training? MemoryError

h0
SHTERM: killing over-night session
Found 1 shards at ./data/example/*_seg_words.txt
Session closed, press ENTER to start, l to last session or q to exit.^[:q^[^[:q^C
Loaded 130432009 sentences.
Finished loading
Traceback (most recent call last):
File "train_elmo_try.py", line 67, in
main(args)
File "train_elmo_try.py", line 54, in main
shuffle_on_load=True)
File "/elmo_china/bilm/data.py", line 441, in init
shuffle_on_load=shuffle_on_load)
File "/elmo_china/bilm/data.py", line 341, in init
self._ids = self._load_random_shard()
File "/elmo_china/bilm/data.py", line 364, in _load_random_shard
ids = self._load_shard(shard_name)
File "/elmo_china/bilm/data.py", line 404, in _load_shard
return list(zip(ids, chars_ids))
MemoryError

Could you post some experiment performance?

FileNotFoundError: [Errno 2] No such file or directory: './log\options.json'' when try to run " train_with_single_core(options, data, tf_save_dir, tf_log_dir)" in train_elmo_try.py

do we need pre-defined a json in the directory?

i need pip install ?

ub16hp@UB16HP:/ub16_prj/ELMo_Chin/try$ PYTHONPATH=../ python3 train_elmo_try.py --save_dir ./log --vocab_file ../data/vocab_seg_words.txt --train_prefix '../data/example/*_seg_words.txt'
h0
Traceback (most recent call last):
File "train_elmo_try.py", line 67, in
main(args)
File "train_elmo_try.py", line 12, in main
vocab = load_vocab(args.vocab_file, 10)
File "/home/ub16hp/ub16_prj/ELMo_Chin/bilm/training.py", line 1416, in load_vocab
validate_file=True)
File "/home/ub16hp/ub16_prj/ELMo_Chin/bilm/data.py", line 113, in init
super(UnicodeCharsVocabulary, self).init(filename, **kwargs)
File "/home/ub16hp/ub16_prj/ELMo_Chin/bilm/data.py", line 48, in init
raise ValueError("Ensure the vocabulary file has "
ValueError: Ensure the vocabulary file has , , tokens
ub16hp@UB16HP:/ub16_prj/ELMo_Chin/try$

在中文语料中，字一般在2000-6000，词典的话基本破万，试了下ELMO，初步按频率过滤部分，字库大小5000+，词库7万+，模型训练下来直接 1个g，想和你请教下是不是哪里错了，或者需要优化的

018-10-22 14:31:06.246539: W tensorflow/core/framework/op_kernel.cc:1158] Invalid argument: You must feed a value for placeholder tensor 'lm/tokens_characters_reverse' with dtype int64 and shape [64,20,10]
[[Node: lm/tokens_characters_reverse = Placeholderdtype=DT_INT64, shape=[64,20,10], _device="/job:localhost/replica:0/task:0/cpu:0"]]
2018-10-22 14:31:06.246585: W tensorflow/core/framework/op_kernel.cc:1158] Invalid argument: You must feed a value for placeholder tensor 'lm/tokens_characters_reverse' with dtype int64 and shape [64,20,10]
[[Node: lm/tokens_characters_reverse = Placeholderdtype=DT_INT64, shape=[64,20,10], _device="/job:localhost/replica:0/task:0/cpu:0"]]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, You must feed a value for placeholder tensor 'lm/tokens_characters_reverse' with dtype int64 and shape [64,20,10]
[[Node: lm/tokens_characters_reverse = Placeholderdtype=DT_INT64, shape=[64,20,10], _device="/job:localhost/replica:0/task:0/cpu:0"]]
[[Node: lm/gradients/lm/sampled_softmax_loss_1/embedding_lookup_grad/Shape/_243 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_2658_lm/gradients/lm/sampled_softmax_loss_1/embedding_lookup_grad/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op 'lm/tokens_characters_reverse', defined at:
File "train_elmo_try.py", line 67, in
main(args)
File "train_elmo_try.py", line 57, in main
train_with_single_core(options, data, tf_save_dir, tf_log_dir)
File "/usr/local/lib/python3.6/dist-packages/bilm-0.1-py3.6.egg/bilm/training.py", line 1091, in train_with_single_core
model = LanguageModel(options, True)
File "/usr/local/lib/python3.6/dist-packages/bilm-0.1-py3.6.egg/bilm/training.py", line 71, in init
self._build()
File "/usr/local/lib/python3.6/dist-packages/bilm-0.1-py3.6.egg/bilm/training.py", line 334, in _build
self._build_word_char_embeddings()
File "/usr/local/lib/python3.6/dist-packages/bilm-0.1-py3.6.egg/bilm/training.py", line 165, in _build_word_char_embeddings
name='tokens_characters_reverse')
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py", line 1530, in placeholder
return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 1954, in _placeholder
name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1269, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'lm/tokens_characters_reverse' with dtype int64 and shape [64,20,10]
[[Node: lm/tokens_characters_reverse = Placeholderdtype=DT_INT64, shape=[64,20,10], _device="/job:localhost/replica:0/task:0/cpu:0"]]
[[Node: lm/gradients/lm/sampled_softmax_loss_1/embedding_lookup_grad/Shape/_243 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_2658_lm/gradients/lm/sampled_softmax_loss_1/embedding_lookup_grad/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]]

WARNING:tensorflow:Error encountered when serializing lstm_output_embeddings.
Type is unsupported, or the types of the items don't match field type in CollectionDef.
'list' object has no attribute 'name'
Training for 10 epochs and 37300 batches
WARNING:tensorflow:Error encountered when serializing lstm_output_embeddings.
Type is unsupported, or the types of the items don't match field type in CollectionDef.
'list' object has no attribute 'name'
Loading data from: ../data/example/303_seg_words.txt
Loaded 8183 sentences.
Finished loading
Loading data from: ../data/example/302_raw_seg_words.txt
Loaded 2785 sentences.
Finished loading
Batch 100, train_perplexity=737.8248
Total time: 44.099915504455566
Loading data from: ../data/examp