erickrf / autoencoder Goto Github PK

Text autoencoder with LSTMs

License: MIT License

Python 100.00%

autoencoder's Introduction

Text Autoencoder

This is an implementation of a recurrent neural network that reads an input text, encodes it in its memory cell, and then reconstructs the inputs. This is basically the idea presented by Sutskever et al. (2014)

Why? The point of training an autoencoder is to make an RNN learn how to compress a relatively long sequence into a limited, dense vector. Once we have a fixed-size representation of a sentence, there's a lot we can do with it.

We can work with single sentences (classifying them with respect to sentiment, topic, authorship, etc), or more than one at a time (checking for similarities, contradiction, question/answer pairs, etc.) Another successful application is to encode one sentence in one language and use a different autoencoder to decode it into another language, e.g. Cho et al. (2014).

Implementation

The autoencoder is implemented with Tensorflow. Specifically, it uses a bidirectional LSTM (but it can be configured to use a simple LSTM instead).

In the encoder step, the LSTM reads the whole input sequence; its outputs at each time step are ignored.

Then, in the decoder step, a special symbol GO is read, and the output of the LSTM is fed to a linear layer with the size of the vocabulary. The chosen word (i.e., the one with the highest score) is the next input to the decoder. This goes on until a special symbol EOS is produced.

The weights of the encoder and decoder are shared.

Performance notes

Even for small vocabularies (a few thousand words), training the network over all possible outputs at each time step is very expensive computationally. Instead, we just sample the weights of 100 possible words. During inference time, there is no way around it, but the computational cost is much lesser.
For better decoder performance, a beam search is preferable to the currently used greedy choice.

Scripts

prepare-data.py: reads a text file and create numpy files that can be used to train an autoencoder
train-autoencoder.py: train a new autoencoder model
interactive.py: run a trained autoencoder that reads input from stdin. It can be fun to test the boundaries of your trained model :)
codify-sentences.py: run the encoder part of a trained autoencoder on sentences read from a text file. The encoded representation is saved as a numpy file

You can run any of the scripts with -h to get information about what arguments they accept.

autoencoder's People

Contributors

Stargazers

Watchers

Forkers

hardikmeisheri chriscummins xriver007 dilip-dmk wangyannhao deeppay akbari59 xapcloud chapzq77 elani0 zencoding mohitpokharna mvpcom zilongzhong girijaravishankar linyia01 calculatedcontent chenmoshushi essalj junhaowang roger-zhe-li summatic tedio85 aman229 kevinadda deepaklabh neil535 nicolovendramin llliuzhen movinghera crizzy9 andreymerz chavesliu nguyenanhtien abiraja2004 haofengrushui204 grseb9s ziboyi aracelimanzanoch heibaipei raymonddixon summerqing sunxm2357 jinzhilin auserj lijuan-su ledihdug youlei5898 parthtandel mimir-head jamesgu14 autowonderman nicolalandro libin19861023 geetanshu-m leefree-git ychervonyi chengjingfeng lyzl2010 subbaraomanchala 1601120453 cswangyuhui raftliu ozydingo agiant greyonee leenftr thntran rakheshkumbi eminemrain isuruboyagane15 castrol68 ecodev-cloud szprestonhuang vraebfdsb leiqi slzbywdf bijoyboban7 yolandazeng1004 richard-gangan namratarshivagunde sayannag akank139 physicphy klassikcat gaojie-wang bnucsy lathamaheswari-kandasamy akashmavle5 tonylibing elijahahianyo

autoencoder's Issues

tensorflow.python.framework.errors_impl.InvalidArgumentError: Sampler's range is too small.

Hi, I'm having this problem while running (vocabulary.txt train-data.npz valid-data.npz are generated by prepare-data.py):
train-autoencoder.py output_directory vocabulary.txt train-data.npz valid-data.npz

autoencoder/embeddings:0: (23, 300) (6900 parameters)
autoencoder/fw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/fw/lstm_cell/bias:0: (2000,) (2000 parameters)
autoencoder/bw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/bw/lstm_cell/bias:0: (2000,) (2000 parameters)
projection/dense/kernel:0: (1000, 23) (23000 parameters)
projection/dense/bias:0: (23,) (23 parameters)
Total: 3233923 parameters
INFO:root:Initialized the model and all variables. Starting training.
Traceback (most recent call last):
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Sampler's range is too small.
[[{{node sampled_softmax_loss/LogUniformCandidateSampler}} = LogUniformCandidateSamplernum_sampled=100, num_true=1, range_max=23, seed=0, seed2=0, unique=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\fanta\Desktop\autoencoder-master\src\train-autoencoder.py", line 113, in
args.dropout_keep, 5.0, report_interval=args.interval)
File "C:\Users\fanta\Desktop\autoencoder-master\src\autoencoder.py", line 257, in train
_, loss = session.run([self.train_op, self.loss], feeds)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Sampler's range is too small.
[[node sampled_softmax_loss/LogUniformCandidateSampler (defined at C:\Users\fanta\Desktop\autoencoder-master\src\autoencoder.py:194) = LogUniformCandidateSamplernum_sampled=100, num_true=1, range_max=23, seed=0, seed2=0, unique=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'sampled_softmax_loss/LogUniformCandidateSampler', defined at:
File "C:\Users\fanta\Desktop\autoencoder-master\src\train-autoencoder.py", line 106, in
bidirectional=args.bidirectional)
File "C:\Users\fanta\Desktop\autoencoder-master\src\autoencoder.py", line 163, in init
self._create_training_tensors()
File "C:\Users\fanta\Desktop\autoencoder-master\src\autoencoder.py", line 194, in _create_training_tensors
self.vocab_size)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\nn_impl.py", line 1349, in sampled_softmax_loss
seed=seed)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\nn_impl.py", line 1046, in _compute_sampled_logits
seed=seed)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\candidate_sampling_ops.py", line 145, in log_uniform_candidate_sampler
seed2=seed2, name=name)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_candidate_sampling_ops.py", line 730, in log_uniform_candidate_sampler
range_max=range_max, seed=seed, seed2=seed2, name=name)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
op_def=op_def)
File "C:\Users\fanta\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Sampler's range is too small.
[[node sampled_softmax_loss/LogUniformCandidateSampler (defined at C:\Users\fanta\Desktop\autoencoder-master\src\autoencoder.py:194) = LogUniformCandidateSamplernum_sampled=100, num_true=1, range_max=23, seed=0, seed2=0, unique=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

What I can do to solve this erros?
Thank you.

not expected output

python interactive.py /home/acer/PycharmProjects/VAE_MODEL/autoencoder-master/src/save_dir /home/acer/PycharmProjects/VAE_MODEL/autoencoder-master/src/save/vocabulary.txt
INFO:root:Reading model
2018-12-18 22:47:29.512330: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
INFO:tensorflow:Restoring parameters from /home/acer/PycharmProjects/VAE_MODEL/autoencoder-master/src/save_dir/model
INFO:tensorflow:Restoring parameters from /home/acer/PycharmProjects/VAE_MODEL/autoencoder-master/src/save_dir/model
Type tokenized sentence: i am an AI Engineer
Model output: a a a a a a a a a a a a
Type tokenized sentence:

UnimplementedError: TensorArray has size zero, but element shape <unknown> is not fully defined.

I encountered an error but not sure where the problem is, whether my data, config, etc. Here is the error output trying to run train-autoencoder. I'd greatly appreciate any help you might provide.

Thanks!!!

C:\Users\adammil\Documents\autoencoder-master\src>python train-autoencoder.py output vocabulary.txt train-data.npz valid-data.npz
2018-07-25 13:24:33.172002: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-07-25 13:24:33.742632: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1392] Found device 0 with properties:
name: Quadro M1200 major: 5 minor: 0 memoryClockRate(GHz): 1.148
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.35GiB
2018-07-25 13:24:33.750262: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1471] Adding visible gpu devices: 0
2018-07-25 13:24:35.118646: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-25 13:24:35.123184: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:958] 0
2018-07-25 13:24:35.125820: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0: N
2018-07-25 13:24:35.129404: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3077 MB memory) -> physical GPU (device: 0, name: Quadro M1200, pci bus id: 0000:01:00.0, compute capability: 5.0)
INFO:root:Reading training data
INFO:root:Reading validation data
INFO:root:Creating model
WARNING:tensorflow:From C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\rnn.py:417: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version.
Instructions for updating:
seq_dim is deprecated, use seq_axis instead
WARNING:tensorflow:From C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\rnn.py:417: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version.
Instructions for updating:
seq_dim is deprecated, use seq_axis instead
WARNING:tensorflow:From C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\deprecation.py:432: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version.
Instructions for updating:
batch_dim is deprecated, use batch_axis instead
WARNING:tensorflow:From C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\deprecation.py:432: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version.
Instructions for updating:
batch_dim is deprecated, use batch_axis instead
autoencoder/embeddings:0: (30125, 300) (9037500 parameters)
autoencoder/fw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/fw/lstm_cell/bias:0: (2000,) (2000 parameters)
autoencoder/bw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/bw/lstm_cell/bias:0: (2000,) (2000 parameters)
projection/dense/kernel:0: (1000, 30125) (30125000 parameters)
projection/dense/bias:0: (30125,) (30125 parameters)
Total: 42396625 parameters
INFO:root:Initialized the model and all variables. Starting training.
Traceback (most recent call last):
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
return fn(*args)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, autoencoder/autoencoder/bw/bw/TensorArrayUnstack/range, gradients/autoencoder/autoencoder/bw/bw/while/TensorArrayReadV3/Enter_1_grad/b_acc_3)]]
[[Node: sampled_softmax_loss/sparse_indices/_45 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_625_sampled_softmax_loss/sparse_indices", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train-autoencoder.py", line 113, in
args.dropout_keep, 5.0, report_interval=args.interval)
File "C:\Users\adammil\Documents\autoencoder-master\src\autoencoder.py", line 257, in train
_, loss = session.run([self.train_op, self.loss], feeds)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
run_metadata_ptr)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
run_metadata)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, autoencoder/autoencoder/bw/bw/TensorArrayUnstack/range, gradients/autoencoder/autoencoder/bw/bw/while/TensorArrayReadV3/Enter_1_grad/b_acc_3)]]
[[Node: sampled_softmax_loss/sparse_indices/_45 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_625_sampled_softmax_loss/sparse_indices", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'gradients/autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3', defined at:
File "train-autoencoder.py", line 106, in
bidirectional=args.bidirectional)
File "C:\Users\adammil\Documents\autoencoder-master\src\autoencoder.py", line 163, in init
self._create_training_tensors()
File "C:\Users\adammil\Documents\autoencoder-master\src\autoencoder.py", line 200, in _create_training_tensors
gradients, v = zip(*optimizer.compute_gradients(self.loss))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\optimizer.py", line 511, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 532, in gradients
gate_gradients, aggregation_method, stop_gradients)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 701, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 396, in _MaybeCompile
return grad_fn() # Exit early
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 701, in
lambda: grad_fn(op, *out_grads))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\tensor_array_grad.py", line 186, in _TensorArrayScatterGrad
grad = g.gather(indices)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\tensor_array_ops.py", line 873, in gather
return self._implementation.gather(indices, name=name)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\tensor_array_ops.py", line 303, in gather
element_shape=element_shape)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 6479, in tensor_array_gather_v3
flow_in=flow_in, dtype=dtype, element_shape=element_shape, name=name)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

...which was originally created as op 'autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3', defined at:
File "train-autoencoder.py", line 106, in
bidirectional=args.bidirectional)
File "C:\Users\adammil\Documents\autoencoder-master\src\autoencoder.py", line 85, in init
scope=self.scope)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\rnn.py", line 429, in bidirectional_dynamic_rnn
time_major=time_major, scope=bw_scope)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\rnn.py", line 618, in dynamic_rnn
dtype=dtype)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\rnn.py", line 745, in dynamic_rnn_loop
for ta, input in zip(input_ta, flat_input))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\rnn.py", line 745, in
for ta, input_ in zip(input_ta, flat_input))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\tf_should_use.py", line 118, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\tensor_array_ops.py", line 907, in unstack
return self._implementation.unstack(value, name=name)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\tf_should_use.py", line 118, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\tensor_array_ops.py", line 331, in unstack
indices=math_ops.range(0, num_elements), value=value, name=name)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\tf_should_use.py", line 118, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\tensor_array_ops.py", line 347, in scatter
name=name)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 7087, in tensor_array_scatter_v3
flow_in=flow_in, name=name)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\adammil\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)

UnimplementedError (see above for traceback): TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/autoencoder/autoencoder/bw/bw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, autoencoder/autoencoder/bw/bw/TensorArrayUnstack/range, gradients/autoencoder/autoencoder/bw/bw/while/TensorArrayReadV3/Enter_1_grad/b_acc_3)]]
[[Node: sampled_softmax_loss/sparse_indices/_45 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_625_sampled_softmax_loss/sparse_indices", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

a question about line124 in autoencoder.py

autoencoder/src/autoencoder.py

Line 124 in 6c0aa77

decoder_input = [embedded_eos] + list_input

why not "decoder_input = list_input + [embedded_eos]" ?
The symbol EOS supposes to mark the end of the sentence.

applying on 20 news group dataset

Hi,

I would like to apply your code on 20 newsgroup data set, but I am not sure how can I change the prepare data?

my problem mainly is in preparing labels. how it can be different from yours when it apply on 20 newsgroup data set?

Thanks for your help:)

what to feed as input for decoder

Hi~, I am using your software now. I think that's exactly what I want at the moment. But there is still a problem that seems quite confusing. That is I notice you are using the embedded input sentence to feed decoder. But in the article https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf, they are using the output at each timestep as the input for next timestep. Have I misunderstood your code or you are not doing what's going on in the article.

Validation loss unstable

Hi,

I've been run training autoencoder for a couple minutes and find validation loss was unstable, while average batch loss keep smaller.

INFO:root:68 epochs, 480 batches	Avg batch loss: 0.069513	Validation loss: 2.792187
INFO:root:69 epochs, 490 batches	Avg batch loss: 0.072354	Validation loss: 3.593376
INFO:root:71 epochs, 500 batches	Avg batch loss: 0.066709	Validation loss: 2.902308
INFO:root:72 epochs, 510 batches	Avg batch loss: 0.068916	Validation loss: 2.456438
INFO:root:74 epochs, 520 batches	Avg batch loss: 0.063203	Validation loss: 3.335535
INFO:root:75 epochs, 530 batches	Avg batch loss: 0.061095	Validation loss: 2.947956
INFO:root:77 epochs, 540 batches	Avg batch loss: 0.053819	Validation loss: 2.774727
INFO:root:78 epochs, 550 batches	Avg batch loss: 0.049284	Validation loss: 2.505195
INFO:root:79 epochs, 560 batches	Avg batch loss: 0.048512	Validation loss: 2.992307
INFO:root:81 epochs, 570 batches	Avg batch loss: 0.046487	Validation loss: 2.499180
INFO:root:82 epochs, 580 batches	Avg batch loss: 0.047605	Validation loss: 2.706982
INFO:root:84 epochs, 590 batches	Avg batch loss: 0.043647	Validation loss: 3.071584
INFO:root:85 epochs, 600 batches	Avg batch loss: 0.043592	Validation loss: 2.697872
INFO:root:87 epochs, 610 batches	Avg batch loss: 0.040263	Validation loss: 2.809274
INFO:root:88 epochs, 620 batches	Avg batch loss: 0.036155	Validation loss: 3.256417
INFO:root:89 epochs, 630 batches	Avg batch loss: 0.038276	Validation loss: 3.183321
INFO:root:91 epochs, 640 batches	Avg batch loss: 0.034546	Validation loss: 3.526584
INFO:root:92 epochs, 650 batches	Avg batch loss: 0.035519	Validation loss: 3.173420
INFO:root:94 epochs, 660 batches	Avg batch loss: 0.033806	Validation loss: 2.976058
INFO:root:95 epochs, 670 batches	Avg batch loss: 0.035120	Validation loss: 4.046538
INFO:root:97 epochs, 680 batches	Avg batch loss: 0.030202	Validation loss: 2.641236

What actually cause this problem ?

Thanks !

good for feature extraction

Hello,

Thanks for sharing your code:)
based on your experience with this code does this code is also suitable for feature extraction>?

UnimplementedError: TensorArray has size zero, but element shape <unknown> is not fully defined.

Hi,
I tried to run your code, while running train-autoencoder.py, I am getting the following error. Full trace is pasted below:
autoencoder/embeddings:0: (571, 300) (171300 parameters)
autoencoder/fw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/fw/lstm_cell/bias:0: (2000,) (2000 parameters)
autoencoder/bw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/bw/lstm_cell/bias:0: (2000,) (2000 parameters)
projection/dense/kernel:0: (1000, 571) (571000 parameters)
projection/dense/bias:0: (571,) (571 parameters)
Total: 3946871 parameters
INFO:root:Initialized the model and all variables. Starting training.
Traceback (most recent call last):
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:CPU:0"](gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, autoencoder/autoencoder/fw/fw/TensorArrayUnstack/range, gradients/autoencoder/autoencoder/fw/fw/while/TensorArrayReadV3/Enter_1_grad/b_acc_3)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train-autoencoder.py", line 113, in
args.dropout_keep, 5.0, report_interval=args.interval)
File "/Users/abubakar/Documents/chatbot/tensorflow/autoencoder/src/autoencoder.py", line 257, in train
_, loss = session.run([self.train_op, self.loss], feeds)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:CPU:0"](gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, autoencoder/autoencoder/fw/fw/TensorArrayUnstack/range, gradients/autoencoder/autoencoder/fw/fw/while/TensorArrayReadV3/Enter_1_grad/b_acc_3)]]

Caused by op 'gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3', defined at:
File "train-autoencoder.py", line 106, in
bidirectional=args.bidirectional)
File "/Users/abubakar/Documents/chatbot/tensorflow/autoencoder/src/autoencoder.py", line 163, in init
self._create_training_tensors()
File "/Users/abubakar/Documents/chatbot/tensorflow/autoencoder/src/autoencoder.py", line 200, in _create_training_tensors
gradients, v = zip(*optimizer.compute_gradients(self.loss))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 526, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 494, in gradients
gate_gradients, aggregation_method, stop_gradients)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 636, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 385, in _MaybeCompile
return grad_fn() # Exit early
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 636, in
lambda: grad_fn(op, *out_grads))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_grad.py", line 186, in _TensorArrayScatterGrad
grad = g.gather(indices)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 910, in gather
return self._implementation.gather(indices, name=name)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 305, in gather
element_shape=element_shape)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 6018, in tensor_array_gather_v3
flow_in=flow_in, dtype=dtype, element_shape=element_shape, name=name)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

...which was originally created as op 'autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3', defined at:
File "train-autoencoder.py", line 106, in
bidirectional=args.bidirectional)
File "/Users/abubakar/Documents/chatbot/tensorflow/autoencoder/src/autoencoder.py", line 85, in init
scope=self.scope)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py", line 412, in bidirectional_dynamic_rnn
time_major=time_major, scope=fw_scope)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py", line 627, in dynamic_rnn
dtype=dtype)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py", line 754, in dynamic_rnn_loop
for ta, input in zip(input_ta, flat_input))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py", line 754, in
for ta, input_ in zip(input_ta, flat_input))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 118, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 944, in unstack
return self._implementation.unstack(value, name=name)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 118, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 333, in unstack
indices=math_ops.range(0, num_elements), value=value, name=name)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 118, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 349, in scatter
name=name)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 6590, in tensor_array_scatter_v3
flow_in=flow_in, name=name)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/Users/abubakar/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)

how to run the program

hi,I'm interested in this project.But I got a problem when I run it. I have some questions:

What is the format of the input data?My input like this: 我是一个小孩（Chinese words splited by whitespace）Is it right?
In the second step,I run this command:
python train-autoencoder.py /model data/vocabulary.txt data/train-data.npz data/valid-data.npz，but there is no model is saved in the directory '/model'. Can you tell me the reason?

I will really appreciate it if you can answer my questions.Thank you for you patience!

Minor bug in prepare-data.py

Happy to do a pull request by basically:

     train_data, valid_data, words = load_data_memory_friendly(args.input,
                                                               args.max_length,
                                                               args.min_freq)

Needs to be replaced by

     train_data, valid_data, words = load_data_memory_friendly(args.input,
                                                               args.max_length,
                                                               args.min_freq,
                                                               args.valid_proportion)

Also, I made the code compatible with Python3. Let me know if you are interested.

Is there a way to decode the vectors into words?

Hi, I understand you are using the state.c as a representation of the sentences, but suppose I can get a state.c without knowing anything about the sentence, is there a way to decode it into the sentence?

Model initialisation problem

I have tried to run your code but I got the following exception:

INFO:root:Reading training data
INFO:root:Reading validation data
INFO:root:Creating model
Traceback (most recent call last):
  File "/Users/i342044/virtualenv-dir/py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py", line 578, in as_dtype
    if key == type_value:
TypeError: data type not understood

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/i342044/Documents/Codes/autoencoder-master/src/train-autoencoder.py", line 113, in <module>
    bidirectional=args.bidirectional)
  File "/Users/i342044/Documents/Codes/autoencoder-master/src/autoencoder.py", line 84, in __init__
    initializer = tf.zeros_initializer((self.vocab_size,))
  File "/Users/i342044/virtualenv-dir/py3/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py", line 88, in __init__
    self.dtype = dtypes.as_dtype(dtype)
  File "/Users/i342044/virtualenv-dir/py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py", line 581, in as_dtype
    raise TypeError("Cannot convert {} to a dtype. {}".format(type_value, e))
TypeError: Cannot convert (4477,) to a dtype. data type not understood

Process finished with exit code 1

Any idea on how to solve it?

Thanks

getting error in concatenating

Hi,

I am getting this error while trying to run train-autoencoder.py

INFO:root:Reading training data
INFO:root:Reading validation data
INFO:root:Creating model
Traceback (most recent call last):
  File "./src/train-autoencoder.py", line 113, in <module>
    bidirectional=args.bidirectional)
  File "/home/sgnbx/Downloads/projects/lstm-autoencoder-master/src/autoencoder.py", line 157, in __init__
    output = tf.concat(1, output)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1096, in concat
    dtype=dtypes.int32).get_shape().assert_is_compatible_with(
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 836, in convert_to_tensor
    as_ref=False)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 926, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 229, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 208, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 383, in make_tensor_proto
    _AssertCompatible(values, dtype)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 303, in _AssertCompatible
    (dtype.name, repr(mismatch), type(mismatch).__name__))
TypeError: Expected int32, got list containing Tensors of type '_Message' instead.

Any idea how to fix it?

How do I choose embedding_size

Thanks for explanation last time. But I am still thinking about how to set up embedding size, is it a really bad thing when it is set to only 1 or 2 ?

Model Saving Problem

Hello, I have tried to train a model but I got the following exception while saving the model:

autoencoder/embeddings:0: (1380, 300) (414000 parameters)
autoencoder/fw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/fw/lstm_cell/bias:0: (2000,) (2000 parameters)
autoencoder/bw/lstm_cell/kernel:0: (800, 2000) (1600000 parameters)
autoencoder/bw/lstm_cell/bias:0: (2000,) (2000 parameters)
projection/dense/kernel:0: (1000, 1380) (1380000 parameters)
projection/dense/bias:0: (1380,) (1380 parameters)
Total: 4999380 parameters
INFO:root:Initialized the model and all variables. Starting training.
Traceback (most recent call last):
  File "train-autoencoder.py", line 113, in <module>
    args.dropout_keep, 5.0, report_interval=args.interval)
  File "/autoencoder-master/src/autoencoder.py", line 285, in train
    self.save(saver, session, save_path)
  File "/autoencoder-master/src/autoencoder.py", line 305, in save
    json.dump(metadata, f)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 180, in dump
    fp.write(chunk)
 TypeError: a bytes-like object is required, not 'str'

Do you have any idea?
Thanks

when using on raw text it is working but, on preprocessed and stemmed version of same text it does not!

I am a beginner in this. I applied your code to get sentence vectors, it works well on the raw text but when using it on the preprocessed (with stemming) version of the same text it does not work. The "prepare-data" works fine and successfully supplied train_data.npz, valid_data.npz and vocabulary.txt. The next step which is the "train_autoencoder" didn't work. Please check the error message given below
autoencoder/embeddings:0: (2845, 100) (284500 parameters)
autoencoder/fw/lstm_cell/kernel:0: (228, 512) (116736 parameters)
autoencoder/fw/lstm_cell/bias:0: (512,) (512 parameters)
autoencoder/bw/lstm_cell/kernel:0: (228, 512) (116736 parameters)
autoencoder/bw/lstm_cell/bias:0: (512,) (512 parameters)
projection/dense/kernel:0: (256, 2845) (728320 parameters)
projection/dense/bias:0: (2845,) (2845 parameters)
Total: 1250161 parameters
INFO:root:Initialized the model and all variables. Starting training.
2020-11-23 13:40:47.008677: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnimplementedError: 2 root error(s) found.
(0) Unimplemented: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[{{node gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3}}]]
[[truediv/_139]]
(1) Unimplemented: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[{{node gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/content/drive/My Drive/Autoencoder/train-autoencoder.py", line 117, in
args.dropout_keep, 5.0, report_interval=args.interval)
File "/content/drive/My Drive/Autoencoder/autoencoder.py", line 259, in train
_, loss = session.run([self.train_op, self.loss], feeds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnimplementedError: 2 root error(s) found.
(0) Unimplemented: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[node gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 (defined at content/drive/My Drive/Autoencoder/autoencoder.py:202) ]]
[[truediv/_139]]
(1) Unimplemented: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[node gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 (defined at content/drive/My Drive/Autoencoder/autoencoder.py:202) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3:
autoencoder/autoencoder/fw/fw/TensorArrayUnstack/range (defined at content/drive/My Drive/Autoencoder/autoencoder.py:87)

Input Source operations connected to node gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3:
autoencoder/autoencoder/fw/fw/TensorArrayUnstack/range (defined at content/drive/My Drive/Autoencoder/autoencoder.py:87)

Original stack trace for 'gradients/autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3':
File "content/drive/My Drive/Autoencoder/train-autoencoder.py", line 110, in
bidirectional=args.bidirectional)
File "content/drive/My Drive/Autoencoder/autoencoder.py", line 165, in init
self._create_training_tensors()
File "content/drive/My Drive/Autoencoder/autoencoder.py", line 202, in _create_training_tensors
gradients, v = zip(*optimizer.compute_gradients(self.loss))
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/training/optimizer.py", line 512, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py", line 172, in gradients
unconnected_gradients)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py", line 669, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py", line 336, in _MaybeCompile
return grad_fn() # Exit early
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py", line 669, in
lambda: grad_fn(op, *out_grads))
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_grad.py", line 204, in _TensorArrayScatterGrad
grad = g.gather(indices)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_ops.py", line 1190, in gather
return self._implementation.gather(indices, name=name)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_ops.py", line 305, in gather
element_shape=element_shape)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 6481, in tensor_array_gather_v3
element_shape=element_shape, name=name)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3485, in _create_op_internal
op_def=op_def)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1949, in init
self._traceback = tf_stack.extract_stack()

...which was originally created as op 'autoencoder/autoencoder/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3', defined at:
File "content/drive/My Drive/Autoencoder/train-autoencoder.py", line 110, in
bidirectional=args.bidirectional)
File "content/drive/My Drive/Autoencoder/autoencoder.py", line 87, in init
scope=self.scope)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 447, in bidirectional_dynamic_rnn
scope=fw_scope)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 691, in dynamic_rnn
dtype=dtype)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 823, in dynamic_rnn_loop
ta.unstack(input) for ta, input_ in zip(input_ta, flat_input))
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 823, in
ta.unstack(input_) for ta, input_ in zip(input_ta, flat_input))
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_should_use.py", line 247, in wrapped
return _add_should_use_warning(fn(*args, **kwargs),
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_ops.py", line 1224, in unstack
return self._implementation.unstack(value, name=name)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_should_use.py", line 247, in wrapped
return _add_should_use_warning(fn(*args, **kwargs),
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/tensor_array_ops.py", line 328, in unstack
indices=math_ops.range(0, num_elements), value=value, name=name)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_should_use.py", line 247, in wrapped
return _add_should_use_warning(fn(*args, **kwargs),

text classification

Hi and thanks for the nice project.
I have never used autoencoders, so forgive my ignorance.
Questions:
1 - Can I use your code to classify a text into 2 categories ? - I have a dataset where 1st column contains 1 sentence and 2nd column a category, either 0 or 1.
2. If yes to Question 1, how should the train and test data look?

UnimplementedError: TensorArray has size zero, but element shape <unknown> is not fully defined.

Hi, I have this problem:
UnimplementedError (see above for traceback): TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.

I know this problem was already report. But did you fix this problem? Or there is a way that can I do it by my self?

Thank You.

Sketch_AE

Border situation in prepare-data.py

In the line 62 of prepare-data.py, threshold shoud be replaced with min(threshold, max_size). Otherwise, there will be bug when max_size is not divisible by 10.

raise TypeError("Cannot convert {} to a dtype. {}".format(type_value, e))

hi , so am using tensorflow1.0 on python 3.5.2 on windows ,ive successfully run "prepare-data.py" but when am running "train-autoencoder.py" i get the following error

raise TypeError("Cannot convert {} to a dtype. {}".format(type_value, e))
TypeError: Cannot convert (19528,) to a dtype. data type not understood

i think its because of the initializer so i changed "initializer = tf.zeros_initializer((self.vocab_size,))"
to "initializer = tf.zeros_initializer" and It worked but i get another error when it tries to use the "projection_b" variable , seems like we have to specify the shape of it
"Shape of a new variable (autoencoder/projection_b) must be fully defined, but instead was ."
i dont know how to fix this one ,i really appreciate it if anyone could help me , thanks

Problema ao salvar o modelo treinado.

Após executar o script train-autoencoder.py não consigo encontrar nenhum arquivo na pasta de saída explicitada.

Poderia ser problema com a versão do Tensorflow? Estou usando 1.8.

Multilayered LSTM

Hello @erickrf ,

Thanks for the great reference implementation. I noticed that the paper mentioned that they have 4 layers of LSTMs. Is this adopted in your code? I am not sure which are the 4 layers. Is it 3 for encoder and 1 for decoder? Sorry if I understood it incorrectly. Thanks!

erickrf / autoencoder Goto Github PK

autoencoder's Introduction

Text Autoencoder

Implementation

Performance notes

Scripts

autoencoder's People

Contributors

Stargazers

Watchers

Forkers

autoencoder's Issues

Recommend Projects

Recommend Topics

Recommend Org