currie32 / text-summarization-with-amazon-reviews Goto Github PK

@Currie32 Can you please help - this is producing the same summary for each review. in my case repeating "not" many times.

What can be the root cause

In your code it seems like you are trying to create a multilayer encoder, however what is actually happening is that multiple single layer decoders are being created. As you can see in this graph image created by tensorflow http://imgur.com/a/jqAn5 the rnn inputs are feed into each bidirectional rnn rather than one feeding into the other. I created this image by running your encoding_layer function and passing in some placeholders and then using the tf.summary.FileWriter to draw the graph.

To create a multilayer encoder you would want something like this

def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):
    '''Create the encoding layer'''
    layer_input = rnn_inputs
    for layer in range(num_layers):
        with tf.variable_scope('encoder_{}'.format(layer)):
            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, 
                                                    input_keep_prob = keep_prob)

            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, 
                                                    input_keep_prob = keep_prob)

            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, 
                                                                    cell_bw, 
                                                                    layer_input,
                                                                    sequence_length,
                                                                    dtype=tf.float32)
            layer_input = tf.concat(enc_output, 2)
    # Join outputs since we are using a bidirectional RNN
    enc_output = tf.concat(enc_output,2)
    return enc_output, enc_state

You can see in this graph image http://imgur.com/a/bJANa That this creates the multi layer structure I assume you are going for. (The caveat is that in this function your rnn size must be half of the embedding size so if you want them to be different just move the first birnn out of the loop and have the layers after the first have the same size)

Thank you very much your tutorial. it really helped me a lot.
I am wondering if you can help me with this. In you set up, it seems that you are making every encoding layer as bidirectional. But i think it is also very common that we only make 1 bidirectional layer and making other layers as regular rnn layers.(like google's translation model).

I am not sure how to connect a bidirectional layer with a regular rnn layer?

Hi,
I'm trying to restore the session for the last checkpoint saved. I do this uncommenting this two lines of the code:

loader = tf.train.import_meta_graph("./" + checkpoint + '.meta')
loader.restore(sess, checkpoint)

But I'm getting this error:
ValueError: cannot add op with name encoder_0/bidirectional_rnn/fw/lstm_cell/kernel/Adam as that name is already used

Am I doing something wrong or is this an issue?

Hi, I have trained with Fine Food Reviews data with 100 epochs. After that I have changed dataset. downloaded dataset from amazon reviews dataset. There I have taken Home and Kitchen dataset in that dataset I have taken Review Text and Summary. And trained with 100 epochs.

After completion of training I have tested the model. getting same summary for distinct reviews. I have trained so many times getting the same summary while I am testing

The model made some good predictions when I tested it immediately after training was finished. But when I loaded the model after opening the notebook next day, it makes some random meaningless predictions.
Any reason for this?

Anyone else having difficulty opening the numberbatch-en-17.02.txt.

Please provide guidance on how you were able to extract it.

AttributeError: module 'tensorflow' has no attribute 'placeholder'

Are they random vectors?

hi, when I use a similar code to solve a seq2seq problem, during the training, the loss decrease very quickly. but during the testing, all the output sequence are same. Can you example what sample_id means?

Here is the encoding_layer for ease:

def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):
    '''Create the encoding layer'''
    
    for layer in range(num_layers):
        with tf.variable_scope('encoder_{}'.format(layer)):
            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, 
                                                    input_keep_prob = keep_prob)

            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, 
                                                    input_keep_prob = keep_prob)

            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, 
                                                                    cell_bw, 
                                                                    rnn_inputs,
                                                                    sequence_length,
                                                                    dtype=tf.float32)
    # Join outputs since we are using a bidirectional RNN
    enc_output = tf.concat(enc_output,2)
    
    return enc_output, enc_state

I'm new to tensorflow, so here's my understanding:
Based on num_layers, it'll create that many tf.nn.bidirectional_dynamic_rnns. So shouldn't the line enc_output = tf.concat(enc_output,2) be inside the for loop ? Will the return statement returns all the enc_output. If so how ?

And how to find accuracy, can you suggest briefly ?