farizrahman4u / recurrentshop Goto Github PK

In requirements, it is written that theano>=0.8.0 . I want to use this with tensorflow. Is it possible? It starts downloading theano when I run python setup.py install

First of all, thanks for working on such a useful library!

I noticed you chose GPL-3 as a license, which is much more restrictive than Keras' license (MIT). Is there any chance you could relicense to a less restrictive license, such as MIT or BSD?

Thanks!

I've been using the recurrentshop-1 branch and unable to save models that have RecurrentModel layers in them. Here's an example using the test_recurrent_model.py in the tests.

from recurrentshop import RecurrentModel
from keras.models import Model
from keras.layers import *

x = Input((5,), name = 'xinput' )
h_tm1 = Input((10,), name = 'hinput' )
h = add([Dense(10)(x), Dense(10, use_bias=False)(h_tm1)])
h = Activation('tanh')(h)
a = Input((7, 5), name = 'ainput' )

rnn = RecurrentModel(input=x, output=h, initial_states=h_tm1, final_states=h)
b = rnn(a)
model = Model(a, b)

model.compile(loss='mse', optimizer='sgd')
model.save( 'test.h5' )

it returns the following error:

Traceback (most recent call last):
  File "test.py", line 18, in <module>
    model.save( 'test.h5' )
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.3-py2.7.egg/keras/engine/topology.py", line 2457, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.3-py2.7.egg/keras/models.py", line 102, in save_model
    'config': model.get_config()
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.3-py2.7.egg/keras/engine/topology.py", line 2320, in get_config
    new_node_index = node_conversion_map[node_key]
KeyError: 'ainput_ib-0'

What I've found is that somehow the input node names in the rnn layer are getting changed. This is problematic when connecting the rnn layer with the rest of the network. I found that if I give a specific name to the input node that connects to the rnn layer that the error goes away and the model is successfully saved.

from recurrentshop import RecurrentModel
from keras.models import Model
from keras.layers import *

x = Input((5,), name = 'xinput' )
h_tm1 = Input((10,), name = 'hinput' )
h = add([Dense(10)(x), Dense(10, use_bias=False)(h_tm1)])
h = Activation('tanh')(h)
a = Input((7, 5), name = 'private__optional_input_place_holder_1' )

rnn = RecurrentModel(input=x, output=h, initial_states=h_tm1, final_states=h)
b = rnn(a)
model = Model(a, b)

model.compile(loss='mse', optimizer='sgd')
model.save( 'test.h5' )

Though saving works doing this, I am unable to load the model afterwards. I get the following error:

ValueError: Unknown layer: _OptionalInputPlaceHolder

Should I be specifying _OptionalInputPlaceHolder as a custom_object when I'm using load_model?

Thanks!

Hello!
I am trying to write a simple GRU decoder, such as depicted below

My code:

z = Input(batch_shape=(batch_shape[0], latent_dim), name='latent')

def create_recshop_dec(z):
    with tf.variable_scope("decoder"):
        rnn1_dim = 64

        h_tm1 = Input(batch_shape=(batch_size, rnn1_dim))  # Previous hidden state
        x_t   = Input(batch_shape=(batch_size, rnn1_dim))  # The input to the RNN at time t
        output, h_t = GRUCell(rnn1_dim)([x_t, h_tm1])
        output = Dense(rnn1_dim, activation='softmax')(output)
        rnn = RecurrentModel(input=x_t, initial_states=[h_tm1], output=output, decode=True, output_length=MAX_LEN, readout_input=True, final_states=[h_t], return_sequences=True)

        x = Dense(rnn1_dim, activation='relu')(z)

        decoded, h = rnn(K.zeros(shape=(batch_size, rnn1_dim)), initial_state=x)

    return Model(z, decoded)

decoder = create_recshop_dec(z)

and I run into an error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-143-f1c8c9de3feb> in <module>()
      1 encoder, z_meaner, z_log_varer = create_enc(x)
----> 2 decoder = create_recshop_dec(z)

<ipython-input-141-795744eebc4e> in create_recshop_dec(z)
     11 
     12         x = Dense(rnn1_dim, activation='relu')(z)
---> 13         decoded, h = rnn(K.zeros(shape=(batch_size, rnn1_dim)), initial_state=x)
     14 
     15     return Model(z, decoded)

/Users/mikhail/anaconda/lib/python3.6/site-packages/recurrentshop-1.0.0-py3.6.egg/recurrentshop/engine.py in __call__(self, inputs, initial_state, initial_readout, ground_truth, **kwargs)
    413                         kwargs['mask'] = previous_mask
    414             input_shape = _collect_input_shape(inputs)
--> 415             output = self.call(inputs, **kwargs)
    416             output_mask = self.compute_mask(inputs[0], previous_mask)
    417             output_shape = self.compute_output_shape(input_shape[0])

/Users/mikhail/anaconda/lib/python3.6/site-packages/recurrentshop-1.0.0-py3.6.egg/recurrentshop/engine.py in call(self, inputs, initial_state, initial_readout, ground_truth, mask, training)
    440             inputs_list = inputs_list[len(self.states):]
    441             if self.readout:
--> 442                 initial_readout = inputs_list.pop(0)
    443                 if self.teacher_force:
    444                     ground_truth = inputs_list.pop()

IndexError: pop from empty list

I wonder, if I am doing it wrong, or there is a bug somewhere?

By the way, thanks for this wonderful peace of software, which Keras lacks so much.

Hi @farizrahman4u
Your packages (both seq2seq and recurrentshop) are unbelievably useful, but they both require more documentation and example as well as reference part for all available predefined layers.

Could you please provide us with such necessary examples, documentations, and glossaries?

I can help you if don't have enough time...just ping me in that case...

I am using teacher_force with tensorflow backend. But it failed when fit function called.
when using theano as backend, my source code worked.
my condition is as followeing.
Keras 2.0.6
tensorflow 1.2.1
theano 0.10.0b1

and my minimal code is like below.

input_length = MAX_SL
input_dim = len(corpus)
output_length = MAX_TL
output_dim = len(corpus)
LSTM_hidden_dim = 50

# make recurrent model
input_node = Input((input_length, input_dim), name = "main_input_decoder") 
readout_tm = Input([output_dim], name = "readout_input") 
lstm_st1 = Input((LSTM_hidden_dim,), name = "lstm_hidden_1") 
lstm_st2 = Input((LSTM_hidden_dim,), name = "lstm_hidden_2") 
s, lstm_st_t1, lstm_st_t2 = rs.LSTMCell(LSTM_hidden_dim)([readout_tm, lstm_st1, lstm_st2]) 
output = Dense(output_dim)(s)
output_model = rs.RecurrentModel(input = input_node, output=output 
                     , readout_input=readout_tm, return_sequences = True 
                     , initial_states = [lstm_st1, lstm_st2]
                     , final_states = [lstm_st_t1, lstm_st_t2] 
                     , decode = True, output_length = MAX_TL, unroll = False
                    , teacher_force = True
                     , stateful = False, name = "decoder_model") 

#use above recurrent model
input_node3 = Input((input_length, input_dim))
ground_truth = Input([output_length, output_dim], name="ground_truth")
decoded = output_model(input_node3, ground_truth=ground_truth)

model = Model([input_node3, ground_truth], decoded) 
model.compile(optimizer='adam', loss='binary_crossentropy') 

model.fit([inputs, outputs], outputs, epochs=1000, verbose = 1)

error is like this.

InvalidArgumentError: The node 'decoder_model/while_1/Variable_1/Assign' has inputs from different frames. The input 'decoder_model/while_1/Const_1' is in frame 'decoder_model/while_1/decoder_model/while_1/'. The input 'decoder_model/while_1/Variable_1' is in frame ''.

Is this bag? or is there wrong point on my code?

AttributeError: 'RecurrentSequential' object has no attribute '_preprocess_function'.

The backend of Keras is Tensorflow and my python envorinment is v3.4.3. After installed the package by pip I tried to run test.py in the test folder without any modification. My program is blocked when It runs to initial_states = self.get_initial_states(x) in engine.py and pop up a type error:
"TypeError: step() missing 1 required positional argument: 'weights."

The trackback looks like below:
Traceback (most recent call last):
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1556, in
globals = debugger.run(setup['file'], None, None, is_module)
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 940, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/apple/Documents/workspace/dl/recurrentshop/tests/test.py", line 11, in
model.add(rc)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/keras/models.py", line 275, in add
layer.create_input_layer(batch_input_shape, input_dtype)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/keras/engine/topology.py", line 367, in create_input_layer
self(x)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/keras/engine/topology.py", line 511, in call
self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/keras/engine/topology.py", line 569, in add_inbound_node
Node.create_node(self, inbound_layers, node_indices, tensor_indices)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/keras/engine/topology.py", line 150, in create_node
output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
File "/Users/apple/Documents/workspace/dl/recurrentshop/recurrentshop/engine.py", line 285, in call
initial_states = self.get_initial_states(x)
File "/Users/apple/Documents/workspace/dl/recurrentshop/recurrentshop/engine.py", line 370, in get_initial_states
input = layer.step(input, layer_initial_states)[0]
TypeError: step() missing 1 required positional argument: 'weights'
Could you give any suggestion to resolve the issue?

If I save to json & load a model which had RecurrentContainer layer with decode=True then I get 'Missing argument: output_length should be specified for decoders.'
Looks like output_length property doesn't get serialized into json file during model save.

On keras 2.02.

Minimalish example:

from recurrentshop.engine import RNNCell, RecurrentModel
from recurrentshop.cells import LSTMCell
from keras.layers import Input, concatenate, Dense, Conv1D, LSTM, GRU, Lambda, Layer
from keras.models import Model

def build_model(n_channels=1, trace_len=100, rnn_hus=11):
    """ Try just building the entire model within a single recurrent cell
    """
    full_input = Input((trace_len, 1), name='full_input')
    rnn_in = Input((1,), name='rnn_in')
    enc_out = Dense(10, activation='relu', name='encode')(rnn_in)
    h_tm1 = Input((rnn_hus,), name='h_tm1')
    c_tm1 = Input((rnn_hus,), name='c_tm1')
    readout_input = Input((n_channels, ), name='readout_input')
    lstm_in = concatenate([enc_out, readout_input], name='concat_lstm_in')
    lstm_out, h_t, c_t = LSTMCell(rnn_hus, name='lstm_cel')([lstm_in, h_tm1, c_tm1])
    decode_out = Dense(n_channels, activation='relu', name='decode')(lstm_out)    
    rnn_layer = RecurrentModel(input=rnn_in, initial_states=[h_tm1, c_tm1], output=decode_out, final_states=[h_t, c_t],  unroll=False, return_sequences=True, name='rnn_layer', readout_input=readout_input)
    m = Model(inputs=(full_input, rnn_in, h_tm1, c_tm1), outputs=rnn_out)
    return m

m = build_model()

Now executing m.summary() throws

RuntimeError                              Traceback (most recent call last)
<ipython-input-135-4c5d74d34759> in <module>()
----> 1 m.summary()

~/anaconda3/envs/py2_tf_gpu/lib/python2.7/site-packages/keras/engine/topology.pyc in summary(self, line_length, positions)
   2564         print_layer_summary(self,
   2565                             line_length=line_length,
-> 2566                             positions=positions)
   2567 
   2568 

~/anaconda3/envs/py2_tf_gpu/lib/python2.7/site-packages/keras/utils/layer_utils.pyc in print_summary(model, line_length, positions)
    106             print_layer_summary(layers[i])
    107         else:
--> 108             print_layer_summary_with_connections(layers[i])
    109         if i == len(layers) - 1:
    110             print('=' * line_length)

~/anaconda3/envs/py2_tf_gpu/lib/python2.7/site-packages/keras/utils/layer_utils.pyc in print_layer_summary_with_connections(layer)
     94         else:
     95             first_connection = connections[0]
---> 96         fields = [name + ' (' + cls_name + ')', output_shape, layer.count_params(), first_connection]
     97         print_row(fields, positions)
     98         if len(connections) > 1:

~/anaconda3/envs/py2_tf_gpu/lib/python2.7/site-packages/keras/engine/topology.pyc in count_params(self)
   1223                                    self.name + ', but the layer isn\'t built. '
   1224                                    'You can build it manually via: `' +
-> 1225                                    self.name + '.build(batch_input_shape)`.')
   1226         return sum([K.count_params(p) for p in self.weights])
   1227 

RuntimeError: You tried to call `count_params` on private__optional_input_place_holder_83, but the layer isn't built. You can build it manually via: `private__optional_input_place_holder_83.build(batch_input_shape)`.

This threw me off for a while, as I thought I had misspecified my model. However, the model compiles and runs just fine, so this seems to be a bug with recurrentshop.

Hi, I guess this error also caused by keras version 2, but I do not know how to fix it. Is anybody here can help me?

Hi,

Thanks for this great package. I just came across it and wanted to try. However, when I run the speed_test.py in the examples, the code segment for testing the recurrentshop model reported the following errors:

Traceback (most recent call last):
File "D:\software\Python\Miniconda3\lib\site-packages\theano\theano\tensor\var.py", line 592, in iter
for i in xrange(theano.tensor.basic.get_vector_length(self)):
File "D:\software\Python\Miniconda3\lib\site-packages\theano\theano\tensor\basic.py", line 4463, in get_vector_length
v)
TypeError: argument must be symbolic vector, got '<TensorType(float32, matrix)>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File ".\test.py", line 164, in
model.train_on_batch(x[:1], y[:1]) # force compile
File "D:\software\Python\Miniconda3\lib\site-packages\keras\keras\models.py", line 758, in train_on_batch
class_weight=class_weight)
File "D:\software\Python\Miniconda3\lib\site-packages\keras\keras\engine\training.py", line 1266, in train_on_batch
self._make_train_function()
File "D:\software\Python\Miniconda3\lib\site-packages\keras\keras\engine\training.py", line 720, in _make_train_function
**self._function_kwargs)
File "D:\software\Python\Miniconda3\lib\site-packages\keras\keras\backend\theano_backend.py", line 902, in function
return Function(inputs, outputs, updates=updates, **kwargs)
File "D:\software\Python\Miniconda3\lib\site-packages\keras\keras\backend\theano_backend.py", line 881, in init
for v, nv in updates:
File "D:\software\Python\Miniconda3\lib\site-packages\theano\theano\tensor\var.py", line 596, in iter
raise TypeError(('TensorType does not support iteration. '
TypeError: TensorType does not support iteration. Maybe you are using builtin.sum instead of theano.tensor.sum? (Maybe .max?)

I am using Python 3.5, Theano 0.9 dev, Keras 1.1.2. I wonder if somebody can help me to identify the problem.

Issue/clarification reported by @jachiam
the code snippet I'm looking at is in 'step' in the engine...
https://github.com/datalogai/recurrentshop/blob/master/recurrentshop/engine.py#L247
if self.state_sync:
x, new_states = layer._step(x, states[:len(layer.states)])
states[:len(layer.states)] = new_states
So, it looks like this uses the "states" variable to keep track of the hidden states of the system.
Let's say that h(t, l) denotes the hidden state after tick 't' at layer 'l'.
The code seems to indicate that h(t-1,L), where L is the last layer, goes as the input hidden state to the first layer at time t - that is, h(t-1,L) = h(t,1).
description sounds like you want it to be the case that h(t,2) ALSO equals h(t-1,L). But the code suggests that h(t,2) is output by layer 1 at time t, because you overwrite the content of the "states" variable after the layer._step, using its output.
Am I mistaken in my interpretation of the "states" variable?

First of all, this library looks like a great addition to Keras, thanks for providing it. Now for my question: This might be obvious, but I'm not totally clear on the uses of readout=True and decode=True in your examples. When would I use either one, and when would I want to specify both arguments?

I see that decode=True requires setting an output length (so it can be used for n-to-m seq2seq learning), and that readout simply basically adds the previous timesteps final output to the current input. Isn't this sort of "readout" required for decoders in most settings? When does readout=True by itself become useful? Also, slightly less related: how do I use the RecurrentContainer to specify the initial hidden state of a decoder?

Finally, there seem to be multiple readout modes: https://github.com/datalogai/recurrentshop/blob/master/recurrentshop/engine.py#L244-L251 , are they documented somewhere?

I know these are a lot of questions, but perhaps you could just generally clarify the concepts a little. Thank you.

The classes GRNCell and LSTMCell are included in both basic_cells.py and cells.py. Is this intended?

please consider supporting multiple input tensor

I have the following RecurrentContainer:- 3 layer LSTM + linear RNN layer.
rc = RecurrentContainer(readout='call', return_sequences=True, stateful=False, input_shape= (timesteps, n_input_dim))
rc.add(LSTMCell(hidden_size, input_dim=n_input_dim, activation='tanh'))
rc.add(LSTMCell(hidden_size, activation='tanh'))
rc.add(LSTMCell(hidden_size, activation='tanh'))
rc.add(SimpleRNNCell(n_output_dim, activation='linear'))
input_seq = Input(shape=(timesteps, n_input_dim,))
output_seq=rc(input_seq)

1. Is this the correct usage in the case where I want the input to be something like input_seq(t), output_seq(t-1) . Since readout=call gives me the same result as readout=False.
2. How do I access the readout, so that I can use it as an input to another layer?

I am getting the following error when I am trying to install recurrentshop:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "recurrentshop/__init__.py", line 1, in <module> from .engine import RNNCell, RecurrentContainer, weight File "recurrentshop/engine.py", line 3, in <module> from keras import initializations, regularizers, constraints ImportError: cannot import name initializations

I have keras and tensorflow running, using ubuntu. Any help would be appreciated

Running into some shape handling trouble when using initial_state in with RecurrentModel. This is the error
ValueError: Shape must be rank 2 but is rank 1 for 'recurrent_model_1/concatenate_1/concat' (op: 'ConcatV2') with input shapes: [?,20], [10], [].

My RNN

def RWA(input_dim, output_dim):
    x = Input((input_dim,))
    h_tm1 = Input((output_dim, ))
    n_tm1 = Input((output_dim, ))
    d_tm1 = Input((output_dim, ))

    x_h = concatenate([x, h_tm1])  # Fail happens here


    u = Dense(output_dim)(x)
    g = Dense(output_dim, activation='tanh')(x_h)

    a = Dense(output_dim, use_bias=False)(x_h)
    e_a = Lambda(lambda x: exp(x))(a)

    z = multiply([u, g])
    nt = add([n_tm1, multiply([z, e_a])])
    dt = add([d_tm1, e_a])
    ht = Lambda(lambda x: tf.divide(1.0, x))(dt)
    ht = Activation('tanh')(ht)

    return RecurrentModel(input=x, output=ht,
                          initial_states=[h_tm1, n_tm1, d_tm1],
                          final_states=[ht, nt, dt],)

This is my model using it

input_dim = 20
output_dim = 10
timesteps = 7

starting_states = [K.random_normal_variable((output_dim,), 0, 1),
                 K.zeros((output_dim,)),
                 K.zeros((output_dim,))]

rwa = RWA(input_dim, output_dim)

inp = Input((timesteps, input_dim))
out = rwa(inp, initial_state=starting_states)
# out = rwa(inp)			# Model builds fine without the initial state
RWAModel = Model(inp, out)

I am trying to connect a tensorflow placeholder with a keras recuurent shop model,Everything works fine when i use a simple keras model and feed it with a tensorflow placeholder , but the code throws an error when i 'm trying to use a recurrent shop model with it.

import keras.backend as K
from keras.layers import *
from keras.models import *
from recurrentshop import *
import tensorflow as tf`
sess = tf.Session()
K.set_session(sess) 
x_t = Input((None,5)) 
h_tm1 = Input((10,))  
h_t = add([Dense(10)(x_t), Dense(10, use_bias=False)(h_tm1)])
h_t = Activation('tanh')(h_t)
rnn = RecurrentModel(input=x_t, initial_states=[h_tm1], output=h_t, final_states=[h_t])
x = Input((None,5))
y = rnn(x)
model = Model(x, y)
image_data = tf.placeholder(tf.float32, shape=[None,None,5])
sess.run(tf.global_variables_initializer())
y1=model(image_data)
sess.run(y1,feed_dict={image_data :np.random.random((7,5, 5))})

This throws a key error with trace :-

 ' Found: ' + str(self.outputs))
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/sam/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 880, in runfile
   execfile(filename, namespace)
 File "/home/sam/ana
```conda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile
   builtins.execfile(filename, *where)
 File "/home/sam/test/garbage.py", line 233, in <module>
   y1=model(image_data)
 File "/home/sam/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 615, in __call__
   output_shape = self.compute_output_shape(input_shape)
 File "/home/sam/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 2123, in compute_output_shape
   input_shape = layers_to_output_shapes[shape_key]
KeyError: 'private__optional_input_place_holder_3_0_0'

While a non recurrent model like below works just fine :-

from keras.layers import *
from keras.models import *
from recurrentshop import *
import tensorflow as tf
sess = tf.Session()
K.set_session(sess) 
inp = Input(([5]))
h_t = Dense(10)(inp)
h_t = Activation('tanh')(h_t)
model = Model(inp, h_t)
image_data = tf.placeholder(tf.float32, shape=[None,5])
y1=model(image_data)
sess.run(tf.global_variables_initializer())
sess.run(y1,feed_dict={image_data :np.random.random((7, 5))})

When I'm up to date with the Keras Branch and using the backend, I get the following error with the examples on the github readme when calling fit.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Other\Python27\python-2.7.10.amd64\lib\site-packages\keras\models.py", line 664, in fit
    sample_weight=sample_weight)
  File "C:\Other\Python27\python-2.7.10.amd64\lib\site-packages\keras\engine\training.py", line 1115                                                                                                 , in fit
    self._make_train_function()
  File "C:\Other\Python27\python-2.7.10.amd64\lib\site-packages\keras\engine\training.py", line 720,                                                                                                  in _make_train_function
    **self._function_kwargs)
  File "C:\Other\Python27\python-2.7.10.amd64\lib\site-packages\keras\backend\theano_backend.py", li                                                                                                 ne 929, in function
    return Function(inputs, outputs, updates=updates, **kwargs)
  File "C:\Other\Python27\python-2.7.10.amd64\lib\site-packages\keras\backend\theano_backend.py", li                                                                                                 ne 908, in __init__
    for v, nv in updates:
  File "C:\Other\Python27\python-2.7.10.amd64\lib\site-packages\theano\tensor\var.py", line 584, in                                                                                                  __iter__
    raise TypeError(('TensorType does not support iteration. '
TypeError: TensorType does not support iteration. Maybe you are using builtin.sum instead of theano.                                                                                                 tensor.sum? (Maybe .max?)

However, if I revert to this then it starts working. What broke?

I created this basic LSTM network while learning RecurrentShop. However the output of the code alternates between two results. 50% of the time the networks trains and outputs something reasonable, while the other half of the runs the network does not train at all.

My complete code and outputs from two different runs(take a look at loss).

from recurrentshop import RecurrentContainer, LSTMCell
import numpy
from keras.layers import Dense, Input
from keras.models import Model

nb_examples = 10000
x = [[[1, 2, 3, 4]]]*nb_examples + [[[2, 3, 4, 5]]]*nb_examples
y = [10]*nb_examples + [14]*nb_examples

rc = RecurrentContainer(readout=True)

rc.add(LSTMCell(4, input_shape=(4,)))

input = Input(shape=(None, 4))

l = (rc)(input)
output = Dense(1, activation='relu')(l)
model = Model(input=input, output=[input, output])
model.compile(loss='mse', optimizer='adam')
model.fit(x=numpy.asarray(x), y=[numpy.asarray(x), numpy.asarray(y)])

print(model.predict(numpy.asarray([[[1, 2, 3, 4]]])))

Epoch 1/10
20000/20000 [==============================] - 1s - loss: 111.0854 - input_1_loss: 0.0000e+00 - dense_1_loss: 111.0854     
Epoch 2/10
20000/20000 [==============================] - 0s - loss: 61.1840 - input_1_loss: 0.0000e+00 - dense_1_loss: 61.1840     
Epoch 3/10
20000/20000 [==============================] - 0s - loss: 37.5625 - input_1_loss: 0.0000e+00 - dense_1_loss: 37.5625     
Epoch 4/10
20000/20000 [==============================] - 0s - loss: 22.2517 - input_1_loss: 0.0000e+00 - dense_1_loss: 22.2517     
Epoch 5/10
20000/20000 [==============================] - 0s - loss: 12.7959 - input_1_loss: 0.0000e+00 - dense_1_loss: 12.7959     
Epoch 6/10
20000/20000 [==============================] - 0s - loss: 7.5595 - input_1_loss: 0.0000e+00 - dense_1_loss: 7.5595     
Epoch 7/10
20000/20000 [==============================] - 0s - loss: 5.1144 - input_1_loss: 0.0000e+00 - dense_1_loss: 5.1144     
Epoch 8/10
20000/20000 [==============================] - 0s - loss: 4.0420 - input_1_loss: 0.0000e+00 - dense_1_loss: 4.0420     
Epoch 9/10
20000/20000 [==============================] - 0s - loss: 1.7059 - input_1_loss: 0.0000e+00 - dense_1_loss: 1.7059     
Epoch 10/10
20000/20000 [==============================] - 0s - loss: 0.4829 - input_1_loss: 0.0000e+00 - dense_1_loss: 0.4829     
[array([[[ 1.,  2.,  3.,  4.]]], dtype=float32), array([[ 10.01733208]], dtype=float32)]

Epoch 1/10
20000/20000 [==============================] - 1s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 2/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 3/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 4/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 5/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 6/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 7/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 8/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 9/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
Epoch 10/10
20000/20000 [==============================] - 0s - loss: 148.0000 - input_1_loss: 0.0000e+00 - dense_1_loss: 148.0000     
[array([[[ 1.,  2.,  3.,  4.]]], dtype=float32), array([[ 0.]], dtype=float32)]

Very simple, I want to connect an RNN's state to another RNN's state.
The issue is that the RNN's state is not symbolic, RNN.states = [(int, int)].

rc1 = RecurrentContainer()
rc1.add(GRUCell(output_dim), input_dim=15)

rc2 = RecurrentContainer()
rc2.add(GRUCell(output_dim), input_dim=30)

# It assigns a tuple value, not a tensor.
rc2.model.layers[1].states = rc1.model.layers[1].states

How can I connect those 2 RNNs? If it is not implemented yet, let me know.

i cannot find a way of combining the two examples from the documentation.
E.g., i want to decode a 13-dimensional state vector s0 into a sequence of length 11. Each element of the sequence is a softmax over a vocabulary of 3 words. I'm using batch size of 7. This is what I've got, but it fails in fit due to a shape problem:

import numpy as np
from keras.layers import Input, Dense
from keras.engine import Model
from recurrentshop import RecurrentModel, RecurrentSequential
from recurrentshop.cells import GRUCell

rnn = RecurrentSequential(decode=True, output_length=11, readout='readout_only',
          teacher_force=True, return_sequences=True)
rnn.add(GRUCell(13, input_dim=3))
rnn.add(Dense(3, activation='softmax', input_dim=13))

x = Input((3,), name='x')
y0 = Input((3,), name='y0')
s0 = Input((13,), name='s0')
yt = Input((11, 3), name='yt')
y = rnn(s0, ground_truth=yt, initial_readout=y0)

model = Model([x, y0, yt, s0], y)
model.compile('sgd', 'categorical_crossentropy')

npyt = np.ones((7, 11, 3))
npx = np.zeros((7, 3))
npy0 = np.ones((7, 3))
nps0 = np.ones((7, 13))
model.fit([npx, npy0, npyt, nps0], npyt)
y2 = model.predict([npx, npy0, npyt, nps0])

i also tried passing s0 in the initial_state parameter of rnn, but that breaks even earlier (pop from empty inputs_list in the rnn(...) call.

Hello!

I try to use tensorflow as backend, and got this error.

AttributeError Traceback (most recent call last)
in ()
1 model = Sequential()
----> 2 model.add(Seq2Seq(input_shape=(inpLen, inpDim), output_length=inpLen, output_dim=inpDim, hidden_dim = 100, dropout=0.3,depth=2))
3 model.add(Activation('softmax'))
4 model.compile(loss='categorical_crossentropy', optimizer='adam')

C:\Users\Public\Documents\Python2\lib\site-packages\seq2seq-0.1.0-py3.5.egg\seq2seq\models.py in Seq2Seq(output_dim, output_length, hidden_dim, depth, broadcast_state, inner_broadcast_state, teacher_force, peek, dropout, **kwargs)
180 input._keras_history[0].supports_masking = True
181 encoded_seq = dense1(input)
--> 182 encoded_seq = encoder(encoded_seq)
183 if broadcast_state:
184 states = encoded_seq[-2:]

C:\Users\Public\Documents\Python2\lib\site-packages\recurrentshop-0.0.1-py3.5.egg\recurrentshop\engine.py in call(self, x, mask)
569 break
570 if inbound_layers:
--> 571 self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
572 input_added = True
573 if input_added:

C:\Users\Public\Documents\Python2\lib\site-packages\keras\engine\topology.py in add_inbound_node(self, inbound_layers, node_indices, tensor_indices)
569 # creating the node automatically updates self.inbound_nodes
570 # as well as outbound_nodes on inbound layers.
--> 571 Node.create_node(self, inbound_layers, node_indices, tensor_indices)
572
573 def get_output_shape_for(self, input_shape):

C:\Users\Public\Documents\Python2\lib\site-packages\keras\engine\topology.py in create_node(cls, outbound_layer, inbound_layers, node_indices, tensor_indices)
153
154 if len(input_tensors) == 1:
--> 155 output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
156 output_masks = to_list(outbound_layer.compute_mask(input_tensors[0], input_masks[0]))
157 # TODO: try to auto-infer shape if exception is raised by get_output_shape_for.

C:\Users\Public\Documents\Python2\lib\site-packages\recurrentshop-0.0.1-py3.5.egg\recurrentshop\engine.py in call(self, x, mask)
329 else:
330 if self.uses_learning_phase:
--> 331 with learning_phase(0):
332 last_output_0, outputs_0, states_0, updates = rnn(self.step, x, initial_states, go_backwards=self.go_backwards, mask=mask, unroll=unroll, input_length=input_shape[1])
333 with learning_phase(1):

C:\Users\Public\Documents\Python2\lib\site-packages\recurrentshop-0.0.1-py3.5.egg\recurrentshop\engine.py in enter(self)
39
40 def enter(self):
---> 41 self.learning_phase_place_holder = _backend._LEARNING_PHASE
42 _backend._LEARNING_PHASE = self.value
43

AttributeError: module 'keras.backend.tensorflow_backend' has no attribute '_LEARNING_PHASE'

Hi,

I've just updated recurrentshop to work with keras 2 (to keras-2.0.3 recurrentshop-1.0.0) and now on

import recurrentshop I get an indentation error at line 42 of engine.py.

When I correct the indentation the next error is

File "/home/key/software/anaconda3/envs/tf3.5/lib/python3.5/site-packages/recurrentshop/engine.py", line 41 def _get_cells(): ^ SyntaxError: import * only allowed at module level

Could you please have a look?
Thx!

I'm trying to write a simple CNN-LSTM network. I'm having problems with the LSTM part, probably because of some misunderstanding on my side.

The network outline (batch size omitted):
image(shape=(299, 299, 3)) --> CNN --> vector(shape=(4096,))
vector(shape=(4096,)) --> LSTM --> sequence(shape=(40, 256))
The final output is a tensor of 40 time steps, each holding a vector of 256 elements, later used with some word embedding. The embedding part is not shown here, for simplicity.

from keras.layers import Input, Dense, Conv2D, Flatten
from keras.models import Model
from recurrentshop import LSTMCell, RecurrentModel

# Visual Model
# (this is a simple mockup, used to avoid long loading times. It will later 
# be replaced with a pre-trained inception model)
dummy_visual_input = Input(shape=(299, 299, 3), name='visual_input')
dummy_visual_conv = Conv2D(filters=48, kernel_size=[5, 5], name='visual_output')(dummy_visual_input)
dummy_visual_flat = Flatten()(dummy_visual_conv)
dummy_visual_output = Dense(units=1024)(dummy_visual_flat)
visual_model = Model(dummy_visual_input, dummy_visual_output, name='visual_model')


# LSTM (based on the example in the readout documentation)
num_of_lstm_outputs = 40
num_of_lstm_units = 256
word_embedding_size = 256

c_tm1 = Input(shape=(word_embedding_size,), name='initial_RNN_cell_state')
h_tm1 = Input(shape=(word_embedding_size,), name='initial_RNN_hidden_state')
readout_input = Input(shape=(word_embedding_size,), name='first_RNN_readout_input')
rnn_input = Input(name='RNN_input', tensor=visual_model(dummy_visual_input))

depth = 3  # stacking LSTMs
rnn_output, h, c = readout_input, h_tm1, c_tm1
for i in range(depth):
    rnn_output, h, c = LSTMCell(units=num_of_lstm_units)([rnn_output, h, c])
predictions = rnn_output

rnn_model = RecurrentModel(decode=True, output_length=num_of_lstm_outputs, 
            input=rnn_input, initial_states=[h_tm1, c_tm1], output=predictions, final_states=[h, c], 
            readout_input=readout_input, return_sequences=True, state_initializer='random_normal', 
            name='RNN_model')

final_model = Model(inputs=dummy_visual_input, outputs=rnn_model.output)

I get this error:

Traceback (most recent call last):
  File "/opt/pycharm-community-2016.2.3/helpers/pydev/pydevd.py", line 1580, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/opt/pycharm-community-2016.2.3/helpers/pydev/pydevd.py", line 964, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/gilboa/Research/image_tagging/dummy.py", line 152, in <module>
    main()
  File "/home/gilboa/Research/image_tagging/dummy.py", line 144, in main
    final_model = Model(inputs=dummy_visual_input, outputs=rnn_model.output)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 909, in output
    ' has no inbound nodes.')
AttributeError: Layer RNN_model has no inbound nodes.

In addition to understanding and solving the error above, I'd like to ask two more questions:

1. Why h_tm1, c_tm1, readout_input, and rnn_input must be an input layer? Does this mean we have to explicitly feed each one of them with data during training?
2. If we would like to initialize h_tm1 and/or c_tm1, how should it be done? Let's say we'd like to init them with a vector of ones, or with the output of another layer. Could you please give a code example?

Your work on this package (and seq2seq) is incredible, Fariz. Your contribution to the Keras community is invaluable! Many thanks to you and all the people making this happen!

When running the teacher force example from documentation with tf backend I get

tensorflow.python.framework.errors_impl.InvalidArgumentError: The node 'recurrent_sequential_1/while_1/Variable_1/Assign' has inputs from different frames. The input 'recurrent_sequential_1/while_1/Const_1' is in frame 'recurrent_sequential_1/while_1/recurrent_sequential_1/while_1/'. The input 'recurrent_sequential_1/while_1/Variable_1' is in frame ''.

from keras.layers import Activation, Input
from keras.models import  Model
from recurrentshop import RecurrentSequential, LSTMCell
import numpy as np

rnn = RecurrentSequential(readout='add', teacher_force=True, return_sequences=True)
rnn.add(LSTMCell(10, input_dim=10))
rnn.add(LSTMCell(10))
rnn.add(LSTMCell(10))
rnn.add(Activation('softmax'))


x = Input((7, 10))
y_true = Input((7, 10))  # This is where you feed the ground truth values

y = rnn(x, ground_truth=y_true)

model = Model([x, y_true], y)

model.compile(loss='categorical_crossentropy', optimizer='sgd')

# Training

X_true = np.random.random((32, 7, 10))
Y_true = np.random.random((32, 7, 10))

model.fit([X_true, Y_true], Y_true)  # Note that Y_true is part of both input and output


# Prediction

X = np.random.random((32, 7, 10))

model.predict(X)  # >> Error! the graph still has an input for ground truth..

zeros = np.zeros((32, 7, 10)) # Need not be zeros.. any array of same shape would do

model.predict([X, zeros])

tf version = 1.2.0
keras version = 2.0.5

Thanks for this great package! I've looked over your documentation and it certainly seems like this should work and I could use some help. I'm probably missing that I need to control the timestep in some way. I saw the nested code, but I think I'm getting tripped up on the Functional version. Here's the toy code:

import recurrentshop
import numpy
import keras.layers
from keras.layers import Dense, Input, LSTM
from keras.models import Model

prior_out = Input(batch_shape=(1,1,5),name='prior_out')
main_input = Input(batch_shape=(1,1,5),name='main_input')
added_layer = keras.layers.add([prior_out,main_input])
# hidden = LSTM(10,stateful=True,return_sequences=False,name='hidden')(added_layer)
hidden = recurrentshop.cells.LSTMCell(10)(added_layer)
out = Dense(5,name='out')(hidden)
rnn = recurrentshop.RecurrentModel(input=main_input, initial_states=prior_out, output=out, final_states=out)
rnn_input = Input(batch_shape=(1,1,5),name='rnn_input')
rnn_output = rnn(rnn_input)
model = Model(inputs=rnn_input,outputs=rnn_output)
model.compile(optimizer='adam',loss='mae')
print model.predict(numpy.random.random((3,1,5)),batch_size=1)

And I get this error:
ValueError: Operands could not be broadcast together with shapes (0,) (10,)

Thanks again for the help!

https://github.com/datalogai/recurrentshop/blob/bb9a600374b0b481aeea9f9230c197b9329a2033/recurrentshop/backend/theano_backend.py#L75

https://github.com/datalogai/recurrentshop/blob/bb9a600374b0b481aeea9f9230c197b9329a2033/recurrentshop/backend/theano_backend.py#L91

Should the above two functions be from numpy, e.g., np.expand_dims?

There are some tabs instead of spaces at https://github.com/datalogai/recurrentshop/blob/master/recurrentshop/engine.py#L459-L462.
This causes the error from the title at import.

I think the first and third lines do not function well:

https://github.com/datalogai/recurrentshop/blob/cf9b256f16a3912d4a0fbb870348588e4d516815/recurrentshop/engine.py#L1040-L1042

Please fix it, thanks!

Hi,

Can someone provide a minimal example for initializing the hidden state of a RNN with the output of a previous layer and with RecurrentShop?

Do you intend to work on meta-RL any time soon?

In order to better train an agent that interacts with a world, the basic idea involves building an RNN capable of keeping track of the particularities of its environment from one time step to the next.
https://arxiv.org/pdf/1611.05763v3.pdf

This involves concatenating the last action and its reward to the inputs for the next time step.

I am interested in a simpler case which would only need me to add the last output and its associated loss to the input at each time step. And I can see that being particularly inefficient in Keras, not to mention that with timesteps set to 1 there can be no significant BPTT.

For python3,
line 516 of https://github.com/datalogai/recurrentshop/blob/master/recurrentshop/engine.py
x = map(x.get, args)
should be:
x = list(map(x.get, args))
Otherwise, one would get the following error:

File "/usr/lib/python3.5/site-packages/recurrentshop-0.0.1-py3.5.egg/recurrentshop/engine.py", line 522, in call
if x[i] is not None:

TypeError: 'map' object is not subscriptable

I get the following error when trying to load a simple RecurrentContainer saved as json:
loaded_model = model_from_json(loaded_model_json) File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 213, in model_from_json return layer_from_config(config, custom_objects=custom_objects)
File "/usr/local/lib/python2.7/dist-packages/keras/utils/layer_utils.py", line 40, in layer_from_config custom_objects=custom_objects)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2582, in from_config process_layer(layer_data)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2560, in process_layer custom_objects=custom_objects)
File "/usr/local/lib/python2.7/dist-packages/keras/utils/layer_utils.py", line 35, in layer_from_config instantiate=False)
File "/usr/local/lib/python2.7/dist-packages/keras/utils/generic_utils.py", line 125, in get_from_module str(identifier))
ValueError: Invalid layer: RecurrentContainer

Is this like the decoder function in a seq2seq model when it's in the inference state. You feed the output back to next time input. Same thing will be done for a encoder where you then feed output in the previous state as well as input in the current state. I have seen this used inside tensorflow attention_decoder_fn_train.

Hi,

Is there a way to set dropout_W and dropout_U for an LSTMCell?

Thanks for the great work! It really help me a lot. But there is still one point that I can't figure out. What is the principle of readout and teacher forcing? How can we feeding the output(or ground truth) of RNN from the previous time step back to the current time step, by using the output as features together with the input of this step, or using the output as this step's cell state? I have read the code but still it confused me.o(╯□╰)o. Hoping someone can answer for me。

Hi I have try to run the seq2seq application. In that seq2seq lib calling the weight function from recurrentshop lib. But latest recurrentshop removed the weight funtion.

My question is if any other function are updated instead of weight function else how I can use weight function?

Please let me know...

@farizrahman4u

Does recurrentshop support a bidirectional RNN or a backward-directional RNN?
Dose it behaves like RNN in original keras package?
Is there any examples avaliable?

Thank you very much!

(Taken from https://www.tensorflow.org/versions/r0.11/tutorials/recurrent/index.html)

# Placeholder for the inputs in a given iteration.
words = tf.placeholder(tf.int32, [batch_size, num_steps])

lstm = rnn_cell.BasicLSTMCell(lstm_size)
# Initial state of the LSTM memory.
initial_state = state = tf.zeros([batch_size, lstm.state_size])

for i in range(num_steps):
    # The value of state is updated after processing each batch of words.
    output, state = lstm(words[:, i], state)

    # The rest of the code.
    # ...

final_state = state

The reason I ask is that using this style, we can take the ouput of a time step of an LSTM and feed it into the input of the next time step, like in this code :

  for step in range(num_iterations):
      with tf.device('/cpu:0'):
          patches = tf.image.extract_patches(images, tf.constant(patch_shape), inits+dx)
      patches = tf.reshape(patches, (batch_size * num_patches, patch_shape[0], patch_shape[1], num_channels))
      endpoints['patches'] = patches

      with tf.variable_scope('convnet', reuse=step>0):
          net = conv_model(patches)
          ims = net['concat']

      ims = tf.reshape(ims, (batch_size, -1))

      with tf.variable_scope('rnn', reuse=step>0) as scope:
          hidden_state = slim.ops.fc(tf.concat(1, [ims, hidden_state]), 512, activation=tf.tanh)
          prediction = slim.ops.fc(hidden_state, num_patches * 2, scope='pred', activation=None)
          endpoints['prediction'] = prediction
      prediction = tf.reshape(prediction, (batch_size, num_patches, 2))
      dx += prediction
dxs.append(dx)

(taken from https://github.com/trigeorgis/mdm/blob/master/mdm_model.py). Notice in this code that prediction, which is the output of the rnn at each timestep is added to dx, and dx is used earlier in this line
patches = tf.image.extract_patches(images, tf.constant(patch_shape), inits+dx)

  m = # Rucurrentshop containing model here
  cfg = m.get_config()
  m = Model.from_config(cfg)

If m contains RecurrentContainer, the last line will fail:

  File "/Users/anatolii/anaconda/envs/p3/lib/python3.5/site-packages/keras/engine/topology.py", line 2375, in from_config
    process_layer(layer_data)
  File "/Users/anatolii/anaconda/envs/p3/lib/python3.5/site-packages/keras/engine/topology.py", line 2353, in process_layer
    custom_objects=custom_objects)
  File "/Users/anatolii/anaconda/envs/p3/lib/python3.5/site-packages/keras/utils/layer_utils.py", line 35, in layer_from_config
    instantiate=False)
  File "/Users/anatolii/anaconda/envs/p3/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 16, in get_from_module
    str(identifier))
Exception: Invalid layer: RecurrentContainer

This prevents from loading a saved model from a file.

from keras import initializations, regularizers, constraints
from keras.utils.layer_utils import layer_from_config
sol:No initializations ??Modified to Initializers.
what about layer_from_config

I'm trying to train a recurrent model using SGD and it throws this error. Can't seem to figure out what's going on since the error does not come from Recurrent Shop itself.

Here's the code

Using TensorFlow backend.
Traceback (most recent call last):
  File "main.py", line 18, in <module>
    RWAModel.fit(train_data, train_labels)
  File "/home/abhai/keras/keras/engine/training.py", line 1458, in fit
    self._make_train_function()
  File "/home/abhai/keras/keras/engine/training.py", line 1002, in _make_train_function
    self.total_loss)
  File "/home/abhai/keras/keras/optimizers.py", line 398, in get_updates
    m_t = (self.beta_1 * m) + (1. - self.beta_1) * g
  File "/home/abhai/.local/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 793, in binary_op_wrapper
    y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name="y")
  File "/home/abhai/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 637, in convert_to_tensor
    as_ref=False)
  File "/home/abhai/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 702, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/abhai/.local/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 110, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/abhai/.local/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 99, in constant
    tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/home/abhai/.local/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 360, in make_tensor_proto
    raise ValueError("None values not supported.")
ValueError: None values not supported.

xxx@ubuntu:~/$ sudo python speed_test.py
Using Theano backend.
Traceback (most recent call last):
File "speed_test.py", line 1, in
from recurrentshop import*
File "build/bdist.linux-x86_64/egg/recurrentshop/init.py", line 1, in
File "build/bdist.linux-x86_64/egg/recurrentshop/engine.py", line 32, in
AttributeError: 'module' object has no attribute 'backend'

Somehow it gave me this error, any idea what's going on?

 ----> 1 from recurrentshop.cells import *
 
 ImportError: No module named 'recurrentshop.cells'

There is no error with import recurrentshop. How can I fix it? Thank you.

The tests for all three cell types currently fail at assert expected_output_shape == actual_output_shape.

I run the example code in ./examples/speed_test.py

the code here is:

rc = RecurrentContainer(input_length=input_length, unroll=unroll)
for _ in range(depth):
	rc.add(rnn_cell(dim, input_dim=dim))

But it raise TypeError in terminal with:

File "anaconda/lib/python3.6/site-packages/keras/engine/topology.py", line 277, in __init__
    raise TypeError('Keyword argument not understood:', kwarg)
TypeError: ('Keyword argument not understood:', 'input_length')

So can anyone help?