State Synchronization about recurrentshop HOT 4 CLOSED

Consider an LSTM stack with 5 LSTM cells. Such a stack would have 5 * 2 = 10 hidden states, 2 per each cell. When state synchronization is enabled, there will be only 2 states instead of 10. All the 5 cells share these 2 states. Hence the notation h(time, layer) does not make sense, since the "h" is common for all the layers. i.e, h(time, l1) = h(time, l2) = h(time, l3)....

from recurrentshop.

Sorry for the delay in getting back to this, I haven't had the time to sit down and be thorough about it until now.

So, (a) is what a normal 3-hidden-layer RNN looks like. (b) is what your state synchronization code does, as far as I can tell. I ran this script to make a graph that I could inspect in TensorBoard. If you look in the While box in TensorBoard, you can see the SimpleRNNCell calculations - the output from the tanh following add_1 is what I have notated as h_t^1 in the picture above. Notice how it goes into two different MatMul operations, which correspond to the W_2 and U_2 matrix multiplies in the level-2 SimpleRNNCell. The same thing happens with the output of add_3 - the h_t^2 vector gets matrix multiplied by two different matrices, W_3 and U_3, whose outputs get added together. So, effectively, the equations that are getting computed are

h_t^1 = tanh( W_1 * x_t + U_1 * h_{t-1}^3 + b_1 )
h_t^2 = tanh( W_2 * h_t^1 + U_2 * h_t^1 + b_2 ) = tanh( (W_2 + U_2) * h_t^1 + b_2 )
h_t^3 = tanh( W_3 * h_t^2 + U_3 * h_t^2 + b_3) = tanh( (W_3 + U_3) * h_t^2 + b_2 )

I confess, I have a very difficult time interpreting the TensorBoard graph visualization for a dynamic RNN; there are plenty of bits and pieces which I don't understand. So if I am mistaken here, I'm happy to be corrected.

But, is (b) the correct / desired behavior? Or do you want (c)? Or is state synchronization meant to do something else? If you could draw a graph illustrating the intended behavior, or point to a reference in the literature that explains this, it would be greatly appreciated! Thanks.

from recurrentshop.

** b ** is the intended behavior. It would make sense for custom rnn cells, where the output of the a cell is different from its state. This kind of behavior is mentioned in the Seq2seq paper.

from recurrentshop.

Ah, okay. Thanks for the clarification!

from recurrentshop.