Comments (4)
Consider an LSTM stack with 5 LSTM cells. Such a stack would have 5 * 2 = 10 hidden states, 2 per each cell. When state synchronization is enabled, there will be only 2 states instead of 10. All the 5 cells share these 2 states. Hence the notation h(time, layer) does not make sense, since the "h" is common for all the layers. i.e, h(time, l1) = h(time, l2) = h(time, l3)....
from recurrentshop.
Sorry for the delay in getting back to this, I haven't had the time to sit down and be thorough about it until now.
So, (a) is what a normal 3-hidden-layer RNN looks like. (b) is what your state synchronization code does, as far as I can tell. I ran this script to make a graph that I could inspect in TensorBoard. If you look in the While box in TensorBoard, you can see the SimpleRNNCell calculations - the output from the tanh following add_1 is what I have notated as h_t^1 in the picture above. Notice how it goes into two different MatMul operations, which correspond to the W_2 and U_2 matrix multiplies in the level-2 SimpleRNNCell. The same thing happens with the output of add_3 - the h_t^2 vector gets matrix multiplied by two different matrices, W_3 and U_3, whose outputs get added together. So, effectively, the equations that are getting computed are
h_t^1 = tanh( W_1 * x_t + U_1 * h_{t-1}^3 + b_1 )
h_t^2 = tanh( W_2 * h_t^1 + U_2 * h_t^1 + b_2 ) = tanh( (W_2 + U_2) * h_t^1 + b_2 )
h_t^3 = tanh( W_3 * h_t^2 + U_3 * h_t^2 + b_3) = tanh( (W_3 + U_3) * h_t^2 + b_2 )
I confess, I have a very difficult time interpreting the TensorBoard graph visualization for a dynamic RNN; there are plenty of bits and pieces which I don't understand. So if I am mistaken here, I'm happy to be corrected.
But, is (b) the correct / desired behavior? Or do you want (c)? Or is state synchronization meant to do something else? If you could draw a graph illustrating the intended behavior, or point to a reference in the literature that explains this, it would be greatly appreciated! Thanks.
from recurrentshop.
** b ** is the intended behavior. It would make sense for custom rnn cells, where the output of the a cell is different from its state. This kind of behavior is mentioned in the Seq2seq paper.
from recurrentshop.
Ah, okay. Thanks for the clarification!
from recurrentshop.
Related Issues (20)
- Dropout in Readout Model
- RecurrentModel without Cell
- AssertionError: Required 2 inputs, received 3.
- ImportError: cannot import name 'RecurrentSequentiaĺ'
- Error when combining teacherforce/readout with return_states
- Setting states for rnn forecasting
- update version HOT 1
- TypeError: can only concatenate tuple (not "int") to tuple HOT 3
- RNN with missing inputs
- Installation issue
- Decoder variable output_length? HOT 2
- How to transition between readout and teacher forcing?
- load model error with attention model
- Tensorflow2.0 support?
- AttributeError: 'Node' object has no attribute 'output_masks' HOT 1
- NameError: name 'K' is not defined HOT 1
- NameError: name 'activations' is not defined HOT 2
- NameError: name 'activations' is not defined HOT 2
- Error fix
- ImportError: cannot import name 'tensorflow_backend' HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from recurrentshop.