gchrupala / funktional Goto Github PK

Doesn't seem to work properly in the current version of then code.

I believe it would be a good idea to make it possible to refer by name to:

• Network parameters: for regularization or just analysis
• Return values: RNNs return "hidden states", "r states", "z states" and who knows what in the future

This would make experimentation easier. Functions like "last" or anything that has to do with prediction could query for "hidden states", functions related to regularizers could query for "weights" or for "activation vectors" and so on. I think this is would be nicer than say I want regularize(params[10]) or last(output[0]).

Idea based on "Net2Net: Accelerating Learning via Knowledge Transfer" http://arxiv.org/abs/1511.05641
The current grow function should initialize the newly added layer to one implementing an identity function. For a GRU, assuming non-negative inputs, and a relu or clipped relu activation, and the following definiton of the layer:

def GRU(W,U,Wz,Uz,Wr,Ur,xt,htm1):
    r = sigmoid(dot(xt,Wr)+dot(htm1, Ur))
    z = sigmoid(dot(xt,Wz)+dot(htm1, Uz))
    htilde = rectify(dot(xt,W)+dot(r*htm1, U))
    h = (1-z) * htm1 + z*htilde
    return h

we could set:

• W - Identity
• U - Zero
• Wz - Identity + 2 (or any number ensuring z is close to 1)
• Uz - Identity + 2
• Wr - random
• Ur - random

See keras-team/keras#917

The negative values are being propagated from the initial state via
the linear interpolation formula of the GRU.
The initial state is initialized to all zeros, but these parameters
are also learnable so they can become negative.

It's confusing to have these negative numbers pop up. The initial state could be passed through the layer activation also, but if we initialize to 0 and use a clipped_relu, they become all dead.

.fit should do all the mutation, while ._transform should just read the values in IdTable

And it should be possible to have a single layer.

http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization