polzounov / gatenet Goto Github PK

Right now we have a single module type throughout the graph, create a generation function to allow for more complicated structure

When using the save/load functionality of the meta-optimizer there are some issues for module-wise sharing. It seems that under the l2l/networks.py file, in the save function, the to_save variable returns a dictionary of modules rather than a single module.

The following error occurs:

  File ".../l2l/networks.py", line 50, in save
    variables = snt.get_variables_in_module(network)
  File .../python/modules/util.py", line 77, in get_variables_in_module
    return module.get_variables(collection=collection)
AttributeError: 'tuple' object has no attribute 'get_variables'

Do this fix and any others that need to be done to get the save/load functionality working.

Right now we assume the the graph definition will always have the same number of modules per layer. This will cause issues in displaying the gates (determine_gates in graph.py), and possibly others.

Two solutions may work here:

1. Allow differing number of modules per layer by modifying determine_gates (different visualization method)
2. (More likely) Raise errors when the number of modules is incorrect

Few things need to be done to clean up the Layer class:

• Instead of defining the shapes of weights for the modules in Layer (lines 28-29 graph/layer.py), encapsulate the creation of the weights into Module
• The modules should only take in input shape, output shape, and module type
• Assuming that we only use Addition or Identity sublayers Module and Layer output shapes are the same

Make the code more readable and clear.

Also: assigning self.gates = gates_normalized could be improved for calling self.gates later on

The structure of Sonnet and our structure share many similarities.
It should be easy to integrate with Sonnet and get some of Sonnet's features without needing to develop it ourselves.

Integrate Sonnet by subclassing our modules to their snt.AbstractModule, this gives us variable sharing and easy integration with the code from learning to learn

Right now this functions is built to work with 2d inputs only.
Currently a hack is used to convert 4d inputs into 2d, process the modules, then convert each output back into 4d.
Make this work properly with all inputs.

Right now the meta optimizer has to do all of the reshaping to get gradients and deltas in the right shapes in order to work with the RNNs.
Instead of this write a wrapper that works for any RNN that can take in a list of (grads, vars) and output the same type of list.

What we have now in the meta optimizer (dirty!)

        with tf.name_scope('meta_optmizer_step'):
            for optimizer in self._optimizers
                list_deltas = []
                with tf.name_scope('deltas'):
                    OptimizerType = optimizer.rnn # same as optimizer[0]
                    flat_helper = optimizer.flat_helper # same as optimizer[1]

                    # Flatten the gradients from list of (gradient, variable) 
                    # into single tensor (batch_size, k)
                    flattened_grads = flat_helper.flatten(gradients)
                    # Run step for the RNN optimizer
                    flattened_deltas = OptimizerType(flattened_grads)
                    # Get deltas back into original form
                    deltas = flat_helper.unflatten(flattened_deltas)
                    list_deltas.append(deltas)

What we want:

        with tf.name_scope('meta_optmizer_step'):
            for optimizer in self._optimizers
                list_deltas = []
                with tf.name_scope('deltas'):
                    OptimizerType = optimizer.rnn 

                    # Both gradients and deltas are lists of tuples of (gradient, variable)
                    # Run step for the RNN optimizer
                    deltas = OptimizerType(gradients)
                    list_deltas.append(deltas)

Add support for the meta-training problem setup.
Eg. like the following (image from OPTIMIZATION AS A MODEL FOR FEW-SHOT LEARNING)