mgermain / made Goto Github PK

ubgpu@ubgpu:~/github/MADE$ sudo python -u trainMADE.py --name mnist_from_paper binarized_mnist 0.01 0 -1 32 Full 300 100 30 False 0 adagrad 0 [8000,8000] 1234 False Output False hinge Orthogonal 0
Using gpu device 0: GeForce GTX 970

Resuming experiment (mnist_from_paper).

Loading dataset [binarized_mnist] ... (Dim:784 Train:50000 Valid:10000 Test:10000) DONE in 0.3176 seconds.

Initializing MADE ... DONE in 947.4197 seconds.

Traceback (most recent call last):
File "trainMADE.py", line 293, in
load_model_params(model, save_path_experiment)
File "trainMADE.py", line 230, in load_model_params
model.update_rule.parameters[i].set_value(param.get_value())
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/var.py", line 132, in set_value
self.container.value = value # this will copy a numpy ndarray
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 343, in set
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 94, in filter_inplace
strict, old_data)
ValueError: ('object too deep for desired array', 'Container name "mean_square_grad_b0"')
ubgpu@ubgpu:~/github/MADE$

Hi,

Can you explain why this line is computing logp please?

https://github.com/mgermain/MADE/blob/master/MADE/made.py#L94

I just tried to run your code like the example in the README file:
nobody:MADE-ICML2015 apple$ python -u trainMADE.py dna 1e-5 0.95 -1 -1 Full 300 100 30 False 0 adadelta 0 [500] 1234 False Output False hinge Orthogonal 0

DEPRECATION WARNING: softsign was moved from theano.sandbox.softsign to theano.tensor.nnet.nnet

Resuming experiment (ef0f220f13f115d34798a5f7e16e8c539e4eaee42564d07ed8f893b7fadaa8a0).

Loading dataset [dna] ... (Dim:180 Train:1400 Valid:600 Test:1186) DONE in 0.0163 seconds.

Initializing MADE ...

Traceback (most recent call last):
File "trainMADE.py", line 287, in
model = build_model(dataset, trainingparams, hyperparams, hyperparams['hidden_sizes'])
File "trainMADE.py", line 140, in build_model
mask_distribution=hyperparams['mask_distribution'])
File "/Users/apple/Desktop/Thesis/MADE-ICML2015/MADE/made.py", line 129, in init
on_unused_input='ignore') # ignore for when dropout is absent
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/function.py", line 326, in function
output_keys=output_keys)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 449, in pfunc
no_default_updates=no_default_updates)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 219, in rebuild_collect_shared
cloned_v = clone_v_get_shared_updates(v, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 93, in clone_v_get_shared_updates
clone_v_get_shared_updates(i, copy_inputs_over)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 96, in clone_v_get_shared_updates
[clone_d[i] for i in owner.inputs], strict=rebuild_strict)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/gof/graph.py", line 238, in clone_with_new_inputs
new_inputs[i] = curr.type.filter_variable(new)
File "/Users/apple/miniconda2/lib/python2.7/site-packages/theano/tensor/type.py", line 235, in filter_variable
self=self))
TypeError: Cannot convert Type TensorType(float32, matrix) (of Variable Subtensor{int64:int64:}.0) into Type TensorType(float64, matrix). You can try to manually convert Subtensor{int64:int64:}.0 into a TensorType(float64, matrix).

I am having trouble understanding your loss function defined here as:

pre_output = self.layers[-1].lin_output
log_prob = -T.sum(T.nnet.softplus(-target * pre_output + (1 - target) * pre_output), axis=1)
loss = (-log_prob).mean()

It looks like the softplus arg simplifies to 1 - 2 * target * pre_output, but does this form have better numerics? Why is the softplus used here?

How does this loss relate to eqn (5) of your paper, which looks like a standard binary cross entropy?