masa-su / tars Goto Github PK

Hi,
Thank you for sharing.
I have a problem. I want to add convolutional operation to the model. However, it performs poorly. Below is my generator part code:
z = InputLayer((None,z_dim))
network = DenseLayer(z,num_units=1024,nonlinearity=activation)
network = DenseLayer(network,num_units=128 * 7 * 7,nonlinearity=activation)
network = lasagne.layers.ReshapeLayer(network, ((-1,128,7,7)))
network=lasagne.layers.TransposedConv2DLayer(network,num_filters=64,\ filter_size=(3,3),stride=(2,2),output_size=(14,14),crop='same',nonlinearity=rectify)
p_mean=lasagne.layers.TransposedConv2DLayer(network,1,\ filter_size=(3,3),stride=(2,2),output_size=(28,28),crop='same',nonlinearity=sigmoid)
p = Deterministic(p_mean,given=[z]) #p(x|z)
Other codes are not changed except for some minor changes to avoid error.
However, the generation is poor.
So, how can I add conv operation, is there something wrong?
Thank you!

公開日をいつにしようか問題．
・個人的には来月頭，早ければ今月末には公開したい．
・公開に向けてやることリストとスケジュールを考える必要がある．
・何人かに手伝ってもらう予定

https://github.com/masa-su/Tars/blob/master/Tars/model/_mvae.py#L315-L327

VRNNを実装したが，現在うまく動作しない．
おそらく，学習部分か推論部分にバグがあると思われる．

VAEやGANの親クラスを作成した方がいいかどうか検討中（現在はobjectを継承）．
distributionのBernoulliやGaussianはDistributionを継承しているので，そちらに合わせてModelクラスを作ってもいいかもしれない．

Currently temperature is cast to numpy.float32 in the attempt to fix #59:

class ConcreteSample(GumbelSample):
    ...
    def __init__(self, temp=0.1, seed=1):
        super(ConcreteSample, self).__init__(seed=seed)
        self.temp = np.float32(temp)

It would be more preferable if we could use floatX such as lasagne.utils.floatX(temp) while making sure all test cases pass on different machines.

I've implemented the JMVAE models on v0.0.2.
https://arxiv.org/abs/1611.01891
I'm going to implement them on v0.0.3. It is necessary to be aware that some specifications have changed from v0.0.2 to v.0.0.3.

In the current implementation, we implement prior distributions as the Distribution_sample class but I believe that it is better to implement them as the Distribution_model class.

This issue is quite related to #54 .

def test_consistency(self):
    # Ensure that returned values stay the same when setting a fixed seed.
    mu, beta = 0, 1
    gumbel_sample = Gumbel_sample(temp=0.01, seed=1234567890)
    actual = TestGumbelSample.get_sample(mu, beta, gumbel_sample, 5)
    desired = [
        1.7462246417999268,
        0.2874840497970581,
        0.9974965453147888,
        0.3290553689002991,
        -1.0215276479721069
    ]
    assert_array_almost_equal(actual, desired, decimal=15)

Added test cases ensuring the consistency of sampling methods, and they all passed on the local environment.
However, if we set the decimals more than 6 or 7, test cases other than Gaussian and Gumbel distributions will fail when run on other machines including servers.

Example log on travis ci:

_____________________ TestBernoulliSample.test_consistency _____________________
self = <Tars.tests.test_distribution_samples.TestBernoulliSample testMethod=test_consistency>
    def test_consistency(self):
        # Ensure that returned values stay the same when setting a fixed seed.
        mean = 0.5
        bernoulli_sample = Bernoulli_sample(temp=0.1, seed=1234567890)
        actual = TestBernoulliSample.get_sample(mean, bernoulli_sample, 5)
        desired = [
            0.9999971508356551,
            0.9101269246280252,
            0.0248156363467501,
            0.3538078165724645,
            0.1615775890919983
        ]
>       assert_array_almost_equal(actual, desired, decimal=15)
E       AssertionError: 
E       Arrays are not almost equal to 15 decimals
E       
E       (mismatch 20.0%)
E        x: array([ 0.999997150839051,  0.910126924628025,  0.02481563634675 ,
E               0.353807816572465,  0.161577589091998])
E        y: array([ 0.999997150835655,  0.910126924628025,  0.02481563634675 ,
E               0.353807816572465,  0.161577589091998])
Tars/tests/test_distribution_samples.py:97: AssertionError

KL距離をアニーリングできるようにしたい．
今の実装だと，KLに重み付けできないので，theano変数でパラメータを設定する必要がある．
スケジューリングをVAEクラス外部で行うか内部で行うかは要検討．

http://arxiv.org/pdf/1510.02777v2.pdf
http://www.stat.ucla.edu/~ywu/ABP/main.html#exp1

綺麗な画像が生成できそう．

DRAW [Gregor+ 2015]を実装したい．
・https://github.com/skaae/lasagne-draw を参考にすれば実装できそう．

To make sure that the sampling/log_likelihood methods return proper values.

今のVAEクラスで問題ないが，入力が3次元なので，次の問題がある．
・t_repeatが2次元or4次元しか対応していない．
・distributionのmean_sum_samplesが2次元or4次元しか対応していない．

Such as Distribution, Bernoulli, and Gaussian.

https://github.com/jych/cle/blob/master/cle/cost/__init__.py
これに書かれている分布は全て実装したい．
具体的には
・GMM
・BiGMM
・BiGauss（実装済みだが，より汎用的にしたい）

かなりごちゃごちゃしてきたので，思い切って整理したい

• model classの整理
  • あまり設計として良くない（特にtrainとtest）
  • モデルが大量に増えてきた中でいまいち拡張性がない
  • qとpを分けるべきか？

We would like to implement generative adversarial nets as "implicit distribution" as follows:

z = InputLayer((None, z_dim))
g  = batch_norm(DenseLayer(z,num_units=512,nonlinearity=activation))
g  = batch_norm(DenseLayer(g,num_units=512,nonlinearity=activation))
g_mean = DenseLayer(g,num_units=x_dim,nonlinearity=sigmoid)
g = Deterministic(g_mean,given=[z]) #p(x|z)

x = InputLayer((None, x_dim))
d_0  = DenseLayer(x,num_units=512,nonlinearity=leaky_rectify)
d_1  = DenseLayer(d_0,num_units=512,nonlinearity=leaky_rectify)
d_mean = DenseLayer(d_1,num_units=1,nonlinearity=sigmoid)
d = Bernoulli(d_mean,given=[x])

p = ImplicitDistribution(mean=g, loss=d)

すべてのメソッドにdocstringを書く

• 書き方はsklearnをパクる予定

I've implemented the distribution class "Multilayer" which enable to model multi-latent layers.
But it can't be used in the current version, so we should fix it.

Currently abstract methods in Distribution are not implemented in its child classes, but in the derived classes of Distribution_sample even though it calls them in other methods such as sample_given_x and log_likelihood_given_x, assuming its child classes inherit Distribution_sample.

class Distribution(object):
    __metaclass__ = ABCMeta
    ...
    def sample_given_x(self, x, repeat=1, **kwargs):
        ...
        return [x, self.sample(*tolist(mean))]

    def log_likelihood_given_x(self, samples, **kwargs):
        ...
        return self.log_likelihood(sample, *tolist(mean))
    ...
    @abstractmethod
    def sample(self):
        pass

    @abstractmethod
    def log_likelihood(self):
        pass

class Gaussian_sample(Distribution_sample):
    """
    Gaussian distribution
    p(x) = \frac{1}{\sqrt{2*\pi*var}} * exp{-\frac{{x-mean}^2}{2*var}}
    """

    def sample(self, mean, var):
        """
        Paramaters
        ----------
        mean : Theano variable, the output of a fully connected layer (Linear)
        var : Theano variable, the output of a fully connected layer (Softplus)
        """

        eps = self.srng.normal(mean.shape, dtype=mean.dtype)
        return mean + T.sqrt(var) * eps

    def log_likelihood(self, samples, mean, var):
        """
        Paramaters
        --------
        sample : Theano variable
        mean : Theano variable, the output of a fully connected layer (Linear)
        var : Theano variable, the output of a fully connected layer (Softplus)
        """

        loglike = self._gaussian_like(samples, mean, var)
        return mean_sum_samples(loglike)

    def _gaussian_like(self, x, mean, var):
        c = - 0.5 * math.log(2 * math.pi)
        _var = var + epsilon()  # avoid NaN
        return c - T.log(_var) / 2 - (x - mean)**2 / (2 * _var)

class Gaussian(Gaussian_sample, Distribution_double):

    def __init__(self, mean_network, var_network, given, seed=1):
        Distribution_double.__init__(self, mean_network, var_network, given)
        super(Gaussian, self).__init__(seed=seed)
        self._set_theano_func()

This is a bit confusing, and it might be better to change the class inheritance structure to resolve this. For example, we could let Distribution have a variable of Gaussian_sample in which sample and log_likelihood methods are defined, and call them in sample_given_x and log_likelihood_given_x methods. Also, we should use camelcase for class names.