RNN/LSTM Baselines? about tcn HOT 7 CLOSED

Thank you for your interest in our work! They are not available in this repo; but since we designed TCN as a drop-in replacement for RNNs, you can simply replace lines like https://github.com/locuslab/TCN/blob/master/TCN/copy_memory/copymem_test.py#L70
with an RNN/LSTM/GRU, etc.

We did provide hyperparameters for LSTMs in the appendix of the paper :-)

As another source, pytorch examples provide tutorial code of LSTM for word-level language modeling. It is easy to change it (slightly) and use it for character-level LM as well: https://github.com/pytorch/examples/tree/master/word_language_model

from tcn.

@jerrybai1995 Thank you for your excellent work! I used https://github.com/pytorch/examples/tree/master/word_language_model and follow the hyperparameters in your paper(nlayer=3,hidden=700,dropout=0.4,clip=0.3, bias=1.0, sgd, lr=30, emb.=700)，but i can't reproduce 78.93(ppl) on PTB, my result was valid ppl: 89.16, est ppl:85.50.
@millerjohnp What is your result?
Thank you very much!

from tcn.

Did you try tying the input embedding weights with the decoder weights? I think it’ll help.

from tcn.

@jerrybai1995 yes, i have used tied weight.

from tcn.

@jerrybai1995 It is very difficult to reproduce LSTM result. I have seen many papers that report results are different. Can you give a way to reproduce your paper? thank you very much!

from tcn.

Try SGD lr=20, and decay the learning rate when the validation error plateaus. Use a batch size of 16 or 20. Basically, I believe the original pytorch example’s README provides parameters that could achieve about 80 ppl, and it’s mainly a problem of hyperparameter tuning.

from tcn.

@jerrybai1995 Thank you for your excellent work! I used https://github.com/pytorch/examples/tree/master/word_language_model and follow the hyperparameters in your paper(nlayer=3,hidden=700,dropout=0.4,clip=0.3, bias=1.0, sgd, lr=30, emb.=700)，but i can't reproduce 78.93(ppl) on PTB, my result was valid ppl: 89.16, est ppl:85.50.
@millerjohnp What is your result?
Thank you very much!

@jerrybai1995 The test ppl is 85.5(bias=0, there was a problem with the previous reply) ,and when set bais=1 in LSTM use the following code:

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)
        for name, param in self.rnn.named_parameters():
            if 'bias' in name:
                param.data.fill_(1)

the test ppl is 90 that is worse than before.
Also, i tried lr=20,and batch size=20, the result is almost the same, my pytorch version is 0.4.0

from tcn.