Test data is being generated but not used.

Using model(X_test) yields an error

Great code, thanks.

Minor issue probably but I'm getting the following warning:

//anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))

torch.__version__ == '1.2.0'

If I figure out the solution I'll post here.

Comment about stateful LSTM is incorrect because model.hidden is not passed in LSTM. It only stores the hidden state. Implemented LSTM is stateless.

For example:
Stateful LSTM uses hidden states:

rnn = nn.LSTM(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20) c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(input, (h0, c0))

Stateless LSTM doesn't use hidden states:

rnn = nn.LSTM(10, 20, 2)
input = torch.randn(5, 3, 10)
output, (hn, cn) = rnn(input)

Hi Jessica,
I am wonder that if the noise of AR(5) generated data is set to 1.0 (according to your blog). LSTM is very hard (impossible) to learn this generated data.
Could you explain how to debug LSTM with generated AR data (noise=1)? Thanks!

You should first transpose both matrices. The best way to track the problem is using a time series with the values 1, 2, 3, etc. and you will find the error.

So, you should use:
X_train = X_train.t().view(input_size, -1, 1)
X_test = X_test.t().view(input_size, -1, 1)

instead of:
X_train = X_train.view([input_size, -1, 1])
X_test = X_test.view([input_size, -1, 1])

If you set per_element = False (line 19) the NN will fail because the batch_size=num_train assumption will lead to a dimensionality error with self.lstm call on line 77.

One way to fix this would be to set batch_size = num_train / input_size and pass in batch_size=batch_size... but it's also possible to do an arbitrary batch size, trick would again be to get the .view() on line 77 to satisfy the self.lstm dimensionality.

You also will need to transform the loss function on line 104 so that y_train dimensions match y_pred.