Residual block with or without relu to identity connection? about tcn HOT 3 CLOSED

So am I correct that the forward pass performs

• The nn.Sequential stuff - so Conv-Relu-Dropout, Conv-Relu-Dropout
• And then adds the output of the above to the input
• And then finally another ReLU (as seen here https://github.com/locuslab/TCN/blob/master/TCN/tcn.py#L45)

I am only a bit confused of this "outer-ReLU" part since it essentially means you do ReLU twice directly in succession (once at the end of Sequential, and then after you add to the input). And so I was wondering if that makes anything worse to apply a non-linearity almost "twice". But good to know that all the variations basically work?

The reason I am asking in detail is because I was looking for simple residual variants that work well and do not need batchnorm in these kinds of networks. And it seems weird to me to always add something in each layer that is positive (since it was ReLU-d) - do activations not get too large with many layers? Same with always applying relu to the shortcut connection, I thought the point here is to leave the shortcut connection alone so information always gets propagated forward.

So only having Conv at the end and adding that to the input (more like Resnet), and then just outputting that seemed more intuitive to me.

from tcn.

Also how does it make sense to apply ReLU after convolution, add the output to the identity connection, and then perform ReLU again?
Are you following any kind of standard Resnet architecture?

from tcn.

Hi! Regarding your questions:

1. It is not a mistake. The ReLU in the block corresponds to the ReLU in nn.Sequential (i.e., https://github.com/locuslab/TCN/blob/master/TCN/tcn.py#L31). There is an outer-ReLU, too.

2. You can also certainly remove the self.relu2 (in practice I don't find much difference in performance). Then it would be more like a "standard ResNet". But in a high level, the residual blocks are just there to make sure the gradient can flow properly even a very deep network, so I don't think using a ReLU after the identity connection would be a big problem.

Let me know if this helps!

from tcn.