Demo request about 3d-deformable-convolutions HOT 2 OPEN

Hey @MjdMahasneh,

I managed to get the deformable convolutional layer working. Or at least I think I did.

Currently trying to build an auto-encoder for training on medical ultrasonic 3d images. (grayscale)
The results I get from the auto-encoder are not satisfying at all.

It seems as if my decoder is kinda broken. The reconstruction of the ultrasonic image from latent space does not work.
Not sure what I am doing wrong.

Here is my decoder (some of it I left out):

    def __init__(self, num_classes):
        super(newDCNModelDecoder, self).__init__()
        ...
        ...
        ...
        self.maxpool2_transpose = torch.nn.MaxUnpool3d((2, 2, 2))
        self.conv2_transpose = self.create_dconv_layer_transpose(64, 32) #DEFORMABLE LAYER
        self.maxpool1_transpose = torch.nn.MaxUnpool3d((2, 2, 2))
        self.conv1_transpose = self.create_conv_layer_transpose(in_c=32, out_c=1)

    @staticmethod
    def create_conv_layer_transpose(in_c, out_c):
        return torch.nn.Sequential(
            torch.nn.LeakyReLU(),
            torch.nn.ConvTranspose3d(in_c, out_c, kernel_size=(3, 3, 3))
        )

    @staticmethod #THIS IS DEFORMABLE
    def create_dconv_layer_transpose(in_c, out_c):
        return torch.nn.Sequential(
            torch.nn.LeakyReLU(),
            DeformConv3D(in_c, out_c, kernel_size=(3, 3, 3), padding=2)
        )

    def forward(self, encoded, output_size_before_1, output_size_before_2, indices_maxpool1, indices_maxpool2):
        ...
        ...
        ...
        out = self.maxpool2_transpose(out, indices_maxpool2, output_size=output_size_before_2)
        out = self.conv2_transpose(out)
        out = self.maxpool1_transpose(out, indices_maxpool1, output_size=output_size_before_1)
        out = self.conv1_transpose(out)

        return out

As an example:
35000 patches for training
after 94 epochs I get the following output (left: original, right: predicted) [rainbow colorbar from matplotlib]

As you can see:
on the bottom and on right edge is no learning process. (just plain)
All predicted images look like that on the bottom and right edge.

Replacing the deformable layer with a "normal" conv layer everything works fine. --> no plain area on the right and bottom

What am I doing wrong?
Any idea?

from 3d-deformable-convolutions.

@DezzardHD hello, the snippet is not clear enough to be able to anticipate what went wrong. I suggest you share the full code of 1-the working network, and 2-the malfunctioning network.

Additionally, could you provide some context on the task? Loss function? Input and output sizes?

Apart from that, from what I can see in your code, I suspect that the behavior might be caused by stacking of un-pooling layers and transposed conv layers. To be honest the code could use some editing for readability as well (e.g. you seem to be using the terms deconv and transpose_conv as if they refer to different things. To the best of my knowledge, these two terms refer to the same semantic)

Anyhow, i could try debug the snippet with you if you post more info as I explained above.

best :)

from 3d-deformable-convolutions.