Having troble reproducing "ours-FC" about nodis HOT 14 CLOSED

Oh, I will try to train SGCLS model based on this model and test it again.

from nodis.

I'm still not able to reproduce the result, I got 0.67 on R@100 for the PREDCLS task using the trained SGCLS model.

from nodis.

Hi,
you should not replace the Bi-LSTM in ode block by Linear layer. But you can replace the LSTM before ODE-block. Btw, how about R@50?

from nodis.

Sorry, I'm still not clear why the Bi-LSTM in ode block cannot be replaced by Linear layer. I thought this is how the paper describes "ours-FC":

(1) One in which the function used inside the ODE layer is taken to be a linear layer (ours-FC)

Apparently both the version using a linear layer inside the ODE layer (ours-FC) and the one using the bidirectional LSTM (ours-LSTM) perform very similar.

It is sufficient to use a simple linear layer, since the ODE layer is so powerful that it produces similar outputs to those of a function based upon a more complicated and slower bidirectional LSTM.

I guess by " replace the LSTM before ODE-block", you mean:

The visual vectors and semantic vectors of object pairs can be pre-processed by three methods before the P-ODE: FC-Layer: The (3 · 512)-dimensional visual vectors and 400-dimensional semantic vectors are forwarded into two independent fully connection layers that both have 512 neurons. Then, the outputs are concatenated together as 1024-dimensional representation vectors for the P-ODE.

I failed to record the result on R@50 because I thought it was not a satisfying result, but I can reproduce it soon (in about 10 minutes).

from nodis.

ohh, i got you.. That part is written by another author, he misunderstood me. ours-FC means using FC before ODE-block. Then different network structures will provide different representations for the ODE block.

from nodis.

OK, thank you, that makes sense. I will experiment with that setting to see if I can reproduce the result. Btw, the R@50 was 0.648, R@20 was 0.564.

from nodis.

Are these scores (Re@50=0.648) using FC in ODE-block?

from nodis.

Yes

from nodis.

And this the SGCLS results using FC in ODE-block.

from nodis.

Ok that is reasonable, thanks

from nodis.

Hi, as you suggested, I change the LSTMs before the ode block to FC-layers for reproducing "ours-FC", I defined them as follows:

    self.linear_visual = nn.Linear(1536, 512)

    self.linear_semantic = nn.Linear(400, 512)

    self.odeBlock = odeBlock(odeFunc1(bidirectional=True))

And applied them in the forward function of NODIS class:

    perm, inv_perm, ls_transposed = self.sort_rois(rel_inds[:, 0].data, None, union_rois[:, 1:])

    visual_input = PackedSequence(x_visual[perm], torch.tensor(ls_transposed))

    inputs1, lengths1 = pad_packed_sequence(visual_input, batch_first=False)

    semantic_input = PackedSequence(x_semantic[perm], torch.tensor(ls_transposed))

    inputs2, lengths2 = pad_packed_sequence(semantic_input, batch_first=False)   

    output1 = self.linear_visual(inputs1)

    output2 = self.linear_semantic(inputs2)       

    inputs = torch.cat((output1, output2), 2)

    x_fusion = self.odeBlock(inputs, batch_size)

I didn't remove the functions PackedSequence and pad_packed_sequence (I'm not familiar with RNN, but I guess it will not affect how the outputs are computed). Other parts are kept unchanged. The ODE functions are kept as LSTMs.
The results are as follows:

We can see that the performance of SGCLS is still far from reported performance. Did I miss any thing?

from nodis.

But I got a better-than-reported performance with only 6 epochs training of SGDET:

from nodis.

That's a good news! Did you use our object detector?

from nodis.

Sure, I got an even better result (R@50: 0.280 R@100: 0.314) later on epoch 7. Is there any difference between my implementation and the correct one? Though there seems to be no big problem with the other two tasks, the performance on SGCLS decreases significantly.

from nodis.