Question about reproducing CoCLR results about coclr HOT 17 CLOSED

For other researchers' references, I am able to replicate the code and replicate the InfoNCE probe results on paper by training longer

from coclr.

Actually the performance of 45% is normal. In my communication with the author through e-mail. He mentioned 51.8% result is obtained by pretraining for 800 epochs. For 500 epochs he can obtain a 46.8% linear eval performance. You can consider your implementation is correct. I obtain a similar performance when training for 500 epochs. The author mentioned to change the table in paper to 46.8% after NeurIPS 2020.

Besides, the "consistent = True" is not the reason. If you look carefully at his code, it is still clip-wise consistent. He concatenated two clips into one tensor. So you should keep consistent = False.

from coclr.

Sorry, I misunderstood your exp settings. I have updated my comments above.

For linear probe experiments, I find [1] using SGD with large initial learning rate like 1 or 0.1 helps (better than Adam, big lr is also a common practice of linear probe in other's code, e.g. CMC, MoCo). Also decay lr twice, e.g. at ep60, ep80 and trained for totally 100 epochs. You have to tune weight_decay higher like 1e-3 to 1e-2 for each experiment.

[2] Alternatively, I also tried pre-extract S3D features (avg_pool to vector) for both training set and testing set (e.g. similar as MIL-NCE paper), then train a linear layer directly on features. It gives similar results as the method [1].

Hope it helps, GL

from coclr.

Thanks! I will try out this evaluation setting and update if it works.

from coclr.

Actually the performance of 45% is normal. In my communication with the author through e-mail. He mentioned 51.8% result is obtained by pretraining for 800 epochs. For 500 epochs he can obtain a 46.8% linear eval performance. You can consider your implementation is correct. I obtain a similar performance when training for 500 epochs. The author mentioned to change the table in paper to 46.8% after NeurIPS 2020.

Besides, the "consistent = True" is not the reason. If you look carefully at his code, it is still clip-wise consistent. He concatenated two clips into one tensor. So you should keep consistent = False.

I can get 50% with 700 epochs now. I check the code, for TwoClipTransform and OneClipTransform, two clips are processed with transform respectively, one clip by one clip, so consistent=True and False might be the same. By the way, RandomGray use consistent=True by default in augmentation.py, which is not set tobe False in main_nce.py. Discussion is helpful.

from coclr.

which protocol are you talking about,

training from scratch on UCF can at least give you >70 top1 acc, how could it be possible to be only 4-5% ........

from coclr.

Thanks for the reply. I am talking about the InfoNCE, I found out that Adam optimizer is the culprit here. I changed it to SGD optimizer and the top1 accuracy will reach >80 top 1 acc in the end. The reason is still unclear though... Currently I am trying to replicate your InfoNCE RGB result. Is it possible that you can share about the pretraining and linear probe parameters you used for this setting (row 1 table 1 in the paper)?

Also I want to ask is the current version of augmentation the one you mentioned in the paper? It is not really clip-wise consistent so I am not sure if it is obsolete now.

from coclr.

My current replication using your hyperparameters and model can only get 30 top 1 on UCF101 compared to 52.3 reported in paper. But I only pretrain it for 200 epochs. I wonder if 500 epochs is the key here and I am currently training one with 500 epochs. I wonder do you have any results on RGB inputs with less epochs?

from coclr.

[finetuning] When evaluate by finetuning on UCF101, note that we use dropout=0.9 to avoid quick overfitting. I will update the code soon, but the evaluation code we use is very similar to https://github.com/TengdaHan/MemDPC/blob/master/eval/test.py, except changing the network to S3D+1FC.

[linear probing] Pre-training for 500 epochs is not the key here (as you only get 30% top1), when I pre-trained InfoNCE for 200 epochs on UCF101, I can get about 45% on UCF101.

[Augmentation] I sample frame like this:

# in dataloader
def double_sampler(self, total):
    seq1 = self.frame_sampler(total) # clip form random starting time
    seq2 = self.frame_sampler(total) # clip from random starting time, again
    return np.concatenate([seq1, seq2])
def __getitem__(self, index):
    # ... get video length as "vlen", load all frame paths in "raw" ...
    frame_index = self.double_sampler(vlen)
    seq = [pil_from_raw_rgb(raw[i]) for i in frame_index]
    if self.transform is not None: seq = self.transform(seq)
    seq = torch.stack(seq, 1)
    # return seq

ColorJittering is actually clip-wise consistent: https://github.com/TengdaHan/CoCLR/blob/main/utils/augmentation.py#L302
It's consistent within every "seq_len" frames. But if the input is two or more sequences, the second and later will be different sets of random ColorJittering.

I will update more code this weekend, hope now you can continue your experiments.

from coclr.

Hi Tengda. Thanks for the explanation! I still have some questions and want to double confirm some of your statements.

Actually I am doing the linear probe setting (to freeze the visual encoder and train a linear classifier on top). So currently after 200 epochs InfoNCE pre-training, I am training the added linear classifier for another 100 epochs with UCF101 labels. When you talk about finetuning for 350 epochs, do you mean you finetune the linear classifier for 350 epochs when doing the linear probe setting?

Thanks again for your prompt reply!

from coclr.

Hi Tengda,

I have tried out your evaluation setting but to no avail. I have also tried to use your released code entirely with my own evaluation script and data loading script modified from MemDPC but the accuracy is still around 30%. I wonder if my data loading or evaluation code is wrong. Is it possible that you could share your trained model checkpoint for InfoNCE linear probe setting so I will just use your code to see if my data loading script or evaluation script has any bugs?

Best,
Hualin

from coclr.

Hi Tengda,

I am currently trying to replicate your CoCLR result as one of the baselines in our work with the code you provide. However, I encounter some reproduction issues during the training. I understand that the code is not ready yet. It would be much appreciated if you could help us with replication. Thank you so much!

1. I found out that the Top 1 MoCo accuracy is quite low (only 4-5 percent in UCF101) with 1e-3 lr and Adam Optimizer, 1e-5 weight decay, 2048 moco queue size and 128 batch size. I wonder if you could provide a detailed training command for our reference.
2. The augmentation is not really clip-wise consistent since the value passed in is false. I wonder if this version is not final version. Could you provide the correct version of the augmentation you use?
3. Currently the code for data loader is not released and I don't know how input is prepared in data loader to be passed to TwoCropTransform and OneCropTransform. Could you please share the data loader code for our better replication?

Best Regards,
Hualin

I can only get 30% now. What have you done to improve it? By the way, the tensor size of input_seq is not compatible in main_classify.py

from coclr.

You need to train for 500 epochs. My 30% accuracy was obtained by training only 200 epochs. You need to train longer.

Before the author releases the full code, you can refer to the MemDPC repository to copy the data loader part. Regarding the tensor size error, you can reverse engineer that part by looking at the augmentation code

from coclr.

You need to train for 500 epochs. My 30% accuracy was obtained by training only 200 epochs. You need to train longer.

Before the author releases the full code, you can refer to the MemDPC repository to copy the data loader part.

Now the author has published some codes. What is your pretraining schedule?(lr, milestone, batch size, gpu num, momentum coefficient for moco...). And have you used multi-crops when evaluating your linear evaluation results?

from coclr.

I basically follow the author's setting: 1e-3 adam decayed 1/10 at 300 and 400, 64/128 for 4 gpus(depending on your gpu memory). And the rest of MoCo params are all in paper and you can use the default one in his code. I followed the ten-crop protocol to evaluate

from coclr.

I basically follow the author's setting: 1e-3 adam decayed 1/10 at 300 and 400, 64/128 for 4 gpus(depending on your gpu memory). And the rest of MoCo params are all in paper and you can use the default one in his code. I followed the ten-crop protocol to evaluate

where did you find to decay or ar 300/400? I cant find this detail in the paper, thx.

from coclr.

I basically follow the author's setting: 1e-3 adam decayed 1/10 at 300 and 400, 64/128 for 4 gpus(depending on your gpu memory). And the rest of MoCo params are all in paper and you can use the default one in his code. I followed the ten-crop protocol to evaluate

I can get 45% now, Thanks for your advice. and ten crop seems to be useless, my results of center crop is the same as multi-crops. Now there is still 7% gap. I have changed all augmentation procedure to be consistent=True (original one in the code is False), is this the reason?

from coclr.