Applying this Model to CIFAR10 Dataset about deepcluster HOT 12 CLOSED

Hi,
I have tried DeepCluster on smaller datasets such as CIFAR10. Actually the method doesn't seem to work well in this setting.
I experimented with a Network In Network architecture.
With a careful cross validation of parameters (k=100) and whitening the images in input I obtained decent performance but way above the performance reported by RotNet (https://arxiv.org/abs/1803.07728).
For example, when training a SVM on top of block 2 of a NIN3 architecture I got 73% accuracy (RotNet: 82%; Random: 61%).
Hope it helps

from deepcluster.

@KANG-Xudong, which network are you using for the training? Is it your custom network or are you training CIFAR-10 on existing net?

from deepcluster.

@KANG-Xudong, which network are you using for the training? Is it your custom network or are you training CIFAR-10 on existing net?

I simply applied the AlexNet model in the /model/alexnet.py

from deepcluster.

I guess that might be the problem. In case you are using Alexnet or VGG then it would not result in learning of good representations. One of the major reasons being the image size. CIFAR-10 has image of size 32x32 and the above mentioned networks take in 224x224 as input. So you might define a smaller network for CIFAR-10 and then go ahead with the learning.

from deepcluster.

I guess that might be the problem. In case you are using Alexnet or VGG then it would not result in learning of good representations. One of the major reasons being the image size. CIFAR-10 has image of size 32x32 and the above mentioned networks take in 224x224 as input. So you might define a smaller network for CIFAR-10 and then go ahead with the learning.

Thank you for your answer.
I have tried to resize the input image to the shape 244 x 244 to fit the network. Moreover, I have already tried to use this method (using the /models/alexnet.py as model) in supervised training, and get 81% accuracy with CIFAR-10 dataset.
Thus, I think that may not result from the image size.

from deepcluster.

You said you were using k=10, but it need not be the case that the number of classes in your dataset would directly correspond to the number of clusters. You might need to try some ablations with the value of k.
Also when you use k=10, how many points are going in each cluster? Can you type in the stats here?

from deepcluster.

@mathildecaron31
Hi, I am working on a similar problem. Could you please provide more training information. Thanks!!

from deepcluster.

Hi,
I have tried DeepCluster on smaller datasets such as CIFAR10. Actually the method doesn't seem to work well in this setting.
I experimented with a Network In Network architecture.
With a careful cross validation of parameters (k=100) and whitening the images in input I obtained decent performance but way above the performance reported by RotNet (https://arxiv.org/abs/1803.07728).
For example, when training a SVM on top of block 2 of a NIN3 architecture I got 73% accuracy (RotNet: 82%; Random: 61%).
Hope it helps

Hi, Can you explain why this happens on small data sets? I also used the NIN architecture on CIFAR-10, but only got 7% higher classification results than random networks.

from deepcluster.

So, is there a conclusion about the effect on CIFAR10 dataset? I tried this method with several models like alexnet and resnet, but none of them show good results.

from deepcluster.

I have achieved 46.18 on cifar10, resnet18, 400 epochs, 100 clusters (the first conv is 3x3 and modified moco code, knn monitor), updated kmeans every 20 epochs. I do not think it is ok compared to moco ~90. Besides, the accuracy tends to gradually decrease...

from deepcluster.

I just want to add some extra experimental results to this thread. I've experimented on CIFAR-10 with VGG-11 and LeNet. I experimented with 50/100/100 clusters with pca-downampled to 32/64/128, and various training strategies. This method doesn't seem to adapt well. In the best setting (fine-tuning without fixing conv base), there are 2 percent drop in performance. I like the idea of DeepCluster very much but I really cannot figure out why DeepCluster wouldn't work in my case.

from deepcluster.

The approach of deep cluster does not adapt to Cifar-10 dataset. I tried by resizing the images to match the alexnet/vgg16 input size, but model does not learn anything. The debugging showed that there are some nan/zero predictions from the model during the differerent batches of input data. This halts the loss and does not decrease.
Conclusion: Deepcluster does not adapt to Cifar10.

from deepcluster.