uber-research / intrinsic-dimension Goto Github PK

The train_distributed.py script imports from model_builders.py the function build_squeezenet_fastfood, but this is not present there, nor in any other file in the repository. Could you please add it?

I am trying to train the squeezenet or alexnet architecture on a part of ImageNet dataset (in particular, using just a small number of classes). I tried with many choices of learning rate, and all the possible optimizers; in all cases, even adding regularization, the network does not seem to be able to learn. With some combinations, the loss in diverging, while with others, it is remaining roughly constant.
I am training on a machine with 4 GPUs.

Do you know possible reasons of this problem?

What version of tensorflow-gpu version was used?

Hi @yosinski, @rquber, @mimosavvy

I've attempted to reproduce Figure S14 (see figure below) in arxiv version of the paper (https://arxiv.org/pdf/1804.08838.pdf), where you estimate intrinsic dimension on CIFAR-10 using ResNet.

I used ResNet-18 from torchvision.models, fastfood transform, lr=0.0003, batch_size=32, ADAM optimizer, no regularisation, no learning rate schedule. The results I achieved are below.

Would appreciate if you could answer the following questions:

1. What was the learning rate used for your experiment?
2. Did you use learning rate schedule?
3. What was the optimizer used for CIFAR-10+Resnet experiment? (ADAM or SGD, as you do mention in Figure S10 captions that ADAM produces higher int dim)
4. Generally, to estimate intrinsic dimension for how many epochs/iterations do you train? Do you use a stopping criteria?
5. ResNet-18 I used differs from the resnet you used for the experiment. In your case model was "20-layer structure of ResNet with 280k parameters". I was expecting to see that larger Resnet-18 would actually have lower intrinsic dim. Any comments on this?

I don't know if this would help anyone to understand fast random projections:
http://md2020.eu5.org/wht1.html
I wrote it for html5 practice and I can't say it works in anything other than the pale moon browser.