Reproduce results about powerful-benchmarker HOT 2 CLOSED

Thanks for your interest in this project! I just pushed a commit to address this problem. Here is how you can reproduce results now:

1. In the benchmark spreadsheet, click on the google drive link under the "config files" column.
2. Download the folders you want (for example cub200_old_approach_triplet_batch_all), into some folder on your computer. For example, I downloaded into /home/tkm45/experiments_to_reproduce
3. Then run:
   python run.py --reproduce_results /home/tkm45/experiments_to_reproduce/cub200_old_approach_triplet_batch_all --experiment_name cub200_old_approach_triplet_batch_all_reproduced

You cannot override complex (nested) config options at the command line in this case. But you can still override non-nested options. For example, you might like to use a different number of workers for your dataloaders:

python run.py --reproduce_results /home/tkm45/experiments_to_reproduce/cub200_old_approach_triplet_batch_all --experiment_name cub200_old_approach_triplet_batch_all_reproduced --dataloader_num_workers 6 --eval_dataloader_num_workers 6

What still needs to be fixed:
In the configs folder of cub200_old_approach_triplet_batch_all, you'll see a subfolder called resume_training_config_diffs_1. This just indicates that I resumed training and that I changed num_epochs_train to 60 (it was originally 50). At the moment, the "reproduce_results" option does not look for these diff folders. So you may need to copy any changes from the diff folder into the config file. From what I remember, the only parameter I ever changed when resuming training was num_epochs_train.

Let me know if it works for you!

from powerful-benchmarker.

My latest commits fix the config_diff issue. The reproduce_results flag parses the config_diff folders so that training proceeds exactly as it did in the original experiment.

from powerful-benchmarker.