This project is the Pytorch implementation of Neural Image Captioning 2015 paper by Vinyals et. al.[PDF]. The implementation is inspired from the Udacity Image captioning project [Repo link]

• Backend : Pytorch, Pytorch Vision
• Dataset : MS COCO 2014 Dataset [Link]

• data_load.py : Dataloader class and functions for data augmentation.
• model.py : Model class consisting of model definitions and functions.
• vocabulary.py : Model class consisting of vocublary functions.
• training.ipynb : Jupyter notebook with training hyperparameters like learning rate, batch size, embedding size, hidden state size etc.
• inference.ipynb : Jupyter notebook to sample the captions generated by the encoder-decoder model.
• vocabulary and architecture experiments.ipynb : Jupyter notebook to understand the vocabulary generation process and experiment with the CNN-RNN architecture to check whether the model.py implementation is correct or not.

1. Clone this repo: https://github.com/cocodataset/cocoapi

git clone https://github.com/cocodataset/cocoapi.git  

2. Setup the coco API (also described in the readme here)

cd cocoapi/PythonAPI  
make  
cd ..

3. Download some specific data from here: http://cocodataset.org/#download (described below)

• Under Annotations, download:

  • 2014 Train/Val annotations [241MB] (extract captions_train2014.json and captions_val2014.json, and place at locations cocoapi/annotations/captions_train2014.json and cocoapi/annotations/captions_val2014.json, respectively)
  • 2014 Testing Image info [1MB] (extract image_info_test2014.json and place at location cocoapi/annotations/image_info_test2014.json)

• Under Images, download:

  • 2014 Train images [83K/13GB] (extract the train2014 folder and place at location cocoapi/images/train2014/)
  • 2014 Val images [41K/6GB] (extract the val2014 folder and place at location cocoapi/images/val2014/)
  • 2014 Test images [41K/6GB] (extract the test2014 folder and place at location cocoapi/images/test2014/)