noagarcia / context-art-retrieval Goto Github PK

2022/01/05: Downloading links for the pre-trained models have been updated. Sorry for the wait.

Pytorch code for the cross-modal retrieval part of our ICMR 2019 paper Context-Aware Embeddings for Automatic Art Analysis. For the classification part, check this other repository.

1. Download dataset from here.

2. Clone the repository:

   git clone https://github.com/noagarcia/context-art-retrieval.git

3. Install dependencies:

   • Python 2.7
   • pytorch (conda install pytorch=0.4.1 cuda90 -c pytorch)
   • torchvision (conda install torchvision)
   • visdom (check tutorial here)
   • pandas (conda install -c anaconda pandas)
   • nltk (conda install -c anaconda nltk)
   • sklearn (conda install scikit-learn)

4. Download our pre-trained context-aware models obtained with the classification code and save them into Models/ folder:

• MTL
• KGM Type
• KGM School
• KGM Timeframe
• KGM Author

• To train cross-modal retrieval model with MTL context embeddings run:

  python main.py --mode train --model mtl --dir_dataset $semart

• To train cross-modal retrieval model with KGM context embeddings run:

  python main.py --mode train --model kgm --att $attribute --dir_dataset $semart

Where $semart is the path to SemArt dataset and $attribute is the classifier type (i.e. type, school, time, or author).

• To test cross-modal retrieval model with MTL context embeddings run:

  python main.py --mode test --model mtl --dir_dataset $semart

• To test cross-modal retrieval model with KGM context embeddings run:

  python main.py --mode test --model kgm --att $attribute --dir_dataset $semart --model_path $model-file

Where $semart is the path to SemArt dataset, $attribute is the classifier type (i.e. type, school, time, or author), and $model-file is the path to the trained model.

You can download our pre-trained cross-modal retrieva models with context embeddings from:

• MTL Type
• MTL School
• MTL Timeframe
• MTL Author
• KGM Type
• KGM School
• KGM Timeframe
• KGM Author

Text-to-Image retrieval results on SemArt dataset:

@InProceedings{Garcia2017Context,
   author    = {Noa Garcia and Benjamin Renoust and Yuta Nakashima},
   title     = {Context-Aware Embeddings for Automatic Art Analysis},
   booktitle = {Proceedings of the ACM International Conference on Multimedia Retrieval},
   year      = {2019},
}