# Copyright 2015 Yin Zheng, Yu-Jin Zhang, Hugo Larochelle. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without modification, are
# permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this list of
# conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice, this list
# of conditions and the following disclaimer in the documentation and/or other materials
# provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY Yin Zheng, Yu-Jin Zhang, Hugo Larochelle ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
# Yin Zheng, Yu-Jin Zhang, Hugo Larochelle OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and documentation are those of the
# authors and should not be interpreted as representing official policies, either expressed
# or implied, of Yin Zheng, Yu-Jin Zhang, Hugo Larochelle.
'''
@reference: A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data, IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI)
Project page: https://sites.google.com/site/zhengyin1126/home/supdeepdocnade
@Authors: Yin Zheng, Received Ph.D from Tsinghua University, 2015. Homepage: https://sites.google.com/site/zhengyin1126/
Yu-Jin Zhang, Tsinghua University,
Hugo Larochelle, University of Sherbrooke and Twitter. Homepage: http://www.dmi.usherb.ca/~larocheh/index_en.html
@contact: Yin Zheng, [email protected]
'''
We provide:
1. SupDeepDocNADE.py : Classes of DeepDocNADE and SupDeepDocNADE, which is used by run_pretrain_DeepDocNADE.py and run_SupDeepDocNADE.py
2. run_pretrain_DeepDocNADE.py : Python script to pretrain DeepDocNADE model on MIR Flickr Unlabeled data
3. run_SupDeepDocNADE.py : Python script to train and test SupDeepDocNADE mdoel.
4. gen_dataset_labeled.py: Python script to generate the labeled dateset files needed by the model
5. gen_dataset_unlabeled.py: Python script to generate the unlabeled dateset files needed by the model
6. MIR_Flickr_Theano_lab.py: Python script to load labeled dataset and provide interface to SupDeepDocNADE model
7. MIR_Flickr_Theano_Unlab.py: Python script to load unlabeled dataset and provide interface to DeepDocNADE model
=====================================================================================================================
0. Install liblinear (http://www.csie.ntu.edu.tw/~cjlin/liblinear/) and put liblinear into the PYTHONPATH
NOTE: make sure that you can use liblinear in python. To test whether you config liblinear successfully, you could
use "from liblinearutil import *" in python.
1. Generate the dataset:
1) Download the dataset from Nitsh's homepage: http://www.cs.toronto.edu/~nitish/multimodal/index.html
2) Run "python gen_dataset_labeled.py" to generate the labeled dataset.
a) Read the comments in the script about how to use it.
b) create a file IN THE SAME PATH OF THE LABELED DATASET named "meta.txt" with the following lines in the meta.txt
voc_size:2000
n_regions:14
text_voc_size:2000
global_feat_size:1857
n_classes:38
c) Create a file IN THE SAME PATH OF THE LABELED DATASET named "sizes.txt" with the following content:
10000
5000
10000
3) Run "python gen_dataset_unlabeled.py" to generate the unlabeled dataset.
a) Read the comments in the script about how to use it.
b) create a file IN THE SAME PATH OF THE UNLABELED DATASET with the name meta.txt, the content is:
voc_size:2000
n_regions:14
text_voc_size:2000
global_feat_size:1857
dataset_split:50
c) create a file IN THE SAME PATH OF THE UNLABELED DATASET named "sizes.txt" with the following content:
975000
2. Run run_pretrain_DeepDocNADE.py to pretrain DeepDocNADE model on unlabeled data:
python run_pretrain_DeepDocNADE.py n_pretrain pre_learning_rate hidden_size activation_function dropout_rate model_file_dir unlab_dataset_dir batch_size anno_weight polyakexp_weight model_init
The parameters of the script is as follows:
n_pretrain : number of iterations
pre_learning_rate : learning rate of the pretraining
hidden_size : the hidden size of the model, e.g. 2048_2048_2048 is a 3 hidden layers model with 2048 units each layer
activation_function : the activation function of the hidden layers, "sigmoid, relu or tanh"
dropout_rate: the dropout rate for each hidden layer, e.g. "0.5_0.5_0.5" means dropout rate is 0.5 for each layer
model_file_dir: path to save the pretrained model
unlab_dataset_dir: the path to the unlabeled dataset
batch_size: the batch size
anno_weight: annotation weight
polyakexp_weight: polyak weight
model_init: path to the saved pretrain model, which is used to continue_training based on the trained model. It could be NULL if no model saved before.
One example of the scipt is:
python run_pretrain_DeepDocNADE.py 6000 0.03 2048_2048_2048 reclin 0.5_0.5_0.5 PATH_TO_SAVE_THE_MODEL PATH_TO_UNLABELED_DATA 500 12000 std 0.9995 PATH_TO_SAVED_MODEL
3. Run run_SupDeepDocNADE.py to train SupDeepDocNADE model on labeled data, which could be trained from scratch or initialize from pretrained model on unlabeled data :
python run_SupDeepDocNADE.py folder_ID use_pretrain max_iter look_ahead hidden_size learning_rate unsup_weight activation_function Linear_minC, Linear_maxC, dropout_rate uniresult_dir Pretrain_model_name lab_dataset_dir batch_size anno_weight polyakexp_weight
The parameters are as follows:
folder_ID : ID of the dataset ( 1 to 5 )
use_pretrain: Whether use pretrained model or training from randomly initialized parameters (True or False)
max_iter: number of max iterations
look_ahead: early stop if number of iterations without improvement exceed the number of look ahead
hidden_size: the hidden size of the model, e.g. 2048_2048_2048 is a 3 hidden layers model with 2048 units each layer
learning_rate: learning rate of training process
unsup_weight: the weight of the unsupervised part
activation_function: the activation function of the hidden layers, "sigmoid, relu or tanh"
Linear_minC: the minimum value of C for linear SVM (in log_2 space)
Linear_maxC: the max value of C for linear SVM (in log_2 space)
dropout_rate: the dropout rate for each hidden layer, e.g. "0.5_0.5_0.5" means dropout rate is 0.5 for each layer
uniresult_dir: the path to save the results
Pretrain_model_name: the path and the name of the pretrained model if you set use_pretrain True. Otherwise, any string
lab_dataset_dir: the path to the labeled dataset
batch_size : the mini-batch size
anno_weight: the weight of the annotation words
polyakexp_weight: polyak weight
One example of the script is:
python run_SupDeepDocNADE.py 1 True 20000 300 2048_2048_2048 0.01 0.25 reclin -17 10 0.5_0.5_0.5 PATH_TO_SAVE_RESULTS PATH_NAME_OF_PRETRAINED_MODEL PATH_TO_LABELED_DATA 500 12000 0.9995