hyeonwoonoh / dppnet Goto Github PK

View Code? Open in Web Editor NEW

95.0 14.0 45.0 75 KB

DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

License: Other

Shell 1.43% Lua 67.70% Python 1.72% CMake 25.84% Makefile 1.42% Cuda 1.90%

dppnet's Introduction

DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

Created by Hyeonwoo Noh, Paul Hongsuck Seo and Bohyung Han at POSTECH cvlab

Project page: [http://cvlab.postech.ac.kr/research/dppnet/]

Introduction

DPPnet is state-of-the-art Image Question Answering algorithm using dynamic parameter prediction to handle various types of questions.

Detailed description of the system will be provided by our technical report arXiv tech report

Citation

If you're using this code in a publication, please cite our papers.

@article{noh2015image,
  title={Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction},
  author={Noh, Hyeonwoo and Seo, Paul Hongsuck and Han, Bohyung},
  journal={arXiv preprint arXiv:1511.05756},
  year={2015}
}

Licence

This software is for research purpose only. Check LICENSE file for details.

System Requirements

This software is tested on Ubuntu 14.04 LTS (64bit).
At least 12GB gpu memory is required (NVIDIA tital-x gpu is used for training).

Dependencies

torch [https://github.com/torch/distro]
loadcaffe [https://github.com/szagoruyko/loadcaffe]
xxhash [install: luarocks install xxhash]

Setup

Run "setup.sh" for setting up.

Testing

Scripts for testing is in "006_test_DPPnet". Use following commands for testing. 0. Run ./gen_simulinks.sh 0. Run th vqa_test.lua 0. Results will be saved in "006_test_DPPnet/save_result_vqa_test/results/"

Training

Following steps are required for training. 0. Train DPPnet with fixed cnn feature (004_train_DPPnet_fixed_cnn) 0. Finetune CNN from the model trained in the previous step (005_train_DPPnet_finetune_cnn)

Directories

001_porting_VQA_data: porting VQA data for torch implementation
002_extract_image_features: extracting VGG16 features from MSCOCO
003_skipthoughts_porting: porting model parameters for skipthoughts
004_train_DPPnet_fixed_cnn: training DPPnet with extracted feature
005_train_DPPnet_finetune_cnn: fine-tuning cnn from model trained 004
cache: We cache loaded vqa questions ans vocabularies to reduce the time for loading
data: data used for training / testing
model: trained model parameters, model definitions, layer implementations
utils: utilities (loading training data, loading models ...)

dppnet's People

Stargazers

Watchers

dppnet's Issues

/torch/install/bin/luajit: C++ exception Issue

Problem Statement:
When I am running the following command

th vqa_train.lua -gpuid 1

I get the following message :

loading cache..: /home1/badri/badripatro/VQA/workspace_project/image_qa_dpp/DPPnet-master/004_train_DPPnet_fixed_cnn/cache/vqa_data_cache_major_test-dev2015_54
done
creating a neural network with random initialization
/home/cse/torch/install/bin/luajit: C++ exception
badri@cse-desktop:/DPPnet-master/004_train_DPPnet_fixed_cnn$

I have installed all the Dependencies for this code such as

torch [https://github.com/torch/distro]
loadcaffe [https://github.com/szagoruyko/loadcaffe]
xxhash [install: luarocks install xxhash]

Also, I have narrowed it down to the line 79 of file "DPPnet-master_1/model/HashedNets/HasherME.lua" and get get "libhashnn.mysort()" has problem

libhashnn.mysort(self['sort_key_' .. WorB],self['sort_val_'.. WorB])

Does anyone have any advice on how I can try to further determine the problem?

It has a problem that C++ exception when I run vqa_test.lua

when I run vqa_test.lua,it occurs:
/home/jxas/cwq/torch/install/bin/luajit: C++ exception
I find that the problem may be casued by the function " libhashnn.mysort(self['sort_key_' .. WorB],self['sort_val_'.. WorB])".
I wrint the print statement,and find that the program can't catch the function libhashnn.mysort() in the file called myhashnn.cu.
I see the previous Issues about this problem ,and I install the old version torch ,but it doesn't work.

who can give some addvice to solve the problem?

C++ exception

NVCC is running and using older version of torch which is referred in one of the solutions of the problem.
Still getting this exception and not getting any stack trace.

No setup.sh file

In the Readme file, it says Run "setup.sh" for setting up. But I could not find the file in this repository.

Compilation under CUDA 7.5 fails

When compiling under CUDA 7.5 fails with following error

/usr/local/cuda/include/cuda_fp16.h:314:83: error: conflicting declaration of C function ‘__half __ldg(const _half)’
/usr/local/cuda/include/cuda_fp16.h:313:60: note: previous declaration ‘__half2 __ldg(const _half2)’
/usr/local/cuda/include/cuda_fp16.h: In function ‘__half2 __ldg(const _half2)’:
/usr/local/cuda/include/cuda_fp16.h:1180:84: error: conflicting declaration of C function ‘__half2 __ldg(const _half2)’
/usr/local/cuda/include/cuda_fp16.h:314:59: note: previous declaration ‘__half __ldg(const __half*)’
CMake Error at hashnn_generated_myhashnn.cu.o.cmake:262 (message):
Error generating file
/home/ap/DPPnet/model/HashedNets/libhashnn/_build/CMakeFiles/hashnn.dir//./hashnn_generated_myhashnn.cu.o

Results on VQA test-dev

hi
Are quantitative results on VQA test-dev stored somewhere? I can just see the json file for it.

/home/amax/torch/install/bin/luajit: cannot open </home/amax/Desktop/lichunye/DPPnet-master/006_test_DPPnet/data/VQA_torch/Annotations/mscoco_train2014_annotations/annotations.t7> in mode r at /home/amax/torch/pkg/torch/lib/TH/THDiskFile.c:673

hello~
when i run"th vqa_test.lua" or "th vqa_train.lua",the results are as the follows:

/home/amax/torch/install/bin/luajit: cannot open </home/amax/Desktop/lichunye/DPPnet-master/006_test_DPPnet/data/VQA_torch/Annotations/mscoco_train2014_annotations/annotations.t7> in mode r at /home/amax/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: at 0x7f1b638d9450
[C]: in function 'DiskFile'
/home/amax/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
./utils/vqa_loader.lua:37: in function 'load_data'
./utils/vqa_loader.lua:1900: in function 'load_data'
vqa_test.lua:98: in main chunk
[C]: in function 'dofile'
...amax/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

and it is empty in the "006_test_DPPnet/save_result_vqa_test/results/"and "004_train_DPPnet_fixed_cnn/save_result_vqa/results/"
but i don't why this happens.

Problem with libhashnn

Running ./compile.sh compiled libhashnn.
Then when I call HashLinear, it cannot get libhashnn.
It shows me following error-
./HashLinear.lua:119: attempt to index global 'libhashnn' (a nil value)

An illegal memory access was encountered

Problem Statement:
When I am running the following command

th vqa_train.lua -gpuid 1

I get the following message :

/home/cse/torch/install/bin/luajit: C++ exception
badri@cse-desktop:/DPPnet-master/004_train_DPPnet_fixed_cnn$

Also, I have narrowed it down to the line 79 of file "DPPnet-master_1/model/HashedNets/HasherME.lua" and get get "libhashnn.mysort()" has problem

libhashnn.mysort(self['sort_key' .. WorB],self['sort_val_'.. WorB])_

Then I have commented the line -79, and complied again

th vqa_train.lua -gpuid 1

I get the following message :

loading cache..: /home1/badri/badripatro/VQA/workspace_project/image_qa_dpp/DPPnet-master/004_train_DPPnet_fixed_cnn/cache/vqa_data_cache_major_test-dev2015_54
done
creating a neural network with random initialization
initialing weights..
[train2014val2014] set batch order option 1 : shuffle __________________________________________________
THCudaCheck FAIL file=/home1/badri/torch/extra/cutorch/lib/THC/generic/THCStorage.c line=147 error=77 : an illegal memory access was encountered
/home1/badri/torch/install/bin/luajit: cuda runtime error (77) : an illegal memory access was encountered at /home1/badri/torch/extra/cutorch/lib/THC/generic/THCStorage.c:147

I have narrowed this problem down to the line 423 of file

004_train_DPPnet_fixed_cnn/vqa_train.lua

  **dlinear_out[i] = HasherME:backward(dhashed_out)**

Still on more debug, find in line no 114 of file "DPPnet-master_1/model/HashedNets/HasherME.lua" and get get "libhashnn.mysort()" has problem

libhashnn.myreduce(self.sort_key_W,self.gradOBuffer,self.unique_idxW,self.gradInput,self.buffer_W)

Always getting problem in the "libhashnn".
Does anyone have any advice on how I can try to further determine the problem?

Can we use cudnn to speed up this project?

I find that it cost about 48 hours to train 005_train_DPPnet_finetune_cnn.
Can we use cudnn to speed up this project? Have you ever tried to do so ？
If I want to use cudnn, which part do I need to modify?

Problem in running vqa_train.lua with cpu

Hi
I tried to run 004_train_DPPnet_fixed_cnn/vqa_train.lua -gpuid -1 but I encounterd following error:

lua: bad argument #1 to '?' (table expected, got string)
stack traceback:
[C]: ?
[C]: in function 'require'
vqa_train.lua:102: in main chunk
[C]: ?

Can you please suggest possible reasons or ways to fix it.

Thanks

bug in vqa_loader.lua

Line 1874 "local is_cached = path.isfile(cache_path)" in vqa_loader.lua attempt to index global 'path' (a nil value) as it has not been declared anywhere.

C++ exception

I get a C++ exception when I run the vqa_tes.lua in 006_test_DPPNet. Could you help me out?

hyeonwoonoh / dppnet Goto Github PK

dppnet's Introduction

DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

Introduction

Citation

Licence

System Requirements

Dependencies

Setup

Testing

Training

Directories

dppnet's People

Stargazers

Watchers

Forkers

dppnet's Issues

Recommend Projects

Recommend Topics

Recommend Org