kexinyi / ns-vqa Goto Github PK

Hey,
Thanks for your work.
I have read your paper and want to redo it. But I find that my computer is lack of NVIDIA GPU.
Is it possible to run this project without a NVIDIA GPU?

I'm trying to run pretrained models. I'm in Step 1: object detection.
I encounter this error:
Traceback (most recent call last):
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/tools/test_net.py", line 128, in
check_expected_results=True)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 150, in run_inference
all_results = result_getter()
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 130, in result_getter
multi_gpu=multi_gpu_testing
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 180, in test_net_on_dataset
args, dataset_name, proposal_file, output_dir, gpu_id=gpu_id
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 275, in test_net
cls_boxes_i, cls_segms_i, cls_keyps_i = im_detect_all(model, im, box_proposals, timers)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test.py", line 71, in im_detect_all
model, im, cfg.TEST.SCALE, cfg.TEST.MAX_SIZE, box_proposals)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test.py", line 152, in im_detect_bbox
return_dict = model(**inputs)
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/nn/parallel/data_parallel.py", line 108, in forward
outputs = [self.module(*inputs[0], **kwargs[0])]
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 157, in forward
return self._forward(data, im_info, roidb, **rpn_kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 168, in _forward
blob_conv = self.Conv_Body(im_data)
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/FPN.py", line 246, in forward
self.topdown_lateral_modules[i](fpn_inner_blobs[-1], conv_body_blobs[-(i+2)])
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/FPN.py", line 310, in forward
return lat + td
RuntimeError: The size of tensor a (75) must match the size of tensor b (76) at non-singleton dimension 3

Anyone has encountered this before?
My pytorch version is 0.4.0

Hi,
While compiling the libraries for mark R-CNN I got the following error:

gcc: error: /home/<username>/<path>/ns-vqa/scene_parse/mask_rcnn/lib/model/nms/src/nms_cuda.c: No such file or directory

Can you suggest any solution?
Thanks

Hey,

Thanks you for this work. Really enjoyed your paper.

Is it possible to share your detections.pkl with the masks for CLEVR? (or only the masks). Generating the masks using blender is too long.

Thanks.

May i ask,which part of your code mainly embodies symbol inference? Thx !!!

First of all, thanks for this code!
I am trying to adapt your code in order to have it take one of the images, a custom question, and intercept the answer given by the executor. For such questions, however, the only information that is given for each question is the image_index and the question (though I can artificially add a split). I however do not seem to manage to modify to code to make it work without an input program associated to the questions.

Would you have any pointers in how I could go about doing this?

Hi, is the Mask RCNN only trained on the CLEVR or used with the original (official) pre-trained parameters?

I installed the environment requirements following every step as README.md said, but CUDNN error occured at the Step 1:object detection.
with the cmd:

python tools/train_net_step.py
--dataset clevr-mini
--cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml
--bs 8
--set OUTPUT_DIR ../../data/mask_rcnn/outputs

Here is the Detail Error:

INFO test_engine.py: 331: loading checkpoint ../../data/pretrained/object_detector.pt
Traceback (most recent call last):
  File "tools/test_net.py", line 126, in <module>
    check_expected_results=True)
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 129, in run_inference
    all_results = result_getter()
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 109, in result_getter
    multi_gpu=multi_gpu_testing
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 159, in test_net_on_dataset
    args, dataset_name, proposal_file, output_dir, gpu_id=gpu_id
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 254, in test_net
    cls_boxes_i, cls_segms_i, cls_keyps_i = im_detect_all(model, im, box_proposals, timers)
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test.py", line 71, in im_detect_all
    model, im, cfg.TEST.SCALE, cfg.TEST.MAX_SIZE, box_proposals)
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test.py", line 152, in im_detect_bbox
    return_dict = model(**inputs)
  File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/nn/parallel/data_parallel.py", line 108, in forward
    outputs = [self.module(*inputs[0], **kwargs[0])]
  File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 144, in forward
    return self._forward(data, im_info, roidb, **rpn_kwargs)
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 155, in _forward
    blob_conv = self.Conv_Body(im_data)
  File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/modeling/FPN.py", line 228, in forward
    conv_body_blobs = [self.conv_body.res1(x)]
  File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

It happened when load checkpoint
test_engine.py: 331: loading checkpoint ../../data/pretrained/object_detector.pt
Then I try the other steps or Train, the same error appears.

The system information:
Ubuntu16.04
RTX2080Ti
cuda 9.0.176
cuDNN 7.1.2
pytorch 0.4.0
python3.6.7
The code is running in the conda virtual environment.
cuda9.0 and cuDNN 7.3.1 are in base environment. are they linked with make.sh file?

I have tried many solutions from google, e.g. change version of cuda and cuDNN. But still, I got same error. I also tried other project mac-network, and it is working fine on GPU in same virtual environment.

Sincerely hope your reply!
Thanks

Hi, I retrain the model but get lower accuracy (overall_acc=0.72 for pretrained parser and 0.88 for reinforce-finetuning), do you have any suggestions?

when i run :
python tools/test_net.py
--dataset clevr_original_val
--cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml
--load_ckpt ../../data/pretrained/object_detector.pt
--output_dir ../../data/mask_rcnn/results/clevr_val_pretrained

i get this error ImportError: cannot import name 'numpy_type_map'
i use CUDA10.1 pytorch 1.3 torchvision 0.4.1

Hello!

I have a problem running the pretrained models.
Everything has compiled and been downloaded successfully, but when I try to run
python tools/test_net.py \ --dataset clevr_original_val \ --cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml \ --load_ckpt ../../data/pretrained/object_detector.pt \ --output_dir ../../data/mask_rcnn/results/clevr_val_pretrained
it gives the following error:

Traceback (most recent call last):
File "tools/train_net_step.py", line 27, in
from modeling.model_builder import Generalized_RCNN
File "/home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 11, in
from model.roi_pooling.functions.roi_pool import RoIPoolFunction
File "/home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/model/roi_pooling/functions/roi_pool.py", line 3, in
from .._ext import roi_pooling
File "/home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/model/roi_pooling/_ext/roi_pooling/init.py", line 3, in
from ._roi_pooling import lib as _lib, ffi as _ffi
ImportError: /home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/model/roi_pooling/_ext/roi_pooling/_roi_pooling.so: undefined symbol: cudaLaunchKernel

Please could somebody lend me a hand?

I was wondering if there are any updated links for downloading the smaller subset of CLEVR(~4k Images)? The current download.sh seems to be out of date.

I get this error :

File "tools/preprocess_questions.py", line 11, in
import utils.programs as program_utils
ModuleNotFoundError: No module named 'utils'

python tools/test_net.py
--dataset clevr_original_val
--cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml
--load_ckpt ../../data/pretrained/object_detector.pt
--output_dir ../../data/mask_rcnn/results/clevr_val_pretrained

When I run the script above, it gives out a cudaCheckError

cudaCheckError() failed : no kernel image is available for execution on the device .

it happens with the RoIAlignFunction

Do you have the code for testing on CLEVR-Human?
Or can you briefly describe where to change for testing on CLEVR-Human.
Thank you.

Hi,
I encounter this error message
module 'modeling.roi_xfrom.roi_align._ext.roi_align' has no attribute 'roi_align_forward_cuda'
both running pretrain scene parser and training scene parser.
Do you have solution of this?

Thanks.

My envs setting is below:
python=3.5
pytorch=0.4.0
CUDA=9.0

Hello,

In the paper, you say: "Because the original CLEVR dataset does not include object masks, we generate these 4,000 training images ourselves using the CLEVR dataset generation tool". However, I don't see this process in this code release.

Can you release the code for how to do this, or give some insight into the process you used to do this?

Thanks!