violetteshev / bottom-up-features Goto Github PK

View Code? Open in Web Editor NEW

71.0 3.0 19.0 182 KB

Bottom-up features extractor implemented in PyTorch.

License: MIT License

Python 72.30% MATLAB 0.45% Shell 0.55% Cuda 11.23% C 14.85% C++ 0.61%

vqa visual-question-answering faster-rcnn feature-extraction pytorch

bottom-up-features's People

Contributors

Stargazers

Watchers

Forkers

memozhu virginie-do sungjune-p stevenji btjhjeon yshen47 yuzcccc jasonmusespresso bigredt congve1 taaccoo-beta leyuan neverwelljd acodec yichao96 probets fawnliu pandapyh

bottom-up-features's Issues

Does the use the attributes?

Hi, thanks for you code.
When look through the code, I didn't find the code related to the attributes. So did you implement the attribute parts?

trained with faster rcnn and backbone resnet152?

Hi,
Thanks for the excellent code base.
can i get the feature trained with faster rcnn and backbone resnet152?
I'm waiting for your reply.

evaluation results

Can this converted model reproduce exactly the same MAP score as that calculated in the original caffe framework?

How can I extract the features using 2000 classes (1600 objects +400 attributes) ?

In your code, we find N_CLASSES = 1601, does it mean 1600 objects + 1 background?
And how can I extract the features using 2000 classes (1600 objects +400 attributes)? Modity the N_CLASSES = 1601 to N_CLASSES = 2001 directly?
Thanks a lot and wait for your reply!

error running sh make.sh

Hi There. I have make a new environment in anaconda and install all the requirements in it.

# packages in environment at /home/hisham/anaconda3/envs/py36:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
blas                      1.0                         mkl  
bzip2                     1.0.8                h516909a_1    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
cairo                     1.16.0            hfb77d84_1002    conda-forge
certifi                   2019.9.11                py36_0  
cffi                      1.12.3           py36h2e261b9_0    anaconda
cuda90                    1.0                  h6433d27_0    pytorch
cudatoolkit               9.2                           0  
cycler                    0.10.0                     py_1    conda-forge
cython                    0.29.13          py36he6710b0_0    anaconda
dbus                      1.13.6               he372182_0    conda-forge
easydict                  1.9                        py_0    conda-forge
expat                     2.2.5             he1b5a44_1003    conda-forge
ffmpeg                    4.1.3                h167e202_0    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
giflib                    5.1.9                h516909a_0    conda-forge
glib                      2.58.3            h6f030ca_1002    conda-forge
gmp                       6.1.2             hf484d3e_1000    conda-forge
gnutls                    3.6.5             hd3a4fd2_1002    conda-forge
graphite2                 1.3.13            hf484d3e_1000    conda-forge
gst-plugins-base          1.14.5               h0935bb2_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_0    conda-forge
harfbuzz                  2.4.0                h9f30f68_3    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1103    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
intel-openmp              2019.4                      243  
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
kiwisolver                1.1.0            py36hc9558a2_0    conda-forge
lame                      3.100             h14c3975_1001    conda-forge
libblas                   3.8.0                    12_mkl    conda-forge
libcblas                  3.8.0                    12_mkl    conda-forge
libclang                  9.0.0                hc9558a2_0    conda-forge
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0                    12_mkl    conda-forge
liblapacke                3.8.0                    12_mkl    conda-forge
libllvm9                  9.0.0                hc9558a2_2    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libprotobuf               3.9.2                h8b12597_0    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.0.10            h57b8799_1003    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libwebp                   1.0.2                h576950b_1    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxkbcommon              0.8.4                h516909a_0    conda-forge
libxml2                   2.9.9                hee79883_5    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
matplotlib                3.1.1                    py36_1    conda-forge
matplotlib-base           3.1.1            py36he7580a8_1    conda-forge
mkl                       2019.4                      243  
mkl-service               2.3.0            py36he904b0f_0  
mkl_fft                   1.0.14           py36ha843d7b_0  
mkl_random                1.1.0            py36hd6b4f25_0  
msgpack-python            0.6.1            py36hfd86e86_1    anaconda
ncurses                   6.1                  he6710b0_1  
nettle                    3.4.1             h1bed415_1002    conda-forge
ninja                     1.9.0            py36hfd86e86_0  
nspr                      4.20              hf484d3e_1000    conda-forge
nss                       3.46                 he751ad9_0    conda-forge
numpy                     1.17.2           py36haad9e8e_0  
numpy-base                1.17.2           py36hde5b4d6_0  
olefile                   0.46                     py36_0  
opencv                    4.1.1            py36ha799480_1    conda-forge
openh264                  1.8.0             hdbcaa40_1000    conda-forge
openssl                   1.1.1d               h7b6447c_2  
pcre                      8.41              hf484d3e_1003    conda-forge
pillow                    6.1.0            py36h34e0f95_0  
pip                       19.2.3                   py36_0  
pixman                    0.38.0            h516909a_1003    conda-forge
protobuf                  3.9.2            py36he1b5a44_1    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
pycparser                 2.19                     py36_0  
pyparsing                 2.4.2                      py_0    conda-forge
pyqt                      5.9.2            py36hcca6a23_4    conda-forge
python                    3.6.9                h265db76_0  
python-dateutil           2.8.0                      py_0    conda-forge
pytorch                   0.4.0           py36_cuda9.0.176_cudnn7.1.2_1  [cuda90]  pytorch
pyyaml                    5.1.2            py36h7b6447c_0    anaconda
qt                        5.9.7                h0c104cb_3    conda-forge
readline                  7.0                  h7b6447c_5  
scipy                     1.3.1            py36h7c811a0_0    anaconda
setuptools                41.2.0                   py36_0  
sip                       4.19.8          py36hf484d3e_1000    conda-forge
six                       1.12.0                   py36_0  
sqlite                    3.29.0               h7b6447c_0  
tensorboardx              1.8                        py_0    conda-forge
tk                        8.6.9             hed695b0_1003    conda-forge
torchvision               0.2.1                      py_2    pytorch
tornado                   6.0.3            py36h516909a_0    conda-forge
tqdm                      4.36.1                     py_0    conda-forge
wheel                     0.33.6                   py36_0  
x264                      1!152.20180806       h14c3975_0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.8                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                h96e3832_1    anaconda
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.0                h3b9ef0a_0    conda-forge

But when i run sh make.sh it gives this error

running build_ext
skipping 'model/utils/bbox.c' Cython extension (up-to-date)
skipping 'pycocotools/_mask.c' Cython extension (up-to-date)
skipping 'numpy_nms/cpu_nms.c' Cython extension (up-to-date)
Compiling nms kernels by nvcc...
make.sh: 26: make.sh: nvcc: not found
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch
  File "/home/hisham/anaconda3/envs/py36/lib/python3.6/site-packages/torch/__init__.py", line 78, in <module>
    from torch._C import *
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
Compiling roi pooling kernels by nvcc...
make.sh: 36: make.sh: nvcc: not found
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch
  File "/home/hisham/anaconda3/envs/py36/lib/python3.6/site-packages/torch/__init__.py", line 78, in <module>
    from torch._C import *
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
Compiling roi align kernels by nvcc...
make.sh: 45: make.sh: nvcc: not found
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch
  File "/home/hisham/anaconda3/envs/py36/lib/python3.6/site-packages/torch/__init__.py", line 78, in <module>
    from torch._C import *
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
Compiling roi crop kernels by nvcc...
make.sh: 54: make.sh: nvcc: not found
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch
  File "/home/hisham/anaconda3/envs/py36/lib/python3.6/site-packages/torch/__init__.py", line 78, in <module>
    from torch._C import *
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

I am tried many things to resolve this but it doesn't work for me. can you please help me?

torch.utils.ffi is deprecated

torch version which is 0.4 really not fit with the project these days.
Hope for an update for a newer version~

Lion and tiger

Why this model does not identify loin and tiger?

module has no attribute

Hello, I encountered the following problem when extracting the image features of the custom dataset, can you help me complete the code?

Traceback (most recent call last):
File "extract_features.py", line 82, in
pooled_feat = fasterRCNN(im_data, im_info, gt_boxes, num_boxes)
File "/root/miniconda3/envs/feature/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/bottom-up-features-master/lib/model/faster_rcnn/faster_rcnn.py", line 73, in forward
pooled_feat = self.RCNN_roi_pool(base_feat, rois.view(-1, 5))
File "/root/miniconda3/envs/feature/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/bottom-up-features-master/lib/model/roi_pooling/modules/roi_pool.py", line 14, in forward
return RoIPoolFunction(self.pooled_height, self.pooled_width, self.spatial_scale)(features, rois)
File "/home/bottom-up-features-master/lib/model/roi_pooling/functions/roi_pool.py", line 25, in forward
roi_pooling.roi_pooling_forward_cuda(ctx.pooled_height, ctx.pooled_width, ctx.spatial_scale,
AttributeError: module 'model.roi_pooling._ext.roi_pooling' has no attribute 'roi_pooling_forward_cuda'

Error when running make.sh

When I run the make.sh, I got the following error:

gcc: error: .../bottom-up-features/lib/model/nms/src/nms_cuda.c: No such file or directory

There is no nms_cuda.c in this GitHub repository. Is it normal?

Thanks in advance.

Batches with multiple images implementation

Hi,

First of all, thanks a lot for your useful repository!
For a project I'm working on, I need your code to support batches with multiple images.
Did you try to implement multiple images extraction at some point? If so, what were your difficulties when trying to do so?

pretrained model for fixed 36 features per image

Hi,
Thank you for providing this code, could you please provide the pretrained model that extract 36 features per image.

Best

Hi, is your pretrained model the same as in original paper? both trained on Visual Gnome?

.npz file

Hi there can you please help me in making a .tsv file or files like this on single image.
It makes two files .npz and .npy using pre-extracted feature from Here

Download the pretrained model

Sorry to disturb you, I can not open the dropbox to download the pretrained model in my country.So can you share another URL to save the model such as google driver. Thank you very much!

torch.jit.frontend.NotSupportedError: slicing multiple dimensions at the same time isn't supported yet

when i extracted the features and boxes, i received the error message following. would you please help me to find the reason? thanks a lot.

error message:
raise NotSupportedError(base.range(), "slicing multiple dimensions at the same time isn't supported yet")
torch.jit.frontend.NotSupportedError: slicing multiple dimensions at the same time isn't supported yet
proposals (Tensor): boxes to be encoded
"""

# perform some unpacking to make it JIT-fusion friendly
wx = weights[0]
wy = weights[1]
ww = weights[2]
wh = weights[3]

proposals_x1 = proposals[:, 0].unsqueeze(1)
               ~~~~~~~~~ <--- HERE
proposals_y1 = proposals[:, 1].unsqueeze(1)
proposals_x2 = proposals[:, 2].unsqueeze(1)
proposals_y2 = proposals[:, 3].unsqueeze(1)

reference_boxes_x1 = reference_boxes[:, 0].unsqueeze(1)
reference_boxes_y1 = reference_boxes[:, 1].unsqueeze(1)
reference_boxes_x2 = reference_boxes[:, 2].unsqueeze(1)
reference_boxes_y2 = reference_boxes[:, 3].unsqueeze(1)

Display Features

How can we display the features that the model detects?

How to convert caffe model to pytorch model?

Thanks for your valuable work!
Could you please tell me how to convert one model (e.g., resnet101_faster_rcnn_final_iter_320000.caffemodel) from caffe to pytorch for your codes? Just any convert code or tutorial is fine!

error while running sh make.sh

running build_ext
skipping 'model/utils/bbox.c' Cython extension (up-to-date)
building 'model.utils.cython_bbox' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/model
creating build/temp.linux-x86_64-3.6/model/utils
{'gcc': ['-Wno-cpp', '-Wno-unused-function']}
gcc -pthread -B /home/faaiz/anaconda3/envs/py36/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -fPIC -I/home/faaiz/anaconda3/envs/py36/lib/python3.6/site-packages/numpy/core/include -I/home/faaiz/anaconda3/envs/py36/include/python3.6m -c model/utils/bbox.c -o build/temp.linux-x86_64-3.6/model/utils/bbox.o -Wno-cpp -Wno-unused-function
gcc -pthread -shared -B /home/faaiz/anaconda3/envs/py36/compiler_compat -L/home/faaiz/anaconda3/envs/py36/lib -Wl,-rpath=/home/faaiz/anaconda3/envs/py36/lib -Wl,--no-as-needed -Wl,--sysroot=/ -std=c99 build/temp.linux-x86_64-3.6/model/utils/bbox.o -o /home/faaiz/Downloads/bottom-up-features/lib/model/utils/cython_bbox.cpython-36m-x86_64-linux-gnu.so
/home/faaiz/anaconda3/envs/py36/compiler_compat/ld: cannot find -lpthread
/home/faaiz/anaconda3/envs/py36/compiler_compat/ld: cannot find -lc
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
Compiling nms kernels by nvcc...
Traceback (most recent call last):
File "build.py", line 4, in
from torch.utils.ffi import create_extension
ModuleNotFoundError: No module named 'torch.utils.ffi'
Compiling roi pooling kernels by nvcc...
Traceback (most recent call last):
File "build.py", line 4, in
from torch.utils.ffi import create_extension
ModuleNotFoundError: No module named 'torch.utils.ffi'
Compiling roi align kernels by nvcc...
Traceback (most recent call last):
File "build.py", line 4, in
from torch.utils.ffi import create_extension
ModuleNotFoundError: No module named 'torch.utils.ffi'
Compiling roi crop kernels by nvcc...
Traceback (most recent call last):
File "build.py", line 4, in
from torch.utils.ffi import create_extension
ModuleNotFoundError: No module named 'torch.utils.ffi'