Hello, it looks like the link to your slides in the README.md file is broken. Would you happen to have an updated link?

https://www.dropbox.com/s/gpvbaf9o4et9d5v/iccv15_tutorial_training_faster.pdf?dl=0

Hi.
I already successfully run fast-rcnn code written by the same author of this.
Then, I would like to try to use faster-rcnn..
However, when I run demo.py, I got this error.
'''
Traceback (most recent call last):
File "./demo.py", line 18, in
from fast_rcnn.test import im_detect
File ".../py-faster-rcnn-master/tools/../lib/fast_rcnn/test.py", line 17, in
from fast_rcnn.nms_wrapper import nms
File ".../py-faster-rcnn-master/tools/../lib/fast_rcnn/nms_wrapper.py", line 11, in
from nms.gpu_nms import gpu_nms
ImportError: No module named gpu_nms
'''
Anyone has any idea about this error?
Thank you.

I'm trying to train a single RPN for region proposal using tools/train_net.py with --solver py-faster-rcnn/models/VGG16/faster_rcnn_alt_opt/stage1_rpn_solver60k80k.pt

However, the following error message show up

[libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 11:14: Message type "caffe.PythonParameter" has no field named "param_str".

I found that no matter how I re-build the py-faster-rcnn/caffe-fast-rcnn/ branch without error, make pytest always fail while executing "test_python_layer_with_param_str.py", is there any problem with this branch ?

if I use another branch of caffe-fast-rcnn, which is not on the faster-rcnn branch, make pytest would pass but smooth_l1_loss layer would not be included :(

any idea ?

Just a note to those who may come across with cuDNN error when compiling py-faster-rcnn with cuDNN v4.

Found that the current py-faster-rcnn works with cuDNN v3 cudnn-7.0-linux-x64-v3.0-prod but not v4 cudnn-7.0-linux-x64-v4.0-rc, which is released on 10 Dec 2015.

Maybe it can be added to the remarks.

Hello!

I just finished the two downloads for the models listed here: http://www.cs.berkeley.edu/~rbg/faster-rcnn-data/

I've then the liberty of mirroring them on academic torrents. If this is not desired by the authors, I can de-list them from the tracker.

Hopefully this will make downloading these models faster, as I just dumped 6 hours into downloading these 2 GB.

I modify model to use 2 classes instead of 21 classes and create an annotation file for each face image
However, during stage1 of rpn training, it has following warning:

RuntimeWarning: invalid value encountered in log
targets_dh = np.log(gt_heights / ex_heights)

Any idea of what the above warning mean? Is this due to something wrong in my annotation?

Hi @rbgirshick,
Thanks for being so freaking awesome and porting it in Python.

I followed the installation instructions point-by-point but got stuck at the make step in $FRCN_ROOT/lib

I do not have cuda on my laptop, therefore, I tweaked the setup.py file to remove the cuda parts. See below -

# --------------------------------------------------------
# Fast R-CNN
# Copyright (c) 2015 Microsoft
# Licensed under The MIT License [see LICENSE for details]
# Written by Ross Girshick

import os
from os.path import join as pjoin
from setuptools import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import subprocess
import numpy as np

def find_in_path(name, path):
    "Find a file in a search path"
    #adapted fom http://code.activestate.com/recipes/52224-find-a-file-given-a-search-path/
    for dir in path.split(os.pathsep):
        binpath = pjoin(dir, name)
        if os.path.exists(binpath):
            return os.path.abspath(binpath)
    return None

# Obtain the numpy include directory.  This logic works across numpy versions.
try:
    numpy_include = np.get_include()
except AttributeError:
    numpy_include = np.get_numpy_include()

# run the customize_compiler
class custom_build_ext(build_ext):
    def build_extensions(self):
        build_ext.build_extensions(self)


ext_modules = [
    Extension(
        "utils.cython_bbox",
        ["utils/bbox.pyx"],
        extra_compile_args={'gcc': ["-Wno-cpp", "-Wno-unused-function"]},
        include_dirs = [numpy_include]
    ),
    Extension(
        "nms.cpu_nms",
        ["nms/cpu_nms.pyx"],
        extra_compile_args={'gcc': ["-Wno-cpp", "-Wno-unused-function"]},
        include_dirs = [numpy_include]
    ),
]

setup(
    name='fast_rcnn',
    ext_modules=ext_modules,
    # inject our custom trigger
    cmdclass={'build_ext': [custom_build_ext]},
)

I got this error -

python setup.py build_ext --inplace
running build_ext
skipping 'utils/bbox.c' Cython extension (up-to-date)
building 'utils.cython_bbox' extension
Traceback (most recent call last):
  File "setup.py", line 149, in <module>
    cmdclass={'build_ext': custom_build_ext},
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 163, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python2.7/distutils/command/build_ext.py", line 337, in run
    self.build_extensions()
  File "setup.py", line 109, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 171, in build_extensions
    self.build_extension(ext)
  File "/usr/lib/python2.7/distutils/command/build_ext.py", line 496, in build_extension
    depends=ext.depends)
  File "/usr/lib/python2.7/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/lib/python2.7/distutils/unixccompiler.py", line 120, in _compile
    extra_postargs)
TypeError: can only concatenate list (not "dict") to list
make: *** [all] Error 1

I can see that the error is being cause because I am supplying dict, but it is adding it to a list.
Could you please help ?

when I run train_faster_rcnn_alt_opt.py,error occurs:

I1210 14:39:47.685863 223288 layer_factory.hpp:76] Creating layer input-data
I1210 14:39:47.786041 223288 net.cpp:112] Creating Layer input-data
I1210 14:39:47.786516 223288 net.cpp:435] input-data -> data
I1210 14:39:47.787017 223288 net.cpp:435] input-data -> im_info
I1210 14:39:47.788019 223288 net.cpp:435] input-data -> gt_boxes
I1210 14:39:47.804545 223288 python_layer.hpp:43] Traceback (most recent call last):
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\roi_data_layer\layer.py", line 125, in setup
top[idx].reshape(1, self._num_classes * 4)
IndexError: Index out of range
Process Process-1:
Traceback (most recent call last):
File "D:\Anaconda\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "D:\Anaconda\lib\multiprocessing\process.py", line 114, in run
self._target(_self._args, *_self._kwargs)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools\train_faster_rcnn_alt_opt.py", line 130, in train_rpn
max_iters=max_iters)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\fast_rcnn\train.py", line 134, in train_net
pretrained_model=pretrained_model)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\fast_rcnn\train.py", line 43, in init
self.solver = caffe.SGDSolver(solver_prototxt)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\roi_data_layer\layer.py", line 125, in setup
top[idx].reshape(1, self._num_classes * 4)
IndexError: Index out of range

Anyone knows how to fix it?

I'm trying to train a ZF network on one class "person" but I'm getting the following error when I run the code:

File "/home/edward/caffee/r-cnn/faster-rcnn/py-faster-rcnn/tools/../lib/rpn/proposal_target_layer.py", line 138, in _get_bbox_regression_labels
bbox_targets[ind, start:end] = bbox_target_data[ind, 1:]
ValueError: could not broadcast input array from shape (4) into shape (0)

I initialize the training process by running the following command:
./experiments/scripts/faster_rcnn_end2end.sh 0 ZF

I changed the following files:
models/ZF/faster_rcnn_end2end/train.prototxt

name: "ZF"
layer {
name: 'input-data'
type: 'Python'
top: 'data'
top: 'im_info'
top: 'gt_boxes'
python_param {
module: 'roi_data_layer.layer'
layer: 'RoIDataLayer'
param_str: "'num_classes': 21"
}
}

to

name: "ZF"
layer {
name: 'input-data'
type: 'Python'
top: 'data'
top: 'im_info'
top: 'gt_boxes'
python_param {
module: 'roi_data_layer.layer'
layer: 'RoIDataLayer'
param_str: "'num_classes': 2"
}

}

layer {
name: "cls_score"
type: "InnerProduct"
bottom: "fc7"
top: "cls_score"
param { lr_mult: 1.0 }
param { lr_mult: 2.0 }
inner_product_param {
num_output: 21
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}

to

layer {
name: "cls_score"
type: "InnerProduct"
bottom: "fc7"
top: "cls_score"
param { lr_mult: 1.0 }
param { lr_mult: 2.0 }
inner_product_param {
num_output: 2
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}

layer {
name: "bbox_pred"
type: "InnerProduct"
bottom: "fc7"
top: "bbox_pred"
param { lr_mult: 1.0 }
param { lr_mult: 2.0 }
inner_product_param {
num_output: 84
weight_filler {
type: "gaussian"
std: 0.001
}
bias_filler {
type: "constant"
value: 0
}
}
}

to

layer {
name: "bbox_pred"
type: "InnerProduct"
bottom: "fc7"
top: "bbox_pred"
param { lr_mult: 1.0 }
param { lr_mult: 2.0 }
inner_product_param {
num_output: 8
weight_filler {
type: "gaussian"
std: 0.001
}
bias_filler {
type: "constant"
value: 0
}
}

}

I also changed :

lib/datasets/pascal_voc.py
Line 28:

    self._classes = ('__background__', # always index 0
                     'person', 'bicycle', 'bird', 'boat',
                     'bottle', 'bus', 'car', 'cat', 'chair',
                     'cow', 'diningtable', 'dog', 'horse',
                     'motorbike', 'person', 'pottedplant',
                     'sheep', 'sofa', 'train', 'tvmonitor')

to

    self._classes = ('__background__', # always index 0
                     'person')

I would like to document how to train a faster r-cnn network on a single class, by documenting how a single class is trained I and other should then be able to more easily adapted the code to train multiple classes as well as train using our own data-sets.

Size of a trained VGG16 model is quite big. Is it possible to replace VGG16 with Googlenet/Inception to reduce size of parameters?

I am able to use py-faster-rcnn to do training with voc-2007 dataset.
However, I just need 4 classes, car, person, dog and cat instead of 20 classes in the default setting.
What changes do I need to do for the above 4 classes training with voc-2007 data?
Will 4 classes model reduce computation, size of the trained model and improve accuracy?

Thanks,
Kaishi,

Dear all,

I have this error when trying to compile gpu_nms.cpp
(using python setup.py build_ext --inplace)

In function 'PyObject* pyx_pf_3nms_7gpu_nms_gpu_nms(PyObject, PyArrayObject, PyObject_, _pyx_t_5numpy_int32_t)':
error: invalid conversion from 'pyx_t_5numpy_int32_t {aka long int}' to 'int' [-fpermissive]

In file included from nms\gpu_nms.cpp:253:0:
nms\gpu_nms.hpp:1:6: error: initializing argument 1 of 'void nms(int, int_, const float_, int, int, float, int)' [-fpermissive]
nms\gpu_nms.cpp: In function 'void _Pyx_RaiseArgtupleInvalid(const char, int, Py_ssize_t, Py_ssize_t, Py_ssize_t)':

I am using windows 7 (64-bit)
My python is:
Python 2.7.10 |Anaconda 2.4.0 (64-bit)|

My numpy is:
Metadata-Version: 2.0
Name: numpy
Version: 1.9.3

Thank you very much.

The link to slides in README.md lands on a 404 page:
https://www.dropbox.com/s/gpvbaf9o4et9d5v/iccv15_tutorial_training_faster.pdf?dl=0

Hello,

I have just tried the recently updated end-to-end training, but it seems that the selective search pre-computed results are needed during training. Has anyone tried end-to-end without selective search? I have modified the code before to tailor my own dataset, and I have traced it to see if I modified anything related, but found nothing helpful. Any suggestions? Thank you in advance.

Hi.
When I run demo.py, I can get to know the number of object proposals, like this.

Demo for data/demo/0.jpg
Detection took 0.065s for 79 object proposals

I would like to increase this.
Where can I change the number of object proposals?
Thanks

Hi, I'm trying to use py-faster-rcnn with 2 classes, person and bottle.
However, I got the following error.

I already changed the code in pascal_voc.py at line 28, like

self._classes = ('__background__', # always index 0
                         'bottle', 'person')

And I also changed /models/VGG_CNN_M_1024/faster_rcnn_alt_opt/*train(test).pt

num_classes 21 → 3
bbox_num     84 → 12

After that, I run ./tools/train_faster_rcnn_alt_opt.py --gpu 0 --net_name VGG_CNN_M_1024 --weights data/imagenet_models/VGG_CNN_M_1024.v2.caffemodel --cfg experiments/cfgs/faster_rcnn_alt_opt.yml

However, during learning process, I got this error.

Loaded dataset `voc_2007_trainval` for training
Set proposal method: rpn
Appending horizontally-flipped training examples...
voc_2007_trainval gt roidb loaded from /py-faster-rcnn/data/cache/voc_2007_trainval_gt_roidb.pkl
loading /py-faster-rcnn/output/faster_rcnn_alt_opt/voc_2007_trainval/vgg_cnn_m_1024_rpn_stage1_iter_80000_proposals.pkl
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "./tools/train_faster_rcnn_alt_opt.py", line 190, in train_fast_rcnn
    roidb, imdb = get_roidb(imdb_name, rpn_file=rpn_file)
  File "./tools/train_faster_rcnn_alt_opt.py", line 68, in get_roidb
    roidb = get_training_roidb(imdb)
  File "/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 121, in get_training_roidb
    imdb.append_flipped_images()
  File "/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 103, in append_flipped_images
    boxes = self.roidb[i]['boxes'].copy()
  File "/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 67, in roidb
    self._roidb = self.roidb_handler()
  File "/py-faster-rcnn/tools/../lib/datasets/pascal_voc.py", line 136, in rpn_roidb
    rpn_roidb = self._load_rpn_roidb(gt_roidb)
  File "/py-faster-rcnn/tools/../lib/datasets/pascal_voc.py", line 150, in _load_rpn_roidb
    return self.create_roidb_from_box_list(box_list, gt_roidb)
  File "/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 178, in create_roidb_from_box_list
    overlaps[I, gt_classes[argmaxes[I]]] = maxes[I]
 IndexError: index 9 is out of bounds for axis 1 with size 3

Could you please help?

--------------log-------
Logging output to experiments/logs/faster_rcnn_alt_opt_VGG16_.txt.2015-10-26_11-22-39
./tools/train_faster_rcnn_alt_opt.py --gpu 2 --net_name VGG16 --weights data/imagenet_models/VGG16.v2.caffemodel --imdb voc_2007_trainval --cfg experiments/cfgs/faster_rcnn_alt_opt.yml
Called with args:
Namespace(cfg_file='experiments/cfgs/faster_rcnn_alt_opt.yml', gpu_id=2, imdb_name='voc_2007_trainval', net_name='VGG16', pretrained_model='data/imagenet_models/VGG16.v2.caffemodel', set_cfgs=None)
Traceback (most recent call last):
File "./tools/train_faster_rcnn_alt_opt.py", line 224, in
mp_queue = mp.Queue()
File "/mark/caffe-soft-bin/anaconda/lib/python2.7/multiprocessing/init.py", line 218, in Queue
return Queue(maxsize)
File "/mark/caffe-soft-bin/anaconda/lib/python2.7/multiprocessing/queues.py", line 63, in init
self._rlock = Lock()
File "/mark/caffe-soft-bin/anaconda/lib/python2.7/multiprocessing/synchronize.py", line 147, in init
SemLock.init(self, SEMAPHORE, 1, 1)
File "/mark/caffe-soft-bin/anaconda/lib/python2.7/multiprocessing/synchronize.py", line 75, in init
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 13] Permission denied

I found the solution from http://stackoverflow.com/questions/2009278/python-multiprocessing-permission-denied, but I don't have root privilege. So, is there any way to solve this problem?

Hi!

I am trying to train on my own dataset which consists of 3 classes. I have already been able to train on VOC2007 dataset with less classes so I am quite sure that the problem isn't caused by different number of classes.

I am able to successfully finish training and evaluate it, however for one of these 3 classes I get 0 mAP. With digging further I found out that while training sometimes there appears numpy RuntimeWarning: invalid value encountered in log. This warning is due to negative value in log function.

In lib/fast_rcnn/bbox_transform.py on line 16 there are two vectors gt_rois[:, 2] and gt_rois[:, 0] which are deducted and then later on log function is applied on their difference. In some cases their difference is suprisingly negative. The pair of numbers is usually like (12.809, 111.236), (161.667, 291.667), (636.667, 788.333) but in these problem cases the first number is much larger (98302.5, 591). The gt_rois array is passed from lib/rpn/anchor_target_layer.py inside forward() method.

At first I thought that problem could be with data, so I checked it and deleted some images which were not RGB (they were part of the unsuccessfully trained class). I have also modified some of xmax and ymax in order to allow only maximum value within range [0, (width-1)] and [0, (height-1)], respectively. Nonetheless, any of these changes helped and I still receive RuntimeWarning: invalid value encountered in log at some points of training.

Any idea what wrong could be with data? Or how could I further track these large values? I know that that mentioned forward() method is activated from lib/fast_rcnn/train.py by self.solver.step(1) command, but I still haven't found a place where bottom parameter containing data is passed.

Thank you!

Martin

If run demo.py in cpu , everything goes fine.
If debug demo.py in gpu mode, step by step, Ok. But if I press the coninue debug button in eclipse never button-up, ubuntu system crash.
If run demo.py in gpu, either net = caffe.Net(prototxt, caffemodel, caffe.TEST) or _, _= im_detect(net, im), or scores, boxes = im_detect(net, im), will crash the sytem randomly.
So, if the demo.py ran slowly, no problem, but faster in gpu will crash.
I use titanx, so memory is enough.
If I ran faster rcnn matlab version, the same thing happens.
I see the logfile(in eclipse run-->debug configuration-->Common-->Standard IO,File , I choose a file), nothing seems abnormal.

Hi all, I am now training the py-faster-rcnn on ImageNet detections data. I create an ilsvrc.py in ./lib/datasets for the ImageNet dataset and modified related codes. My training goes smoothly in the Stage 1 RPN Training and Stage 1 Generate RPN Porposals. But in Stage 1 Fast RCNN Training using RPN Proposals, I came across the following error. I check the related codes but still have no idea of what is wrong.

Could anyone gives me some suggestions? Thanks!

Traceback (most recent call last):
  File "/home/code/lijianchao/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/code/lijianchao/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "./tools/train_faster_rcnn_alt_opt.py", line 196, in train_fast_rcnn
    max_iters=max_iters)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 138, in train_net
    model_paths = sw.train_model(max_iters)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 104, in train_model
    self.solver.step(1)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 143, in forward
    blobs = self._get_next_minibatch()
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 62, in _get_next_minibatch
    return get_minibatch(minibatch_db, self._num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 55, in get_minibatch
    num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 125, in _sample_rois
    roidb['bbox_targets'][keep_inds, :], num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 176, in _get_bbox_regression_labels
    bbox_targets[ind, start:end] = bbox_target_data[ind, 1:]
ValueError: could not broadcast input array from shape (4) into shape (0)

I am testing a model to know the results of mAP of VGG16 faster rcnn model. I have pasted error. Please help me to resolve it.

Command I used:
python tools/test_net.py --net=py-faster-rcnn/data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel --imdb=voc_2007_train --def=py-faster-rcnn/models/VGG16/faster_rcnn_alt_opt/faster_rcnn_test.pt

Error Output:

Error using textread (line 166)
File not found.

Error in VOCevaldet (line 30)
[ids,confidence,b1,b2,b3,b4]=textread(sprintf(VOCopts.detrespath,id,cls),'%s %f %f %f %f %f');

(output of VOCinit.m variables)
dataset: 'VOC2007'
datadir: 'py-faster-rcnn/VOCdevkit/'
resdir: 'py-faster-rcnn/VOCdevkit/results/VOC2007/'
localdir: 'py-faster-rcnn/VOCdevkit/local/VOC2007/'
testset: 'train'
annopath: 'py-faster-rcnn/VOCdevkit/VOC2007/Annotations/%s.xml'
imgpath: 'py-faster-rcnn/VOCdevkit/VOC2007/JPEGImages/%s.jpg'
imgsetpath: 'py-faster-rcnn/VOCdevkit/VOC2007/ImageSets/Main/%s.txt'
clsimgsetpath: 'py-faster-rcnn/VOCdevkit/VOC2007/ImageSets/Main/%s_%s.txt'
clsrespath: 'py-faster-rcnn/VOCdevkit/results/VOC2007/Main/%s_cls_test_%s.txt'
detrespath: 'py-faster-rcnn/VOCdevkit/results/VOC2007/Main/%s_det_test_%s.txt'
seg: [1x1 struct]
layout: [1x1 struct]
classes: {20x1 cell}
nclasses: 20
poses: {5x1 cell}
nposes: 5
parts: {3x1 cell}
maxparts: [1 2 2]
nparts: 3
minoverlap: 0.5000
exannocachepath: 'py-faster-rcnn/VOCdevkit/local/VOC2007/%s_anno.mat'
exfdpath: 'py-faster-rcnn/VOCdevkit/local/VOC2007/%s_fd.mat'

I want to train a model on ImageNet detection dataset, where should I start? Thx.

Hi All:

I am using py-faster-rcnn to train on my own dataset. The program is stuck and I don't know whether it is still running without outputs or it is stuck because of some errors? It is still in stage 1.

Msgs are listed as below:

'TEST': {'BBOX_REG': True,
'HAS_RPN': False,
'MAX_SIZE': 1000,
'NMS': 0.3,
'PROPOSAL_METHOD': 'selective_search',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECT_GROUPING': True,
'BATCH_SIZE': 128,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': False,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.1,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'HAS_RPN': False,
'IMS_PER_BATCH': 2,
'MAX_SIZE': 1000,
'PROPOSAL_METHOD': 'rpn',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 16,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_INFIX': 'stage1',
'SNAPSHOT_ITERS': 10000,
'USE_FLIPPED': True,
'USE_PREFETCH': False},
'USE_GPU_NMS': True}
Loaded dataset train for training
Set proposal method: rpn
Appending horizontally-flipped training examples...

Thanks in advance!!

I have NVidia 960 GPU card with 4 GB RAM and tools/demo.py complains insufficient memory with VGG16 model. ZF model is running OK.
Is there a way to reduce GPU memory usage with VGG16 model?

Thanks,
Kaishi

Hi all:
I would like to train py-faster-rcnn on my own dataset, but what is exactly the data format? i.e. the images, annotations, train, val? Can anyone post an example of annotation file here? As I found one post here: https://github.com/zeyuanxy/fast-rcnn/tree/master/help/train, in which the annotation is text, while I also found PASCAL is XML. What exactly it is? Any reference blogs or tutorials will be highly appreciated. Thanks

I have installed and compiled all the prerequisites and I am getting following error while running
./tools/demo.py.

Traceback (most recent call last):
File "./tools/demo.py", line 18, in
from fast_rcnn.test import im_detect
File "/path/to/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 17, in
from fast_rcnn.nms_wrapper import nms
File "/path/to/py-faster-rcnn/tools/../lib/fast_rcnn/nms_wrapper.py", line 10, in
from nms.cpu_nms import cpu_nms
ImportError: /path/to/py-faster-rcnn/tools/../lib/nms/cpu_nms.so: undefined symbol: PyFPE_jbuf

cpu_nms.so is there in the directory but still I am getting error
Please help me in resolving this.

I am trying to train a model in dataset with 61 classes including background and hence i set the num of output in prototxt file to 61 and that of bbox reggression to 244.
But I am getting this error during stage1 fast rcnn training. Unable to figure it out.

self.bbox_stds[:, np.newaxis]
ValueError: operands could not be broadcast together with shapes (244,4096) (240,1)

1229 13:48:13.383297 3136 net.cpp:283] This network produces output cls_prob
I1229 13:48:13.383335 3136 net.cpp:297] Network initialization done.
I1229 13:48:13.383350 3136 net.cpp:298] Memory required for data: 117093580
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 548317115

Loaded network /opt/practice/py-faster-rcnn/data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel
F1229 13:48:14.865103 3136 syncedmem.cpp:58] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
Aborted (core dumped)
ubuntu@ip-172-30-0-107:/opt/practice/py-faster-rcnn/tools$

Currently if you run the script ./experiments/scripts/faster_rcnn_alt_opt.sh [GPU_ID] [NET] it fails during testing mode with: Input blob arguments do not match net inputs

I traced that back to HAS_RPN = False in lib/fast_rcnn/config.py even though the experiments/cfgs/faster_rcnn_alt_opt.yml sets it to True.

I believe the .yml file should override the default settings given by config.py, is that correct?

I noticed line 47 of config.py the code is:
# These are the values originally used for training VGG16
__C.PIXEL_MEANS = np.array([[[102.9801, 115.9465, 122.7717]]])
But the original VGG16 trained model on model zoo is [103.939, 116.779, 123.68]
Source: https://gist.github.com/ksimonyan/211839e770f7b538e2d8#file-readme-md (linked from author’s project page).
I know the difference is small, and probably doesn’t have a significant impact. But the fact that these numbers are not the same here may indicate the original VGG16 on model zoo is different from the one in Faster RCNN. But Faster RCNN claim it was the original for VGG16. I know in the VGG16 paper there are different kinds of 16 layer models and maybe it uses a different one? But those other 16 layer models aren’t published in their original form the best of my knowledge. Anyway, this inconsistency may mean other things are inconsistent.
Thanks.

Firstly, thanks for openning the faster rcnn code. When I try to transform the python demo into the c++ one. I met a problem when I load the model. I debuged it and found that in layer_factory.cpp the code:
"bp::object layer = module.attr(param.python_param().layer().c_str())(param); "not success. Is that a bug that c++ cann't use the python layer or I just made a mistake??
PS: I have put the lib path in the PYTHONPATH already.
Thank you!!!!

I1207 16:32:10.283551 732336 python_layer.hpp:66] Traceback (most recent call last):
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\rpn\proposal_layer.py", line 143, in forward
keep = nms(np.hstack((proposals, scores)), nms_thresh)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\fast_rcnn\nms_wrapper.py", line 17, in nms
return cpu_nms(dets, thresh)
File "nms\cpu_nms.pyx", line 25, in nms.cpu_nms.cpu_nms (nms\cpu_nms.c:1756)
cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1]
ValueError: Buffer dtype mismatch, expected 'int_t' but got 'long long'
Traceback (most recent call last):
File "./tools/demo.py", line 142, in
_, _= im_detect(net, im)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\fast_rcnn\test.py", line 155, in im_detect
blobs_out = net.forward(**forward_kwargs)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\caffe-fast-rcnn\python\caffe\pycaffe.py", line 95, in _Net_forward
self._forward(start_ind, end_ind)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\rpn\proposal_layer.py", line 143, in forward
keep = nms(np.hstack((proposals, scores)), nms_thresh)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\fast_rcnn\nms_wrapper.py", line 17, in nms
return cpu_nms(dets, thresh)
File "nms\cpu_nms.pyx", line 25, in nms.cpu_nms.cpu_nms (nms\cpu_nms.c:1756)
ValueError: Buffer dtype mismatch, expected 'int_t' but got 'long long'

On line 174 - 175 of lib/rpn/anchor_target_layer.py, I found that bbox_targets is defined twice:

bbox_targets = np.zeros((len(inds_inside), 4), dtype=np.float32)
bbox_targets = _compute_targets(anchors, gt_boxes[argmax_overlaps, :])

I didn't see the first bbox_targets used in the second definition. Probably it is redundant?

py-faster-rcnn runs well in terminal, but when i do a web demo with webpy, it doesn't work, the problem occurs in net.forward(**forward_kwargs). Sometimes forward runs slow with gpu mode, the time costs as cpu mode, sometimes it core dump. And for the first image , it runs well, but after first image, it will run slow as cpu mode randomly. Does anyone can solve this problem?
With fast-rcnn, webpy runs well.
@rbgirshick

Hi!

After did the basic installation, when I ran the demo.py code, I got the check error.

F1210 08:16:15.441911 10686 layer_factory.hpp:80] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: Python (known types: AbsVal, Accuracy, ArgMax, BNLL, Concat, ContrastiveLoss, Convolution, Data, Deconvolution, Dropout, DummyData, Eltwise, Embed, EuclideanLoss, Exp, Filter, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, LRN, Log, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Pooling, Power, ROIPooling, ReLU, Reduction, Reshape, SPP, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, SmoothL1Loss, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, Tile, WindowData)

how to fix this?

I am using py-faster-rcnn to train the 2007 VOC dataset. When I run on the original 21 classes, it works fine, but when I change the original number of classes, I get an error..
I am doing this for 1 class ('person'). I made the following changes to the original codes:

1. file: fcn_root/models/VGG16/faster_rcnn_alt_opt.pt
   layer {
   name: "cls_score"
   type: "InnerProduct"
   bottom: "fc7"
   top: "cls_score"
   inner_product_param {
   num_output: 2
   }
   }

   layer {
   name: "bbox_pred"
   type: "InnerProduct"
   bottom: "fc7"
   top: "bbox_pred"
   inner_product_param {
   num_output: 8
   }
   }

2. file2: fcn_root/lib/datasets/pascal_voc.py

   line 28:self._classes = ('background', # always index 0
   'person')

   removed -1's from the lines below

        x1 = float(get_data_from_tag(obj, 'xmin'))
         y1 = float(get_data_from_tag(obj, 'ymin'))
         x2 = float(get_data_from_tag(obj, 'xmax')) 
         y2 = float(get_data_from_tag(obj, 'ymax'))
   

3. file3: ../VOCdevkit/VOCcode/VOCinit.m

VOCopts.classes={...
'person'};

I get the following error after I run this command,

./experiments/scripts/faster_rcnn_alt_opt.sh 3 VGG16 \ --set EXP_DIR foobarx

I have also attached a full trace of my program running.
Error:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(_self._args, *_self._kwargs)
File "./tools/train_faster_rcnn_alt_opt.py", line 123, in train_rpn
roidb, imdb = get_roidb(imdb_name)
File "./tools/train_faster_rcnn_alt_opt.py", line 68, in get_roidb
roidb = get_training_roidb(imdb)
File "/home/microway/test/pytest/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 121, in get_training_roidb
imdb.append_flipped_images()
File "/home/microway/test/pytest/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 108, in append_flipped_images
assert (boxes[:, 2] >= boxes[:, 0]).all()
AssertionError

error_attachment.txt

Fast R-CNN trains the final sibling fully connected layers for object class aware regions classification and regression using mini-batches of region proposals. RPN trains similar sibling fc layers for object vs. non-object exhaustively on multiple anchor boxes at each position of the final conv layer feature maps. One of the aims of Fast R-CNN is to unify the multi-stage training of previous methods. But RPN uses a 4-step alternating training scheme to share conv layers with the Fast R-CNN. Isn't that a step going backwards? The situation is even more evident in the special case of 1-class object detection such as #10.

If the classification and regression tasks in both Fast R-CNN and RPN can share the conv layers in multi-task learning settings, why can't the four tasks be trained together? The final four sibling fc layers can all use the dense anchor boxes as input collapsing the two-step cascade into a single step and eliminating the separate RPN training and testing completely which is exactly the prediction in the conclusion of the Fast R-CNN paper.

@rbgirshick, @ShaoqingRen what're you opinions on the plausibility?

Hi @rbgirshick , is there any way to compress the trained models using faster-rcnn?

How does joint training work？
I wonder how to coordinate the training between RPN and RCNN? Is it achieved by update the RPN with the RCNN fixed and then the opposite？ To be clearer, is the learning driven by RPN loss in one BackPropagation and the next time by RCNN loss？？

how to change the batch size when training the rpn model, since i have a large dataset, and the code so far, the batch size is 1. And by the way, if change the batch size of rpn training, the performace will increase?
Thanks

I built py-faster-rcnn/lib/setup.py with specifying nvcc -ccbin=/usr/bin/g++-4.9
and built py-faster-rcnn/caffe-fast-rcnn/Makefile with -ccbin=/usr/bin/g++-4.9 added to NVCCFLAGS

no error show up during "make -j 32 && make pycaffe"
but when I run tools/demo.py, the following error occurs:
============= error message ===================
Traceback (most recent call last):
File "demo.py", line 138, in
net = caffe.Net(prototxt, caffemodel, caffe.TEST)
Boost.Python.ArgumentError: Python argument types in
Net.init(Net, str, str, int)
did not match C++ signature:
init(boost::python::api::object, std::string, std::string, int)

init(boost::python::api::object, std::string, int)

any idea what happened?
Why would it says signature did not match? Signature (str, str, int) and (std::string, std::string, int) seems perfectly match for me

Hi,

I had no issues when compiling both py-faster-rcnn/lib and caffe-fast-rcnn. I've been able to successfully compile using CMake and the Makefile.config method. After making sure my PATH and PYTHONPATH vars were updated properly, I tried to run the demo...

$ python tools/demo.py
[libprotobuf ERROR google/protobuf/descriptor_database.cc:57] File already exists in database: caffe.proto
[libprotobuf FATAL google/protobuf/descriptor.cc:1018] CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
Aborted

I've come across BVLC/caffe#1917 but wasn't sure how to approach fixing the "issue" - it seems like it was fixed in a subsequent version of caffe.

Any ideas?

After train It will call the ./tools/test_net.py. I got following error.

File "./tools/test_net.py", line 85, in <module>
    test_net(net, imdb)
  File "/home/foo/alibaba/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 253, in test_net
    roidb = imdb.roidb
  File "/home/foo/alibaba/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 67, in roidb
    self._roidb = self.roidb_handler()
  File "/home/foo/alibaba/py-faster-rcnn/tools/../lib/datasets/pascal_voc.py", line 137, in selective_search_roidb
    roidb = self._load_selective_search_roidb(None)
  File "/home/foo/alibaba/py-faster-rcnn/tools/../lib/datasets/pascal_voc.py", line 168, in _load_selective_search_roidb
    'Selective search data not found at: {}'.format(filename)
AssertionError: Selective search data not found at: /home/foo/alibaba/py-faster-rcnn/data/selective_search_data/voc_2012_\
test.mat

Why call selective search in faster rcnn not RPN?

I revised the codes in lib/datasets to train on the ImageNet detection data: I create an ilsvrc.py (like the pascal_voc.py) to account for the ImageNet data and modify the corresponding codes in factory.py. Then I run the experiment script ./experiments/scripts/faster_rcnn_alt_opt.sh and everything seems to be correct at first, but then the training seems to get stuck with the following information never moving forward again...

Loading pretrained model weights from data/imagenet_models/ZF.v2.caffemodel
Solving...
I1101 15:18:12.302584 40312 solver.cpp:242] Iteration 0, loss = 1.10525
I1101 15:18:12.302647 40312 solver.cpp:258]     Train net output #0: rpn_cls_loss = 0.785905 (* 1 =     0.785905 loss)
I1101 15:18:12.302659 40312 solver.cpp:258]     Train net output #1: rpn_loss_bbox = 0.319344 (* 1 = 0.319344 loss)
I1101 15:18:12.302670 40312 solver.cpp:571] Iteration 0, lr = 0.001

Could anyone give me some suggestions? Thanks!

Hi All,
I have successfully trained faster rcnn on my own dataset. It works with high recall but the precision is quite low with some false positives. I wonder if there is a way to explicitly add negative examples (i.e. as the background) to improve the precision? For example, I can add these false positive areas (from testing) as the negative samples to train a new model.

If yes, then how? thanks!!

There is no problem for me to run the demo.py of fast-rcnn, however, I had the error as follows when I try to run the demo.py of py-faster-rcnn after successfully make -j8 & make pycaffe
Loaded network /home/ubuntu/py-faster-rcnn/data/faster_rcnn_models/ZF_faster_rcnn_final.caffemodel
F1008 04:30:16.139123 5360 roi_pooling_layer.cu:91] Check failed: error == cudaSuccess (8 vs. 0) invalid device function
*** Check failure stack trace: ***

Anyone has the same problem?

Hi, I'm trying to compile this python version of Faster-RCNN on Windows.

The modification was like:

if 'CUDA_PATH_V7_0' in os.environ:
    home = os.environ['CUDA_PATH_V7_0']
    nvcc = pjoin(home, 'bin', 'nvcc.exe')

I fixed the path to fit Windows, and conquered some minor issues. Then I compiled utils\bbox.c, nms\cpu_nms.c, and nms.cpp succesfully. However, I failed when building nms.gpu_nms. Here're the triggered errors:

error: Don't know how to compile nms/nms_kernel.cu to build\temp.win-amd64-2.7\Release\nms/nms_lernel.obj

or sometimes

TypeError: can only concatenate list (not "dist") to list

Did you successfully run this code on Windows?

I am training the model on ImageNet and already finished stage 1, I have zf_rpn_stage1_iter_80000.caffemodel and zf_rpn_stage1_iter_80000_proposals.pkl in my output/faster_rcnn_alt_opt/train/ folder. But then I encountered this error:

File "/home/Documents/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 100, in _sample_rois
    fg_inds, size=fg_rois_per_this_image, replace=False)
  File "mtrand.pyx", line 1070, in mtrand.RandomState.choice (numpy/random/mtrand/mtrand.c:8044)
TypeError: 'numpy.float64' object cannot be interpreted as an index

So the training stacked. After fixing this error, can I start training from the next stage directly? I don't want to do the 80000 iterations from the Stage 1 RPN, init from ImageNet model stage again.

Loaded network F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\data\faster_rcnn_models\VGG16_faster_rcnn_final.caffemodel
I1207 10:04:07.956500 764992 python_layer.hpp:66] Traceback (most recent call last):
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\rpn\proposal_layer.py", line 63, in forward
cfg_key = str(self.phase) # either 'TRAIN' or 'TEST'
AttributeError: 'ProposalLayer' object has no attribute 'phase'
Traceback (most recent call last):
File "./tools/demo.py", line 142, in
_, _= im_detect(net, im)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\fast_rcnn\test.py", line 155, in im_detect
blobs_out = net.forward(**forward_kwargs)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\caffe-fast-rcnn\python\caffe\pycaffe.py", line 95, in _Net_forward
self._forward(start_ind, end_ind)
File "F:\sunqin\DeepLearning\Caffe\Projects\faster-rcnn\tools..\lib\rpn\proposal_layer.py", line 63, in forward
cfg_key = str(self.phase) # either 'TRAIN' or 'TEST'
AttributeError: 'ProposalLayer' object has no attribute 'phase'