makefile / frcnn Goto Github PK

I can train the rfcn_res50,but when I run the convert_model.py,I got a problem:"Key error,bbox_pred_pre",And there isn't 'bbox_pred_pre' layer in test.protext.

如题，编译caffe时出现这个错误，在Makefile里面也加上了-fopenmp，还是报同样的错误

math_functions.cu:130] Check failed: status == CUBLAS_STATUS_SUCCESS (14 vs. 0) CUBLAS_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
@ 0x7f2a5c719b4d google::LogMessage::Fail()
@ 0x7f2a5c71db67 google::LogMessage::SendToLog()
@ 0x7f2a5c71b9e9 google::LogMessage::Flush()
@ 0x7f2a5c71bced google::LogMessageFatal::~LogMessageFatal()
@ 0x7f2a63d10256 caffe::caffe_gpu_dot<>()
@ 0x7f2a63cd4ffc caffe::SoftmaxWithLossLayer<>::Forward_gpu()
@ 0x7f2a63b108cc caffe::Net<>::ForwardFromTo()
@ 0x7f2a63b10ca7 caffe::Net<>::Forward()
@ 0x7f2a6398a308 caffe::Solver<>::Step()
@ 0x7f2a63ae78ea caffe::Worker<>::InternalThreadEntry()
@ 0x7f2a63afb0d0 caffe::InternalThread::entry()
@ 0x7f2a63afb8f6 boost::detail::thread_data<>::run()
@ 0x7f2a5800ee83 thread_proxy
@ 0x7f2a4abbb1c3 start_thread
@ 0x7f2a4a8ed12d __clone

when i use single gpu, the model can train normaly. But when I use multi-gpu to train "sh examples/FRCNN/zf/train_frcnn.sh", it reminds me that ...

@makefile hi，I have successfully run rfcn-resnet50 code, when I use fpn-resnet50(both master branches and fpn branches have tried) get this erro.

How to train on coco anyway?

I0718 10:47:16.587196 12924 net.cpp:256] Network initialization done.
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 140258106165888:
#000: ../../../src/H5F.c line 439 in H5Fis_hdf5(): unable open file
major: File accessibilty
minor: Not an HDF5 file
#1: ../../../src/H5Fint.c line 554 in H5F_is_hdf5(): unable to open file
major: Low-level I/O
minor: Unable to initialize object
#2: ../../../src/H5FD.c line 993 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#3: ../../../src/H5FDsec2.c line 339 in H5FD_sec2_open(): unable to open file: name = 'models/FRCNN/VGG16_faster_rcnn_final.caffemodel', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0

when i run sh examples/FRCNN/res50/test_frcnn.sh 0 get this erro
I0719 11:41:09.491593 2885 frcnn_param.cpp:232] iter_test : -1
F0719 11:41:09.491624 2885 io.cpp:45] Check failed: fd != -1 (-1 vs. -1) File not found: models/FRCNN/res50/test_inference.prototxt
*** Check failure stack trace: ***
@ 0x7f169f8065cd google::LogMessage::Fail()
@ 0x7f169f808433 google::LogMessage::SendToLog()
@ 0x7f169f80615b google::LogMessage::Flush()
@ 0x7f169f808e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f169ff0a930 caffe::ReadProtoFromTextFile()
@ 0x7f169ff49516 caffe::ReadNetParamsFromTextFileOrDie()
@ 0x7f169fe2cd12 caffe::Net<>::Net()
@ 0x7f16a0069dd3 FRCNN_API::Detector::Set_Model()
@ 0x40469d main
@ 0x7f169e252830 __libc_start_main
@ 0x407089 _start
@ (nil) (unknown)
Command terminated by signal 6
0.11user 0.19system 0:00.46elapsed 67%CPU (0avgtext+0avgdata 322720maxresident)k
1768inputs+32outputs (3major+49767minor)pagefaults 0swaps
Called with args:
Namespace(ans_file='examples/FRCNN/results/voc2007_test_res50_2883.frcnn', gt_file='examples/FRCNN/dataset/voc2007.test', overlap=0.5)
Results File(examples/FRCNN/results/voc2007_test_res50_2883.frcnn) does not exist

Hi, I compile follow the steps：

1. cd $CAFFE_ROOT
2. cp Makefile.config.example Makefile.config
3. uncomment the line: USE_CUDNN := 1
4. comment the *_20 and *_21 lines
5. make -j7

and I get errors like that:
.build_release/tools/encrypt_model.o: In function main': encrypt_model.cpp:(.text.startup+0x79): undefined reference to Blowfish::Blowfish(std::vector<char, std::allocator > const&)'
encrypt_model.cpp:(.text.startup+0xbd): undefined reference to Blowfish::Decrypt(char const*, char const*)' encrypt_model.cpp:(.text.startup+0x113): undefined reference to Blowfish::Encrypt(char const*, char const*)'
collect2: error: ld returned 1 exit status
Makefile:711: recipe for target '.build_release/tools/encrypt_model.bin' failed
make: *** [.build_release/tools/encrypt_model.bin] Error 1
make: *** Waiting for unfinished jobs....
.build_release/tools/convert_imageset.o: In function main': convert_imageset.cpp:(.text.startup+0xf2a): undefined reference to caffe::ReadImageToDatum(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, int, int, int, int, bool, std.::build_release__cxx11/:tools:/basic_string<convert_annoset.ochar:, Instd :function: char_traits<mainchar'>:, convert_annoset.cppstd::(:.allocatortext.startup<+char0x9f9>) :> undefinedconst &reference, tocaffe ::caffeDatum::)LabelMap':
:~LabelMap()'
convert_annoset.cpp:(.text.startup+0xc00): undefined reference to caffe::AnnotatedDatum::AnnotatedDatum()' convert_annoset.cppcollect2: error: ld returned 1 exit status :(.text.startup+0xd6f): undefined reference to caffe::AnnotatedDatum::~AnnotatedDatum()'
convert_annoset.cpp:(.text.startup+0xebd): undefined reference to caffe::LabelMap::LabelMap()' convert_annoset.cpp:(.text.startup+0xeea): undefined reference to caffe::MapNameToLabel(caffe::LabelMap const&, bool, std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, int> > >)'
convert_annoset.cpp:(.text.startup+0x12df): undefined reference to caffe::ReadImageToDatum(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&Makefile:711: recipe for target '.build_release/tools/convert_imageset.bin' failed , caffemake: *** [.build_release/tools/convert_imageset.bin] Error 1 ::Datum*)' convert_annoset.cpp:(.text.startup+0x1ec5): undefined reference to caffe::ReadRichImageToAnnotatedDatum(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, int, int, int, bool, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, caffe::AnnotatedDatum_AnnotationType, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, int> > > const&, caffe::AnnotatedDatum*)'
convert_annoset.cpp:(.text.startup+0x2364): undefined reference to caffe::AnnotatedDatum::~AnnotatedDatum()' convert_annoset.cpp:(.text.startup+0x2436): undefined reference to caffe::LabelMap::~LabelMap()'
collect2: error: ld returned 1 exit status
Makefile:711: recipe for target '.build_release/tools/convert_annoset.bin' failed
make: *** [.build_release/tools/convert_annoset.bin] Error 1
.build_release/tools/extract_features.o: In function int feature_extraction_pipeline<float>(int, char**)': extract_features.cpp:(.text._Z27feature_extraction_pipelineIfEiiPPc[_Z27feature_extraction_pipelineIfEiiPPc]+0x1d3): undefined reference to caffe::Net::CopyTrainedLayersFrom(std::__cxx11::basic_string<char, std::char_traits, std::allocator >)'
collect2: error: ld returned 1 exit status
Makefile:711: recipe for target '.build_release/tools/extract_features.bin' failed
make: *** [.build_release/tools/extract_features.bin] Error 1
.build_release/tools/convert_annoset_r.o: In function main': convert_annoset_r.cpp:(.text.startup+0x9f9): undefined reference to caffe::LabelMap::~LabelMap()'
convert_annoset_r.cpp:(.text.startup+0xc00): undefined reference to caffe::AnnotatedDatumR::AnnotatedDatumR()' convert_annoset_r.cpp:(.text.startup+0xd6f): undefined reference to caffe::AnnotatedDatumR::~AnnotatedDatumR()'
convert_annoset_r.cpp:(.text.startup+0xeba): undefined reference to caffe::LabelMap::LabelMap()' convert_annoset_r.cpp:(.text.startup+0x159f): undefined reference to caffe::ReadImageToDatum(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, int, int, int, int, bool, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, caffe::Datum*)'
convert_annoset_r.cpp:(.text.startup+0x21d1): undefined reference to caffe::ReadRichImageToAnnotatedDatumR(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, caffe::AnnotatedDatumR_AnnotationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> > > const&, caffe::AnnotatedDatumR*)' convert_annoset_r.cpp:(.text.startup+0x2634): undefined reference to caffe::AnnotatedDatumR::~AnnotatedDatumR()'
convert_annoset_r.cpp:(.text.startup+0x26e2): undefined reference to caffe::LabelMap::~LabelMap()' collect2: error: ld returned 1 exit status .build_release/tools/caffe.o: In function CopyLayers(caffe::Solver*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)':
caffe.cpp:(.text+0x69d): undefined reference to caffe::Net<float>::CopyTrainedLayersFrom(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)' caffe.cpp:(.text+0x727): undefined reference to caffe::Net::CopyTrainedLayersFrom(std::__cxx11::basic_string<char, std::char_traits, std::allocator >)'
.build_release/toolsMakefile:711: recipe for target '.build_release/tools/convert_annoset_r.bin' failed
/make: *** [.build_release/tools/convert_annoset_r.bin] Error 1
caffe.o: In function test()': caffe.cpp:(.text+0x16e9): undefined reference to caffe::Net::CopyTrainedLayersFrom(std::__cxx11::basic_string<char, std::char_traits, std::allocator >)'
collect2: error: ld returned 1 exit status
Makefile:711: recipe for target '.build_release/tools/caffe.bin' failed
make: *** [.build_release/tools/caffe.bin] Error 1

Why and how to fix it?

After successfully make all, I continue with make test while getting this error:

src/caffe/FRCNN/data_augment/test_augment.cpp: In function 'int test_main()':
src/caffe/FRCNN/data_augment/test_augment.cpp:32:19: warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]
  char *img_path = "test.jpg";
                   ^
src/caffe/FRCNN/data_augment/test_augment.cpp:82:94: error: no matching function for call to 'data_augment(cv::Mat&, std::vector<std::vector<float> >&, int&, float&, float&, float&, float&, float&)'
  Mat result = data_augment(origmat, rois, flip, jitter, rand_scale, hue, saturation, exposure);
                                                                                              ^
src/caffe/FRCNN/data_augment/test_augment.cpp:82:94: note: candidates are:
In file included from src/caffe/FRCNN/data_augment/test_augment.cpp:2:0:
src/caffe/FRCNN/data_augment/data_utils.hpp:55:7: note: image data_augment(image, box_label*, int, int, int, int, float, float, float, float, float)
 image data_augment(image orig, box_label *boxes, int num_boxes, int w, int h, int flip, float jitter, float scale, float hue, float saturation, float exposure);
       ^
src/caffe/FRCNN/data_augment/data_utils.hpp:55:7: note:   candidate expects 11 arguments, 8 provided
src/caffe/FRCNN/data_augment/data_utils.hpp:56:9: note: cv::Mat data_augment(cv::Mat&, std::vector<std::vector<float> >&, int, float, float, bool, float, float, float)
 cv::Mat data_augment(cv::Mat &orig, std::vector<std::vector<float> > &boxes, int flip, float jitter, float scale, bool random_rotate, float hue, float saturation, float exposure);
         ^
src/caffe/FRCNN/data_augment/data_utils.hpp:56:9: note:   candidate expects 9 arguments, 8 provided
make: *** [.build_release/src/caffe/FRCNN/data_augment/test_augment.o] Error 1
make: *** Waiting for unfinished jobs....

Couldn't find any valid solution. Any suggestions? Many Thanks!

I want to use the r-frcnn(rotation) to train my own datasets, What should my data format be ?
(cx,cy,w,h,angle) or (x0,y0,x1,y1,x2,y2,x3,y3)? But if it is the former, What is the calculation method and representation of the angle? radian or not?

看Owner是**人就用中文了(小声BB...)

我直接用caffe.exe训练了 resnet101 (models/FRCNN/res101/solver.proto VOC2007)
我一开始训练了10000步发现推理结果为0 (使用test_frcnn.exe, 直接用训练集当做测试集)
于是我继续训练到20000发现推理结果还是为0
详细调试后发现score肥肠肥肠低都是0.00x,最高的我看到也就0.05...
我浏览了其他问题, 你说可以不用convert_model.py 于是我就直接训练完的模型直接test_frcnn也不行.输出结果如下

I0828 13:00:15.272047 19684 FRCNNtest.cpp:210] Handle 97 th image : 000193.jpg, with image_thresh : 0, 0 -> 0 boxes
I0828 13:00:15.367131 19684 FRCNNtest.cpp:210] Handle 98 th image : 000194.jpg, with image_thresh : 0, 0 -> 0 boxes
I0828 13:00:15.479229 19684 FRCNNtest.cpp:210] Handle 99 th image : 000198.jpg, with image_thresh : 0, 0 -> 0 boxes
I0828 13:00:15.584322 19684 FRCNNtest.cpp:210] Handle 100 th image : 000200.jpg, with image_thresh : 0, 0 -> 0 boxes
I0828 13:00:15.687412 19684 FRCNNtest.cpp:210] Handle 101 th image : 000203.jpg, with image_thresh : 0, 0 -> 0 boxes
I0828 13:00:15.791504 19684 FRCNNtest.cpp:210] Handle 102 th image : 000207.jpg, with image_thresh : 0, 0 -> 0 boxes
I0828 13:00:15.887589 19684 FRCNNtest.cpp:210] Handle 103 th image : 000208.jpg, with image_thresh : 0, 0 -> 0 boxes

我现在不知道怎么修改了...下面是训练时一部分....我认为正常

I0828 11:53:16.375608 19192 solver.cpp:218] Iteration 19920 (2.24295 iter/s, 8.91684s/20 iters), loss = 1.99799
I0828 11:53:16.375608 19192 solver.cpp:237] Train net output #0: bbox_accuracy = 0.75
I0828 11:53:16.376610 19192 solver.cpp:237] Train net output #1: loss_bbox = 0.430361 (* 1 = 0.430361 loss)
I0828 11:53:16.376610 19192 solver.cpp:237] Train net output #2: loss_cls = 1.27145 (* 1 = 1.27145 loss)
I0828 11:53:16.376610 19192 solver.cpp:237] Train net output #3: rpn_cls_loss = 0.898545 (* 1 = 0.898545 loss)
I0828 11:53:16.376610 19192 solver.cpp:237] Train net output #4: rpn_loss_bbox = 0.391415 (* 1 = 0.391415 loss)
I0828 11:53:16.376610 19192 sgd_solver.cpp:105] Iteration 19920, lr = 0.001
I0828 11:53:25.249524 19192 solver.cpp:218] Iteration 19940 (2.25401 iter/s, 8.87307s/20 iters), loss = 1.99201
I0828 11:53:25.249524 19192 solver.cpp:237] Train net output #0: bbox_accuracy = 0.75
I0828 11:53:25.250700 19192 solver.cpp:237] Train net output #1: loss_bbox = 0.644284 (* 1 = 0.644284 loss)
I0828 11:53:25.250700 19192 solver.cpp:237] Train net output #2: loss_cls = 0.905329 (* 1 = 0.905329 loss)
I0828 11:53:25.250700 19192 solver.cpp:237] Train net output #3: rpn_cls_loss = 0.0850422 (* 1 = 0.0850422 loss)
I0828 11:53:25.250700 19192 solver.cpp:237] Train net output #4: rpn_loss_bbox = 0.0457095 (* 1 = 0.0457095 loss)
I0828 11:53:25.251526 19192 sgd_solver.cpp:105] Iteration 19940, lr = 0.001
I0828 11:53:34.160838 19192 solver.cpp:218] Iteration 19960 (2.24479 iter/s, 8.90953s/20 iters), loss = 2.00955
I0828 11:53:34.160838 19192 solver.cpp:237] Train net output #0: bbox_accuracy = 0.75
I0828 11:53:34.161839 19192 solver.cpp:237] Train net output #1: loss_bbox = 0.721698 (* 1 = 0.721698 loss)
I0828 11:53:34.161839 19192 solver.cpp:237] Train net output #2: loss_cls = 1.35276 (* 1 = 1.35276 loss)
I0828 11:53:34.161839 19192 solver.cpp:237] Train net output #3: rpn_cls_loss = 0.107642 (* 1 = 0.107642 loss)
I0828 11:53:34.161839 19192 solver.cpp:237] Train net output #4: rpn_loss_bbox = 0.0488492 (* 1 = 0.0488492 loss)
I0828 11:53:34.161839 19192 sgd_solver.cpp:105] Iteration 19960, lr = 0.001
I0828 11:53:43.032603 19192 solver.cpp:218] Iteration 19980 (2.25453 iter/s, 8.87101s/20 iters), loss = 1.98453
I0828 11:53:43.032603 19192 solver.cpp:237] Train net output #0: bbox_accuracy = 0.96875
I0828 11:53:43.033603 19192 solver.cpp:237] Train net output #1: loss_bbox = 0.000514256 (* 1 = 0.000514256 loss)
I0828 11:53:43.033603 19192 solver.cpp:237] Train net output #2: loss_cls = 0.411482 (* 1 = 0.411482 loss)
I0828 11:53:43.033603 19192 solver.cpp:237] Train net output #3: rpn_cls_loss = 0.192493 (* 1 = 0.192493 loss)
I0828 11:53:43.033603 19192 solver.cpp:237] Train net output #4: rpn_loss_bbox = 0.0110174 (* 1 = 0.0110174 loss)
I0828 11:53:43.033603 19192 sgd_solver.cpp:105] Iteration 19980, lr = 0.001

请问如何修改使其能跑锕....救救我...

@makefile , Thank you for your codes. Can you offer converting code for converting yolov3 weight model to caffe format? What's more , yolov3 tiny model is more smaller, will you update it in the near future?

Hello,

I understand that this is a quite old repo, but I would like to know if the GPU implementation has some bug, since it is commented out.

Thanks

I find the reason why yolov3-tiny model failed with current code.

There are two questions need to be solved.

• change pooling reshape rule in pooling_layer.cpp

if (ceil_mode) {
    // pooled_height_ = static_cast<int>(ceil(static_cast<float>(
    //     height_ + 2 * pad_h_ - kernel_h_) / stride_h_)) + 1;
    // pooled_width_ = static_cast<int>(ceil(static_cast<float>(
    //     width_ + 2 * pad_w_ - kernel_w_) / stride_w_)) + 1;
      pooled_height_ = static_cast<int>((height_+2*pad_h_) / stride_h_);
      pooled_width_ = static_cast<int>((width_+2*pad_h_) / stride_w_);
  } else {

• shutdown net->num_outputs() check in demo_yolov3.cpp

//CHECK_EQ(net->num_outputs(), 3) << "Network should have exactly three outputs.";  

why i didn't make it a merge request

Darknet and caffe takes different measures to deal with reshaping in pooling_layer.

The changed code does not support caffe model... Unless they retrain the models with the new pooling_layer.

I thought there has a Elegant solution. Adding a new parameter in pooling_layer, or adding a new layer named pooling_yolo.

But this is beyond the scope of my work. Middle-aged people have no right to spend time to satisfy elegance.

Thanks for your project, it helps me!

@makefile why did you comment "transform_bbox" and "selected_flags" in frcnn_proposal_layer.cu? This way it doesn't work

Try to train on Resnet50 FasterRCNN on VOC, got nan loss in the begin of training process:

I0812 13:02:23.224562 15010 caffe.cpp:248] Starting Optimization
I0812 13:02:23.224596 15010 solver.cpp:292] Solving ResNet-50
I0812 13:02:23.224606 15010 solver.cpp:293] Learning Rate Policy: multistep
I0812 13:02:23.315627 15010 frcnn_anchor_target_layer.hpp:97] Info_Stds_Means_AvePos : COUNT : 23
I0812 13:02:23.315645 15010 frcnn_anchor_target_layer.hpp:98] STDS   : 0.0124143, 0.142521, 0.0744314, 0.40446
I0812 13:02:23.315654 15010 frcnn_anchor_target_layer.hpp:99] MEANS  : 0.00592739, 0.0176122, 0.0154362, 0.0769088
I0812 13:02:23.315659 15010 frcnn_anchor_target_layer.hpp:101] num_positive ave : 23
I0812 13:02:23.315661 15010 frcnn_anchor_target_layer.hpp:102] num_negitive ave : 233
I0812 13:02:23.694118 15010 solver.cpp:231] Iteration 0 (0.00570228 iter/s, 0.469403s/100 iters), loss = 3.69928  [ 0 / 70000 ] -> [ 0:0 (H:M) ]
I0812 13:02:23.694160 15010 solver.cpp:257]     Train net output #0: bbox_accuracy = 0.0857143
I0812 13:02:23.694181 15010 solver.cpp:257]     Train net output #1: loss_bbox = 0.66597 (* 1 = 0.66597 loss)
I0812 13:02:23.694186 15010 solver.cpp:257]     Train net output #2: loss_cls = 2.69545 (* 1 = 2.69545 loss)
I0812 13:02:23.694191 15010 solver.cpp:257]     Train net output #3: rpn_cls_loss = 0.662394 (* 1 = 0.662394 loss)
I0812 13:02:23.694195 15010 solver.cpp:257]     Train net output #4: rpn_loss_bbox = 0.256747 (* 1 = 0.256747 loss)
I0812 13:02:23.694200 15010 sgd_solver.cpp:148] Iteration 0, lr = 0.00033
I0812 13:02:59.574069 15010 solver.cpp:231] Iteration 100 (2.78717 iter/s, 35.8787s/100 iters), loss = 1.96366  [ 100 / 70000 ] -> [ 7:3 (H:M) ]
I0812 13:02:59.574113 15010 solver.cpp:257]     Train net output #0: bbox_accuracy = 0.857143
I0812 13:02:59.574122 15010 solver.cpp:257]     Train net output #1: loss_bbox = 10.4415 (* 1 = 10.4415 loss)
I0812 13:02:59.574128 15010 solver.cpp:257]     Train net output #2: loss_cls = 0.139943 (* 1 = 0.139943 loss)
I0812 13:02:59.574133 15010 solver.cpp:257]     Train net output #3: rpn_cls_loss = 0.341543 (* 1 = 0.341543 loss)
I0812 13:02:59.574139 15010 solver.cpp:257]     Train net output #4: rpn_loss_bbox = 0.219639 (* 1 = 0.219639 loss)
I0812 13:02:59.574143 15010 sgd_solver.cpp:148] Iteration 100, lr = 0.000464
I0812 13:03:37.712059 15010 solver.cpp:231] Iteration 200 (2.62215 iter/s, 38.1367s/100 iters), loss = 3.6148  [ 200 / 70000 ] -> [ 7:13 (H:M) ]
I0812 13:03:37.712097 15010 solver.cpp:257]     Train net output #0: bbox_accuracy = 0.857143
I0812 13:03:37.712105 15010 solver.cpp:257]     Train net output #1: loss_bbox = 2.76457 (* 1 = 2.76457 loss)
I0812 13:03:37.712110 15010 solver.cpp:257]     Train net output #2: loss_cls = 0.623248 (* 1 = 0.623248 loss)
I0812 13:03:37.712113 15010 solver.cpp:257]     Train net output #3: rpn_cls_loss = 0.37599 (* 1 = 0.37599 loss)
I0812 13:03:37.712117 15010 solver.cpp:257]     Train net output #4: rpn_loss_bbox = 0.135961 (* 1 = 0.135961 loss)
I0812 13:03:37.712121 15010 sgd_solver.cpp:148] Iteration 200, lr = 0.000598
I0812 13:04:17.866065 15010 solver.cpp:231] Iteration 300 (2.4905 iter/s, 40.1526s/100 iters), loss = 5.94563  [ 300 / 70000 ] -> [ 7:24 (H:M) ]
I0812 13:04:17.866216 15010 solver.cpp:257]     Train net output #0: bbox_accuracy = 0.771429
I0812 13:04:17.866282 15010 solver.cpp:257]     Train net output #1: loss_bbox = 1.84373 (* 1 = 1.84373 loss)
I0812 13:04:17.866341 15010 solver.cpp:257]     Train net output #2: loss_cls = 2.36289 (* 1 = 2.36289 loss)
I0812 13:04:17.866400 15010 solver.cpp:257]     Train net output #3: rpn_cls_loss = 0.15481 (* 1 = 0.15481 loss)
I0812 13:04:17.866457 15010 solver.cpp:257]     Train net output #4: rpn_loss_bbox = 0.0201009 (* 1 = 0.0201009 loss)
I0812 13:04:17.866518 15010 sgd_solver.cpp:148] Iteration 300, lr = 0.000732
I0812 13:04:54.086403 15010 solver.cpp:231] Iteration 400 (2.76101 iter/s, 36.2187s/100 iters), loss = 7.70644  [ 400 / 70000 ] -> [ 7:17 (H:M) ]
I0812 13:04:54.086589 15010 solver.cpp:257]     Train net output #0: bbox_accuracy = 0.971429
I0812 13:04:54.086655 15010 solver.cpp:257]     Train net output #1: loss_bbox = 0.276507 (* 1 = 0.276507 loss)
I0812 13:04:54.086716 15010 solver.cpp:257]     Train net output #2: loss_cls = 0.13878 (* 1 = 0.13878 loss)
I0812 13:04:54.086776 15010 solver.cpp:257]     Train net output #3: rpn_cls_loss = 0.22572 (* 1 = 0.22572 loss)
I0812 13:04:54.086835 15010 solver.cpp:257]     Train net output #4: rpn_loss_bbox = 0.0198562 (* 1 = 0.0198562 loss)
I0812 13:04:54.086892 15010 sgd_solver.cpp:148] Iteration 400, lr = 0.000866
I0812 13:05:34.160019 15010 frcnn_anchor_target_layer.hpp:97] Info_Stds_Means_AvePos : COUNT : 40300
I0812 13:05:34.160053 15010 frcnn_anchor_target_layer.hpp:98] STDS   : 0.189493, 0.140276, 0.428416, 0.523535
I0812 13:05:34.160060 15010 frcnn_anchor_target_layer.hpp:99] MEANS  : -0.00275885, 0.0171251, -0.0280274, -0.119019
I0812 13:05:34.160064 15010 frcnn_anchor_target_layer.hpp:101] num_positive ave : 40.2597
I0812 13:05:34.160068 15010 frcnn_anchor_target_layer.hpp:102] num_negitive ave : 215.74
I0812 13:05:34.429374 15010 solver.cpp:231] Iteration 500 (2.47885 iter/s, 40.3413s/100 iters), loss = nan  [ 500 / 70000 ] -> [ 7:23 (H:M) ]
E0812 13:05:34.429399 15010 solver.cpp:236] ======= exit cause of nan loss =======
*** Error in `build/tools/caffe': double free or corruption (out): 0x0000000001c7a240 ***