wentong-dst / up-down-captioner Goto Github PK

Hi, thanks for sharing the code.

I followed the instructions using the caffe you provided and prepared the image features from bottom-up method. However, when I ran the train.sh, the errors happened.

[I0520 05:48:22.639592 1013 net.cpp:255] Network initialization done.
[F0520 05:48:22.641618 1013 net.cpp:793] check failed: data_hid >= 0 (-1 vs. 0) Error reading weights from snapshots/caption_lstm/lstm_scst_iter_1000.caffemodel.h5
*** Check failure stack trace: ***
@ 0x7f0d56aa446d google::LogMessage::Fail()
@ 0x7f0d56aa6a23 google::LogMessage::SendToLog()
@ 0x7f0d56aa3ffb google::LogMessage::Flush()
@ 0x7f0d56aa596e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f0d570223e5 caffe::Net<>::CopyTrainedLayersFromHDF5()
@ 0x7f0d57026704 caffe::Net<>::CopyTrainedLayersFrom()
@ 0x7f0d5895d32c caffe::Net_Init_Load()
@ 0x7f0d58979b43 boost::python::objects::signature_py_function_impl<>::operator()()
@ 0x7f0d562ac5cd boost::python::objects::function::call()
@ 0x7f0d562ac7c8 (unknown)
@ 0x7f0d562b4613 boost::python::handle_exception_impl()
@ 0x7f0d562a9999 (unknown)
@ 0x7f0d670da973 PyObject_Call
@ 0x7f0d670e980d instancemethod_call
@ 0x7f0d670da973 PyObject_Call
@ 0x7f0d67133764 slot_tp_init
@ 0x7f0d6713001b type_call
@ 0x7f0d670da973 PyObject_Call
@ 0x7f0d67170d49 PyEval_EvalFrameEx
@ 0x7f0d6717517e PyEval_EvalFrameEx
@ 0x7f0d671766c9 PyEval_EvalCodeEx
@ 0x7f0d671768ea PyEval_EvalCode
@ 0x7f0d6718fbad run_mod
@ 0x7f0d67190d28 PyRun_FileExFlags
@ 0x7f0d67191f48 PyRun_SimpleFileExFlags
@ 0x7f0d671a416c Py_Main
@ 0x7f0d66374830 libc_start_main
@ 0x55935a2ff07f (unknown)
./experiments/caption_lstm/train.sh: line 27: 1013 Aborted (core dumped) python ./scripts/beam_decode.py --gpu ${GPU_ID:0:1} --model ${BASE_DIR}${NET_NAME}/decoder.prototxt --weights=${SNAPSHOT_DIR}${NET_NAME}/lstm_scst_iter${SCST_MAX_IT}.caffemodel.h5 --vocab ${DATA_DIR}${VOCAB_FILE} --outfile ${OUT_DIR}/${NET_NAME}/scst_iter${SCST_MAX_IT}.json

And also the error happens when reading the weights from lstm_scst_iter_60000.caffemodel.h5. Could you give me some hints about where the bug might be?

Hello,
Thank you for your sharing the code of up-down-captioner.
Actually, I have some question..

During the make -j8 & make pycaffee following your instruction,
some errors occurred in src/caffe/layers/beam_search_node_layer.cpp.
The detailed errors are below.

And I tried make -j8 & make pycaffee step by removing beam_search_node_layer.cpp and beam_search_node_layer.cu files. The step was passed, but I cannot use the Demo.ipynb in the line of 'Load captioning model'.
Maybe it was caused from the beam_search_node_layer.cpp.

So.. have you any comment or idea for solving this problem?
My environment is as follows:

• Ubuntu 16.04
• Python: 2.7.13 (Anaconda)
• gcc version: 5.4.0
• installed needed libs

-- Detailed errors during make beam_search_node_layer.cpp.
....
CXX src/caffe/layers/elu_layer.cpp
CXX src/caffe/layers/lstm_unit_layer.cpp
** occuring error**
src/caffe/layers/beam_search_node_layer.cpp: In instantiation of \u2018void caffe::BeamSearchNodeLayer::Forward_cpu(const std::vector<caffe::Blob>&, const std::vector<caffe::Blob>&) [with Dtype = float]\u2019:
src/caffe/layers/beam_search_node_layer.cpp:208:1: required from here
src/caffe/layers/beam_search_node_layer.cpp:97:51: error: no matching function for call to \u2018make_pair(const float&, float)\u2019
cpu_beams_.push_back(make_pair<Dtype, int>(partial_sum[src], ix + end_of_sequence_));
^
In file included from /usr/include/c++/5/bits/stl_algobase.h:64:0,
from /usr/include/c++/5/vector:60,
from src/caffe/layers/beam_search_node_layer.cpp:1:
/usr/include/c++/5/bits/stl_pair.h:276:5: note: candidate: template<class _T1, class _T2> constexpr std::pair<typename std::__decay_and_strip<_Tp>::__type, typename std::__decay_and_strip<_T2>::__type> std::make_pair(_T1&&, _T2&&)
make_pair(_T1&& __x, T2&& y)
^
/usr/include/c++/5/bits/stl_pair.h:276:5: note: template argument deduction/substitution failed:
src/caffe/layers/beam_search_node_layer.cpp:97:63: note: cannot convert \u2018*(partial_sum + ((sizetype)(((long unsigned int)src) * 4ul)))\u2019 (type \u2018const float\u2019) to type \u2018float&&\u2019
cpu_beams.push_back(make_pair<Dtype, int>(partial_sum[src], ix + end_of_sequence));
^
src/caffe/layers/beam_search_node_layer.cpp:108:54: error: no matching function for call to \u2018make_pair(const float&, int)\u2019
cpu_beams.push_back(make_pair<Dtype,int>(score[ix+w],ix+w));
^
In file included from /usr/include/c++/5/bits/stl_algobase.h:64:0,
from /usr/include/c++/5/vector:60,
from src/caffe/layers/beam_search_node_layer.cpp:1:
/usr/include/c++/5/bits/stl_pair.h:276:5: note: candidate: template<class _T1, class _T2> constexpr std::pair<typename std::__decay_and_strip<_Tp>::__type, typename std::__decay_and_strip<_T2>::__type> std::make_pair(_T1&&, _T2&&)
make_pair(_T1&& __x, T2&& y)
^
/usr/include/c++/5/bits/stl_pair.h:276:5: note: template argument deduction/substitution failed:
src/caffe/layers/beam_search_node_layer.cpp:108:60: note: cannot convert \u2018*(score + ((sizetype)(((long unsigned int)(ix + w)) * 4ul)))\u2019 (type \u2018const float\u2019) to type \u2018float&&\u2019
cpu_beams.push_back(make_pair<Dtype,int>(score[ix+w],ix+w));
^
src/caffe/layers/beam_search_node_layer.cpp: In instantiation of \u2018void caffe::BeamSearchNodeLayer::Forward_cpu(const std::vector<caffe::Blob>&, const std::vector<caffe::Blob>&) [with Dtype = double]\u2019:
src/caffe/layers/beam_search_node_layer.cpp:208:1: required from here
src/caffe/layers/beam_search_node_layer.cpp:97:51: error: no matching function for call to \u2018make_pair(const double&, double)\u2019
cpu_beams.push_back(make_pair<Dtype, int>(partial_sum[src], ix + end_of_sequence));
^
In file included from /usr/include/c++/5/bits/stl_algobase.h:64:0,
from /usr/include/c++/5/vector:60,
from src/caffe/layers/beam_search_node_layer.cpp:1:
/usr/include/c++/5/bits/stl_pair.h:276:5: note: candidate: template<class _T1, class _T2> constexpr std::pair<typename std::__decay_and_strip<_Tp>::__type, typename std::__decay_and_strip<_T2>::__type> std::make_pair(_T1&&, _T2&&)
make_pair(_T1&& __x, T2&& y)
^
/usr/include/c++/5/bits/stl_pair.h:276:5: note: template argument deduction/substitution failed:
src/caffe/layers/beam_search_node_layer.cpp:97:63: note: cannot convert \u2018*(partial_sum + ((sizetype)(((long unsigned int)src) * 8ul)))\u2019 (type \u2018const double\u2019) to type \u2018double&&\u2019
cpu_beams.push_back(make_pair<Dtype, int>(partial_sum[src], ix + end_of_sequence));
^
src/caffe/layers/beam_search_node_layer.cpp:108:54: error: no matching function for call to \u2018make_pair(const double&, int)\u2019
cpu_beams.push_back(make_pair<Dtype,int>(score[ix+w],ix+w));
^
In file included from /usr/include/c++/5/bits/stl_algobase.h:64:0,
from /usr/include/c++/5/vector:60,
from src/caffe/layers/beam_search_node_layer.cpp:1:
/usr/include/c++/5/bits/stl_pair.h:276:5: note: candidate: template<class _T1, class _T2> constexpr std::pair<typename std::__decay_and_strip<_Tp>::__type, typename std::__decay_and_strip<_T2>::__type> std::make_pair(_T1&&, _T2&&)
make_pair(_T1&& __x, _T2&& _y)
^
/usr/include/c++/5/bits/stl_pair.h:276:5: note: template argument deduction/substitution failed:
src/caffe/layers/beam_search_node_layer.cpp:108:60: note: cannot convert \u2018*(score + ((sizetype)(((long unsigned int)(ix + w)) * 8ul)))\u2019 (type \u2018const double\u2019) to type \u2018double&&\u2019
cpu_beams.push_back(make_pair<Dtype,int>(score[ix+w],ix+w));
^
Makefile:584: recipe for target '.build_release/src/caffe/layers/beam_search_node_layer.o' failed
make: *** [.build_release/src/caffe/layers/beam_search_node_layer.o] Error 1
make: *** Waiting for unfinished jobs....