Giter VIP home page Giter VIP logo

hake-action's Introduction

HAKE-Action

HAKE-Action (TensorFlow) is a project to open the SOTA action understanding studies based on our Human Activity Knowledge Engine. It includes reproduced SOTA models and their HAKE-enhanced versions. HAKE-Action is authored by Yong-Lu Li, Xinpeng Liu, Liang Xu, Cewu Lu. Currently, it is manintained by Yong-Lu Li, Xinpeng Liu and Liang Xu.

News: (2022.12.19) HAKE 2.0 is accepted by TPAMI!

(2022.11.19) We release the interactive object bounding boxes & classes in the interactions within AVA dataset (2.1 & 2.2)! HAKE-AVA, [Paper]. BTW, we also release a CLIP-based human body part states recognizer in CLIP-Activity2Vec!

(2022.07.29) Our new work PartMap (ECCV'22) is released! Paper, Code

(2022.02.14) We release the human body part state labels based on AVA: HAKE-AVA and HAKE 2.0 paper.

(2021.10.06) Our extended version of SymNet is accepted by TPAMI! Paper and code are coming soon.

(2021.2.7) Upgraded HAKE-Activity2Vec is released! Images/Videos --> human box + ID + skeleton + part states + action + representation. [Description]

(2021.1.15) Our extended version of TIN (Transferable Interactiveness Network) is accepted by TPAMI! New paper and code will be released soon.

(2020.10.27) The code of IDN (Paper) in NeurIPS'20 is released!

(2020.6.16) Our larger version HAKE-Large (>120K images, activity and part state labels) is released!

We released the HAKE-HICO (image-level part state labels upon HICO) and HAKE-HICO-DET (instance-level part state labels upon HICO-DET). The corresponding data can be found here: HAKE Data.

  • Paper is here.
  • More data and part states (e.g., upon AVA, more kinds of action categories, more rare actions...) are coming.
  • We will keep updating HAKE-Action to include more SOTA models and their HAKE-enhanced versions.
  • HAKE-HICO (PaStaNet* mode in paper): image-level, add the aggression of all part states in an image (belong to one or multiple active persons), compared with original HICO, the only additional labels are image-level human body part states.

  • HAKE-HICO-DET (PaStaNet* in paper): instance-level, add part states for each annotated persons of all images in HICO-DET, the only additional labels are instance-level human body part states.

  • HAKE-Large (PaStaNet in paper): contains more than 120K images, action labels and the corresponding part state labels. The images come from the existing action datasets and crowdsourcing. We mannully annotated all the active persons with our novel part-level semantics.

  • GT-HAKE (GT-PaStaNet* in paper): GT-HAKE-HICO and G-HAKE-HICO-DET. It means that we use the part state labels as the part state prediction. That is, we can perfectly estimate the body part states of a person. Then we use them to infer the instance activities. This mode can be seen as the upper bound of our HAKE-Action. From the results below we can find that, the upper bound is far beyond the SOTA performance. Thus, except for the current study on the conventional instance-level method, continue promoting part-level method based on HAKE would be a very promising direction.

Notion

Activity2Vec and PaSta-R are our part state based modules, which operate action inference based on part semantics, different from previous instance semantics. For example, Pairwise + HAKE-HICO pre-trained Activity2Vec + Linear PaSta-R (the seventh row) achieves 45.9 mAP on HICO. More details can be found in our CVPR2020 paper: PaStaNet: Toward Human Activity Knowledge Engine.

Code

The two versions of HAKE-Action are relesased in two branches of this repo:

Instance-level +Activity2Vec +PaSta-R mAP Few@1 Few@5 Few@10
R*CNN - - 28.5 - - -
Girdhar et.al. - - 34.6 - - -
Mallya et.al. - - 36.1 - - -
Pairwise - - 39.9 13.0 19.8 22.3
- HAKE-HICO Linear 44.5 26.9 30.0 30.7
Mallya et.al. HAKE-HICO Linear 45.0 26.5 29.1 30.3
Pairwise HAKE-HICO Linear 45.9 26.2 30.6 31.8
Pairwise HAKE-HICO MLP 45.6 26.0 30.8 31.9
Pairwise HAKE-HICO GCN 45.6 25.2 30.0 31.4
Pairwise HAKE-HICO Seq 45.9 25.3 30.2 31.6
Pairwise HAKE-HICO Tree 45.8 24.9 30.3 31.8
Pairwise HAKE-Large Linear 46.3 24.7 31.8 33.1
Pairwise HAKE-Large Linear 46.3 24.7 31.8 33.1
Pairwise GT-HAKE-HICO Linear 65.6 47.5 55.4 56.6

Using Object Detections from iCAN

Instance-level +Activity2Vec +PaSta-R Full(def) Rare(def) None-Rare(def) Full(ko) Rare(ko) None-Rare(ko)
iCAN - - 14.84 10.45 16.15 16.26 11.33 17.73
TIN - - 17.03 13.42 18.11 19.17 15.51 20.26
iCAN HAKE-HICO-DET Linear 19.61 17.29 20.30 22.10 20.46 22.59
TIN HAKE-HICO-DET Linear 22.12 20.19 22.69 24.06 22.19 24.62
TIN HAKE-Large Linear 22.65 21.17 23.09 24.53 23.00 24.99
TIN GT-HAKE-HICO-DET Linear 34.86 42.83 32.48 35.59 42.94 33.40

Models on AVA (Frame-based)

Method +Activity2Vec +PaSta-R mAP
AVA-TF-Baseline - - 11.4
LFB-Res-50-baseline - - 22.2
LFB-Res-101-baseline - - 23.3
AVA-TF-Baeline HAKE-Large Linear 15.6
LFB-Res-50-baseline HAKE-Large Linear 23.4
LFB-Res-101-baseline HAKE-Large Linear 24.3

Models on V-COCO

Method +Activity2Vec +PaSta-R AP(role), Scenario 1 AP(role), Scenario 2
iCAN - - 45.3 52.4
TIN - - 47.8 54.2
iCAN HAKE-Large Linear 49.2 55.6
TIN HAKE-Large Linear 51.0 57.5

Training Details

We first pre-train the Activity2Vec and PaSta-R with activities and PaSta labels. Then we change the last FC in PaSta-R to fit the activity categories of the target dataset. Finally, we freeze Activity2Vec and fine-tune PaSta-R on the train set of the target dataset. Here, HAKE works like the ImageNet and Activity2Vec is used as a pre-trained knowledge engine to promote other tasks.

Citation

If you find our work useful, please consider citing:

@article{li2023hake,
     title={HAKE: A Knowledge Engine Foundation for Human Activity Understanding},
    author={Li, Yong-Lu and Liu, Xinpeng and Wu, Xiaoqian and Li, Yizhuo and Qiu, Zuoyu and Xu, Liang and Xu, Yue and Fang, Hao-Shu and Lu, Cewu},
    journal={TPAMI},
    year={2023}
}
@inproceedings{li2020pastanet,
  title={PaStaNet: Toward Human Activity Knowledge Engine},
  author={Li, Yong-Lu and Xu, Liang and Liu, Xinpeng and Huang, Xijie and Xu, Yue and Wang, Shiyi and Fang, Hao-Shu and Ma, Ze and Chen, Mingyang and Lu, Cewu},
  booktitle={CVPR},
  year={2020}
}
@inproceedings{li2019transferable,
  title={Transferable Interactiveness Knowledge for Human-Object Interaction Detection},
  author={Li, Yong-Lu and Zhou, Siyuan and Huang, Xijie and Xu, Liang and Ma, Ze and Fang, Hao-Shu and Wang, Yanfeng and Lu, Cewu},
  booktitle={CVPR},
  year={2019}
}
@inproceedings{lu2018beyond,
  title={Beyond holistic object recognition: Enriching image understanding with part states},
  author={Lu, Cewu and Su, Hao and Li, Yonglu and Lu, Yongyi and Yi, Li and Tang, Chi-Keung and Guibas, Leonidas J},
  booktitle={CVPR},
  year={2018}
}

HAKE[website] is a new large-scale knowledge base and engine for human activity understanding. HAKE provides elaborate and abundant body part state labels for active human instances in a large scale of images and videos. With HAKE, we boost the action understanding performance on widely-used human activity benchmarks. Now we are still enlarging and enriching it, and looking forward to working with outstanding researchers around the world on its applications and further improvements. If you have any pieces of advice or interests, please feel free to contact Yong-Lu Li ([email protected]).

If you get any problems or if you find any bugs, don't hesitate to comment on GitHub or make a pull request!

HAKE-Action is freely available for free non-commercial use, and may be redistributed under these conditions. For commercial queries, please drop an e-mail. We will send the detail agreement to you.

hake-action's People

Contributors

dirtyharrylyl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hake-action's Issues

Format for Test_all_part.pkl?

Hello again. Is there a specification for the format of the input detection file (for example, Test_all_part.pkl)? I looked through it but do not know the significance of some of the numbers. For context, I'm trying to test this method on my own images and have run iCAN and TIN (with satisfying results), but hope to move on to PaSta detection now.

question about processing methods of ava dataset

Thanks your great work, could you please release the pre-processing tools/methods of ava dataset, like how to get the human, object, part box and LFB baseline models. The total process is helpful in understanding your work. Thanks again.

Generating body part boxes from pose joints

First of all, thanks for open sourcing this fantastic work.

I was reading the paper PaStaNet: Toward Human Activity Knowledge Engine and I found that you generate body part boxes by first detecting joints using pose methods, and then use some heuristics/ rules to generate bounding boxes.
I am curious to learn more about the exact rules/ heuristics that you have used.
Would it be possible to find the code used to generate such bounding boxes from joint locations?

Loading checkpoint problem

Hello. I am getting a not found error: "NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: Key pasta_classification/cls_pasta_0/fully_connected/biases not found in checkpoint."

I am using the Test_HICO_DET file and have tried to use the pasta_HICO_DET weights and the pasta_full weights, but both throw the above error. I am wondering if y'all have updated Weights which I can access or if y'all have any ideas about debugging this problem. Thanks!

生成的part box和标注的不一样

推算的 算的是头部和肩部,标注是颈部和肘部, 而且推算的还有些错误的地方, 所以到底最后用的哪部分。 推算的代码用的是part_box_processing。

"Error parsing text-format caffe.NetParameter" in Image-level-HAKE-Action

When I execute python scripts/train.py --pasta-mode linear in docker container, language_model.log is generated. And I checked this file,there are some problems:

I0916 11:37:38.907941   553 caffe.cpp:217] Using GPUs 0
I0916 11:37:38.949647   553 caffe.cpp:222] GPU 0: GeForce GTX 1080 Ti
I0916 11:37:39.380765   553 solver.cpp:51] Initializing solver from parameters: 
test_iter: 2000
test_interval: 1000000
base_lr: 1e-05
display: 20
max_iter: 600000
lr_policy: "cosine"
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
stepsize: 30000
snapshot: 10000
snapshot_prefix: "snaps/language-model"
device_id: 0
net: "models/Language-Models/train_language-model.prototxt"
train_state {
  level: 0
  stage: ""
}
test_initialization: false
average_loss: 100
iter_size: 16
snapshot_format: HDF5
train_size: 38116
te: 2
multfactor: 2
tt: 2.05884722928
tenext: 2
I0916 11:37:39.381000   553 solver.cpp:94] Creating training net from net file: models/Language-Models/train_language-model.prototxt
[libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 3124:7: Message type "caffe.LayerParameter" has no field named "layer".
F0916 11:37:39.382678   553 upgrade_proto.cpp:79] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: models/Language-Models/train_language-model.prototxt
*** Check failure stack trace: ***
    @     0x7feae92775cd  google::LogMessage::Fail()
    @     0x7feae9279433  google::LogMessage::SendToLog()
    @     0x7feae927715b  google::LogMessage::Flush()
    @     0x7feae9279e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7feae9b77ac1  caffe::ReadNetParamsFromTextFileOrDie()
    @     0x7feae996e6eb  caffe::Solver<>::InitTrainNet()
    @     0x7feae996f9c7  caffe::Solver<>::Init()
    @     0x7feae996fd6a  caffe::Solver<>::Solver()
    @     0x7feae995a6c3  caffe::Creator_SGDSolver<>()
    @           0x40afb9  train()
    @           0x4077c8  main
    @     0x7feae7a0d830  __libc_start_main
    @           0x408099  _start
    @              (nil)  (unknown)

I googled this problem, it seems to be the reason for the different version of caffe. So I want to ask what is the version of caffe for this project.
My caffe version is:

>>> caffe.__version__
'1.0.0-rc3'

'make all -j8' error

According to the INSTALL.md, when i tap the command make all -j8 , i found the following error

CXX/LD -o .build_release/tools/create_label_map.bin
CXX/LD -o .build_release/tools/convert_imageset.bin
CXX/LD -o .build_release/tools/get_image_size.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_text.bin
CXX/LD -o .build_release/tools/caffe.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin
CXX/LD -o .build_release/tools/extract_features.bin
CXX/LD -o .build_release/tools/convert_annoset.bin
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/create_label_map.bin' failed
make: *** [.build_release/tools/create_label_map.bin] Error 1
make: *** Waiting for unfinished jobs....
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/get_image_size.bin' failed
make: *** [.build_release/tools/get_image_size.bin] Error 1
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/upgrade_net_proto_binary.bin' failed
make: *** [.build_release/tools/upgrade_net_proto_binary.bin] Error 1
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/upgrade_net_proto_text.bin' failed
make: *** [.build_release/tools/upgrade_net_proto_text.bin] Error 1
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/convert_annoset.bin' failed
make: *** [.build_release/tools/convert_annoset.bin] Error 1
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/convert_imageset.bin' failed
make: *** [.build_release/tools/convert_imageset.bin] Error 1
.build_release/tools/extract_features.o: In function `int feature_extraction_pipeline<float>(int, char**)':
extract_features.cpp:(.text._Z27feature_extraction_pipelineIfEiiPPc[_Z27feature_extraction_pipelineIfEiiPPc]+0xd16): undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/extract_features.bin' failed
make: *** [.build_release/tools/extract_features.bin] Error 1
.build_release/tools/caffe.o: In function `train()':
caffe.cpp:(.text+0x2ada): undefined reference to `google::protobuf::internal::StringTypeHandlerBase::New[abi:cxx11]()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::set(int, double)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFields(google::protobuf::UnknownFieldSet const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadPackedEnumNoInline(google::protobuf::io::CodedInputStream*, bool (*)(int), google::protobuf::RepeatedField<int>*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::allocate(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize32Fallback(unsigned int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getTextSize(cv::String const&, int, double, int, int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::ReadString(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(cv::String const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::release()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::operator>>(cv::Mat&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint64Fallback(unsigned long*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadTagFallback()'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::isOpened() const'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::get(int) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_[abi:cxx11]'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SpaceUsed() const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedInputStream::ReadVarint32Fallback(unsigned int*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray(google::protobuf::UnknownFieldSet const&, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::getRotationMatrix2D(cv::Point_<float>, double, double)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::open(int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::StringTypeHandlerBase::Delete(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::GeneratedMessageReflection::GeneratedMessageReflection(google::protobuf::Descriptor const*, google::protobuf::Message const*, int const*, int, int, int, google::protobuf::DescriptorPool const*, google::protobuf::MessageFactory*, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::String::deallocate()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::InitEmptyString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))'
.build_release/lib/libcaffe.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, cv::String const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::~VideoCapture()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::empty_string_once_init_'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::UnknownFieldSet()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::~Message()'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::RepeatedPtrFieldBase::Swap(google::protobuf::internal::RepeatedPtrFieldBase*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::namedWindow(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::Message::SerializeToOstream(std::ostream*) const'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint32FallbackToArray(unsigned int, unsigned char*)'
.build_release/lib/libcaffe.so: undefined reference to `cv::VideoCapture::VideoCapture()'
collect2: error: ld returned 1 exit status
Makefile:620: recipe for target '.build_release/tools/caffe.bin' failed
make: *** [.build_release/tools/caffe.bin] Error 1

TIN scores format?

I have been able to run PaStaNet to generate bboxes, hdet, keys, odet, scores_A, scores_L, and scores_P on custom images, but in the Generate_detection.py file there are a bunch of TIN scores used to generate the detections:

score_H = pickle.load(open('TIN/score_H.pkl', 'rb'))
score_O = pickle.load(open('TIN/score_O.pkl', 'rb'))
score_sp = pickle.load(open('TIN/score_sp.pkl', 'rb'))
hdet = pickle.load(open('TIN/hdet.pkl', 'rb'))
odet = pickle.load(open('TIN/odet.pkl', 'rb'))
keys = pickle.load(open('TIN/keys.pkl', 'rb'))
pos = pickle.load(open('TIN/pos.pkl', 'rb'))
neg = pickle.load(open('TIN/neg.pkl', 'rb'))
bboxes = pickle.load(open('TIN/bboxes.pkl', 'rb'))

There is also a score_I pulled from the hico_caffe600.h5 file, a key_mapping.pkl, and the generation args file.

I am hoping y'all can release a description of how to generate these files / the formatting, or release the code used to generate these from TIN results / any other necessary sources. I already have TIN running but am unsure of how to format the results for Generate_detection.py. Thank you!

并没有用到language part?

image-level的推理部分并没有用到language part? 因为直接取的mil_cls_score_pvp_all而不是language_600?

where is key_mapping?

Hi, where is key_mapping.pkl in Generate_detection.py? Besides, would u mind provide all 'TIN/xxx.pkl' example files?

Demo code?

I am wondering if y'all are making a demo for instance-level HAKE-Action which includes Pose detection and the TIN network so that it is easy to test on custom images and other datasets?

iamge-level中的evaluate用matlab部分中用的测试文件是哪一个

iamge-level中的evaluate用matlab部分中用的测试文件是哪一个。我测试得到的最后结果是linear-final.csv或者fused.csv,但是我看了matlab中的代码的标注label格式是1,0,-1,(分别代表什么意思这里我也不懂),但是linear-final.csv或者fused.csv的格式是一个600维的数组,看起来是激活值,并不是1,0,-1这样的格式,那么最后的结果我怎么得到1,0,-1呢

why only ii == 3 in fused.py?

fused.py:
for ii in range(1, len(sys.argv)):
print sys.argv[ii]
a1 = np.loadtxt(sys.argv[ii], delimiter = ',', dtype = str,usecols=range(600))
arr1 = a1.astype(np.float)
for i in range(arr1.shape[0]):
for j in range(arr1.shape[1]):
if ii == 3:
global_array[i][j] += arr1[i][j] * 2

why only ii == 3 here? it seems it only uses pairwise.csv result.

Wired results: obtain the similar reported result without training...

Thanks for your released code. However, I faces a strage problem that the HAKE-Action seems do not need to train. I run the code using the command:

python tools/Train_pasta_HICO_DET.py --data 0 --init_weight 1 --train_module 2 --num_iteration 11 --model test_pastanet

Then, I test the model 'test_pastanet/HOI_iter_10.ckpt'.

Unbelievably! The result is:

Default:  0.2194696378436301                                                                                                                                                                                       
Default rare:  0.20412592656443512                                                                                                                                                                                 
Default non-rare:  0.22405282432962342                                                                                                           
Known object:  0.2382635240705128                                                                                                               
Known object, rare:  0.22197228395195176                                                                                                                                                                           
Known object, non-rare:  0.24312973865138168

Have you ever tested your code like this? This result is ....... I'm trying to test snapshot 1. I do not change the code.

为什么先运行PaSta_linear再运行 10v_attention

我的理解是10v_attention是用来计算Part Relevance Predictor和Visual PaSta feature的,而PaSta_linear是用来算PaSta-R,为什么顺序是反过来的。
然后最后的结果为什么是10v_attention、PaSta_linear和language model三个结果的fusion。论文的意思是Visual PaSta feature和Language PaSta进行fusion然后再去做PaSta-R。

怎么得到human part box

作者你好,我看了一下Image-level-action的代码,目前训练还是测试都是直接用生成好的human box, object box还有human part box,我想问一下有没有代码去生成这样的box呢就是从原图

ImportError: No module named select_layer

When I ran python scripts/test.py --pasta-mode linear, it showed

I0527 10:09:12.358407 344 layer_factory.hpp:77] Creating layer select_pairs
ImportError: No module named select_layer
Traceback (most recent call last):
File "scripts/test.py", line 41, in
test_one_caffemodel('models/Language-Models/deploy_language-model.prototxt', 'snaps/language_model.caffemodel.h5', 'results/language-model/', 'pvp', 'language-model')
File "/opt/HAKE-Action/scripts/test_before_mil.py", line 28, in test_one_caffemodel
caffe.TEST) # use test mode (e.g., don't perform dropoutet_mode_cpu()
SystemError: NULL result without error in PyObject_Call.

I found deploy_language-model.prototxt called the module named select_layer.
Would you please check it? Thank you.

No such file or directory: './info/hoi.txt'

In Image-level HAKE-Action branch, when I run 'python scripts/train.py --pasta-mode linear' in docker. The Error is No such file or directory: './info/hoi.txt'. But I don't know where to find the hoi.txt file.
Thank you.

Traceback (most recent call last):
File "/opt/HAKE-Action/python/HOI_data_layer-PaSta-Linear.py", line 99, in forward
im, human_roi, object_roi, rel_roi, scene_roi, part_roi, label,label_wts,hp_list, hp_list_wts, pvp_ankle2, pvp_ankle2_wts, pvp_knee2, pvp_knee2_wts, pvp_hip, pvp_hip_wts, pvp_hand2, pvp_hand2_wts, pvp_shoulder2, pvp_shoulder2_wts, pvp_head, pvp_head_wts, pvp_wts, verb_list, verb_list_wts, object_list, object_list_wts = self.batch_loader.load_next_image()
File "/opt/HAKE-Action/python/HOI_data_layer-PaSta-Linear.py", line 236, in load_next_image
anns = load_annotation(index, self.pascal_root, self.im_shape)
File "/opt/HAKE-Action/python/HOI_data_layer-PaSta-Linear.py", line 407, in load_annotation
verb_no = get_verb_no(i+1)
File "/opt/HAKE-Action/python/HOI_data_layer-PaSta-Linear.py", line 561, in get_verb_no
with open('./info/hoi.txt', 'r') as f:
IOError: [Errno 2] No such file or directory: './info/hoi.txt'

pretrained_models

!python tools/Train_pasta_HICO_DET.py --data <0 for PaStaNet*, 1 for PaStaNet> --init_weight 1 --train_module 2 --num_iteration 2000000 --model

Where can i get pre_trained models ckpt.?

can you help me to get pretrained models?

ValueError: None values not supported.

I use pickle 3 instead of cPickle.
So i change from
Trainval_GT = pickle.load(open(cfg.DATA_DIR + '/' + 'Trainval_GT_all_part.pkl', "rb"))
to
Trainval_GT = pickle.load(open(cfg.DATA_DIR + '/' + 'Trainval_GT_all_part.pkl', "rb"), encoding='bytes')

Then i got error:
ValueError: None values not supported.

This is full error:
`
data path:
/home/ubuntu/project/Action_Recognition/HAKE-Action/Data


WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

Namespace(Neg_select=60, Pos_augment=15, Restore_flag=5, init_weight=1, iter=1800000, max_iters=2000000, model='test_model', module_update=1, train_continue=2, train_module=2, **{'--data': 0})
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:59: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py:272: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py:275: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-05-07 11:34:18.152190: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-07 11:34:18.592120: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2020-05-07 11:34:18.598650: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55754b0e3470 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-07 11:34:18.598733: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-07 11:34:18.818668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-07 11:34:19.178313: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-07 11:34:19.178573: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55754b0e0ba0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-07 11:34:19.178588: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2020-05-07 11:34:19.235338: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-07 11:34:19.235578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:02:00.0
2020-05-07 11:34:19.263674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-05-07 11:34:19.905750: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-05-07 11:34:20.126609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-05-07 11:34:20.214744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-05-07 11:34:21.141719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-05-07 11:34:21.519181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-05-07 11:34:22.472666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-07 11:34:22.472987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-07 11:34:22.473722: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-07 11:34:22.474266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-05-07 11:34:22.485815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-05-07 11:34:22.495643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-07 11:34:22.495663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-05-07 11:34:22.495671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-05-07 11:34:22.507402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-07 11:34:22.606499: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-07 11:34:22.654192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 455 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5)
Solving..., Pos augment = 15, Neg augment = 60, Restore_flag = 5
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py:55: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:116: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:179: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:179: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:389: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/ops/nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:399: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:402: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/networks/pasta_HICO_DET.py:403: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py:71: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/weights:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/weights:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/weights:0
resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/weights:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/weights:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/weights:0
resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/weights:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/weights:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/weights:0
resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/weights:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/weights:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/weights:0
resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/weights:0
resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/weights:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/weights:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/weights:0
resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/weights:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/weights:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/weights:0
resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/weights:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/weights:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/weights:0
resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/weights:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/weights:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/weights:0
resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/weights:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/weights:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/weights:0
resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/weights:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/weights:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/weights:0
resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/weights:0
resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/weights:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/weights:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/weights:0
resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/weights:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/weights:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/weights:0
resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/weights:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/weights:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/weights:0
resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/weights:0
resnet_v1_50/block5/unit_1/bottleneck_v1/shortcut/weights:0
resnet_v1_50/block5/unit_1/bottleneck_v1/conv1/weights:0
resnet_v1_50/block5/unit_1/bottleneck_v1/conv2/weights:0
resnet_v1_50/block5/unit_1/bottleneck_v1/conv3/weights:0
resnet_v1_50/block5/unit_2/bottleneck_v1/conv1/weights:0
resnet_v1_50/block5/unit_2/bottleneck_v1/conv2/weights:0
resnet_v1_50/block5/unit_2/bottleneck_v1/conv3/weights:0
resnet_v1_50/block5/unit_3/bottleneck_v1/conv1/weights:0
resnet_v1_50/block5/unit_3/bottleneck_v1/conv2/weights:0
resnet_v1_50/block5/unit_3/bottleneck_v1/conv3/weights:0
pasta_classification/cls_pasta_0/fully_connected/weights:0
pasta_classification/cls_pasta_0/fully_connected/biases:0
pasta_classification/cls_pasta_0/fully_connected_1/weights:0
pasta_classification/cls_pasta_0/fully_connected_1/biases:0
pasta_classification/cls_pasta_0/fully_connected_2/weights:0
pasta_classification/cls_pasta_0/fully_connected_2/biases:0
pasta_classification/cls_pasta_1/fully_connected/weights:0
pasta_classification/cls_pasta_1/fully_connected/biases:0
pasta_classification/cls_pasta_1/fully_connected_1/weights:0
pasta_classification/cls_pasta_1/fully_connected_1/biases:0
pasta_classification/cls_pasta_1/fully_connected_2/weights:0
pasta_classification/cls_pasta_1/fully_connected_2/biases:0
pasta_classification/cls_pasta_2/fully_connected/weights:0
pasta_classification/cls_pasta_2/fully_connected/biases:0
pasta_classification/cls_pasta_2/fully_connected_1/weights:0
pasta_classification/cls_pasta_2/fully_connected_1/biases:0
pasta_classification/cls_pasta_2/fully_connected_2/weights:0
pasta_classification/cls_pasta_2/fully_connected_2/biases:0
pasta_classification/cls_pasta_3/fully_connected/weights:0
pasta_classification/cls_pasta_3/fully_connected/biases:0
pasta_classification/cls_pasta_3/fully_connected_1/weights:0
pasta_classification/cls_pasta_3/fully_connected_1/biases:0
pasta_classification/cls_pasta_3/fully_connected_2/weights:0
pasta_classification/cls_pasta_3/fully_connected_2/biases:0
pasta_classification/cls_pasta_4/fully_connected/weights:0
pasta_classification/cls_pasta_4/fully_connected/biases:0
pasta_classification/cls_pasta_4/fully_connected_1/weights:0
pasta_classification/cls_pasta_4/fully_connected_1/biases:0
pasta_classification/cls_pasta_4/fully_connected_2/weights:0
pasta_classification/cls_pasta_4/fully_connected_2/biases:0
pasta_classification/cls_pasta_5/fully_connected/weights:0
pasta_classification/cls_pasta_5/fully_connected/biases:0
pasta_classification/cls_pasta_5/fully_connected_1/weights:0
pasta_classification/cls_pasta_5/fully_connected_1/biases:0
pasta_classification/cls_pasta_5/fully_connected_2/weights:0
pasta_classification/cls_pasta_5/fully_connected_2/biases:0
vec_classification/fully_connected/weights:0
vec_classification/fully_connected/biases:0
vec_classification/fully_connected_1/weights:0
vec_classification/fully_connected_1/biases:0
vec_classification/fully_connected_2/weights:0
vec_classification/fully_connected_2/biases:0
vec_attention/fully_connected/weights:0
vec_attention/fully_connected/biases:0
verb_classification/fully_connected/weights:0
verb_classification/fully_connected/biases:0
region_classification/cls_score_P/weights:0
region_classification/cls_score_P/biases:0
region_classification/cls_score_A/weights:0
region_classification/cls_score_A/biases:0
region_classification/fully_connected/weights:0
region_classification/fully_connected/biases:0
region_classification/cls_score_L/weights:0
region_classification/cls_score_L/biases:0
Traceback (most recent call last):
File "tools/Train_pasta_HICO_DET.py", line 93, in
train_net(net, Trainval_GT, Trainval_N, output_dir, tb_dir, args.Pos_augment, args.Neg_select, args.Restore_flag, weight, max_iters=args.max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 279, in train_net
sw.train_model(sess, max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 193, in train_model
lr, train_op = self.construct_graph(sess)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 76, in construct_graph
capped_gvs = [(tf.clip_by_norm(grad, 1.), var) for grad, var in grads_and_vars]
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 76, in
capped_gvs = [(tf.clip_by_norm(grad, 1.), var) for grad, var in grads_and_vars]
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/ops/clip_ops.py", line 166, in clip_by_norm
t.values if isinstance(t, ops.IndexedSlices) else t, name="t")
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1184, in convert_to_tensor
return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1242, in convert_to_tensor_v2
as_ref=False)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1297, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 286, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 227, in constant
allow_broadcast=True)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 265, in _constant_impl
allow_broadcast=allow_broadcast))
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/tensor_util.py", line 437, in make_tensor_proto
raise ValueError("None values not supported.")
ValueError: None values not supported.
`

OS: Ubuntu18.04
Cuda 10.2
CUdnn 7

Does anyone get this error?
Do you have any idea how to fix it?

HAKE-Large+iCAN?

Your repo is set up to use TIN results but there are results from iCAN+HAKE-Large. If one wants to replicate your iCAN+HAKE-Large results to generate PaSta (like reported in your VCOCO results), what are the steps they should take? Specifically, in Generate_detection.py, a lot seems to be dependent on TIN specific results.

A question about the test stage of Instance-level branch

Thanks for your great job on both HOI recognition/detection! I downloaded your code and tried to reproduce your results in the paper, but in the 75L~77L of -Results/Generate_detection.py I felt confused about the details:

sP = torch.sigmoid(torch.from_numpy(score_P[obj_index]).cuda()).cpu().numpy() * P_weight
sA = torch.sigmoid(torch.from_numpy(score_A[obj_index]).cuda()).cpu().numpy() * A_weight
sL = torch.sigmoid(torch.from_numpy(score_L[obj_index]).cuda()).cpu().numpy() * L_weight

As shown, in your code sP, sA, sL are multiplied with P_weight, A_weight and L_weight respectively. The three weights are defined in 42L:

h_fac, o_fac, sp_fac, P_fac, A_fac, L_fac, hthresh, othresh, athresh, bthresh, P_weight, A_weight, L_weight = pickle.load(open('generation_args.pkl', 'rb'))

I carefully checked the content of generation_args.pkl and found that only P_weight has the value of 1.0, while A_weight and L_weight are both 1e-5. And after I removed sA and sL from the final score, the mAP didn't decrease, which means both Attention branch and Language branch will not contribute to the final results with the weight 1e-5.
Is 1e-5 the best weight that you found? Or did I make anything wrong? The problem is important since the reproduction of your ablation study strongly relies on this part.

为什么obj1_roi的dim是3*5

check了10v_attention里面deploy_10v_attention.prototxt
obj1_roi obj2_roi obj3_roi obj4_roi的形状是3*5,那岂不是一共就是12个object了?

What is the function of ```Trainval_Neg_all_part.pkl``` ?

I can understand that Trainval_GT_all_part.pkl provides a lot of info which is useful during training, but what is the function of Trainval_Neg_all_part.pkl? I debuged data loading process and found that sometimes one record is provided, sometimes 60 records are provided. I still can't figure it out. Look forward to your reply.

BTW, I can't find the source link to Trainval_GT_10w.pkl.Can you tell me where to download it? Thank you very much.

    num_pos = Pos_augment
    if image_id in Trainval_Neg.keys():
        NEG = Trainval_Neg[image_id]
        if len(NEG) < Neg_select:
            List = range(len(NEG))
        else:
            List = random.sample(range(len(NEG)), Neg_select)
        for i in range(len(List)):
            Neg = NEG[List[i]]
            Human_augmented.append(np.array([0, Neg[2][0], Neg[2][1], Neg[2][2], Neg[2][3]]).reshape(1, 5))
            Object_augmented.append(np.array([0, Neg[3][0], Neg[3][1], Neg[3][2], Neg[3][3]]).reshape(1, 5))
            Relation_augmented.append(Generate_relation_bbox(Neg[2], Neg[3]))
            part_bbox.append(Neg[7][None, :, :])

Something goes wrong when training PaStaNet in HAKE-Large model!

When I run this training command:
python tools/Train_pasta_HICO_DET.py --data 1 --init_weight 1 --train_module 2 --num_iteration 2000000 --model _pasta_large_pretrain
It turns out that the data loading function is still Augmented_HO_Neg_HICO_DET_for_only_PVP76, and the GT&Neg 10w.pkl has totally different data structure with Trainval_GT_all_part.pkl and Trainval_Neg_all_part.pkl. Is there anything wrong with my code? Looking forward to your replay!

Here is the error:
File "/home/fair/Desktop/wsl/_HOI/Instance_HAKE/tools/../lib/ult/ult_HICO_DET.py", line 433, in Augmented_HO_Neg_HICO_DET_for_only_PVP76 part_bbox.append(GT[i][4]['part_bbox'][None, :, :]) # get part boxes (10,5) TypeError: string indices must be integers

Instance-level HOI Demo

Hi,

It seems that this code does not include the part of human part or object detection. I don't understand why iCAN is used as the object detection? iCAN seems to use Faster-RCNN for object detection.

Is there any way to input an image and output the corresponding instance-level human interaction behavior as a demo?

Thanks,
Tairan Chen

Questions about annotations and annotation parser.

Thanks for your innovative works!

I had some questions while reading the code, and wish for your help.


I will use the image(ID is 32367) and its annotations (from Trainval_GT_all_part.pkl and Trainval_Neg_all_part.pkl) as an example.

Data/hico_20160224_det/images/train2015/HICO_train2015_00032367.jpg:

HICO_train2015_00032367.jpg

Q1: HOI does not match the object class

The image has one GT annotation in effect, others are augmented to Pos_augment(default 15).
Its GT HOI ID is : 201, detail: (201, apple, eat) (refer to list of HOIs)

However, Its GT object80 ID is 47, detail: (047 orange) (refer to list of objects)

For example in trainval_GT[32367][0]:

image

Q2: What the meaning of using 57(corresponding to move) and using Neg[4]-1 in annotation parser of Neg.

action_verb.append(Generate_action_PVP(57, 117))

gt_object.append(Generate_action_object(Neg[4] - 1, 80))

Q3: I am still confused about negative examples.

After reading the explanation, I'am not sure whether my comprehension is correct:

GT samples is the samples with interaction and Neg without interaction.
Neg samples gave some wrong HOIs and using Generate_action_PVP(-1, N) to mark these HOIs are wrong.

For example in Trainval_Neg[32367]:

image

The women is reading the book(corresponding to HOI (251, book, read)), but GT doesn't has the HOI and Neg has(in Trainval_Neg[32367][2])

Why marked this HOI to Neg instead of GT?

Q4: Annotation format

Although I can speculate most of the format of Trainval_Neg and trainval_GT, I wonder if I could learn the official definition about the formats of these annotation files in some where.

Not found error

When i run cmd python tools/Train_pasta_HICO_DET.py --data 0 --init_weight 1 --train_module 2 --num_iteration 2000000 --model test_model, I didn't found resnet_101 model as in your code. So I download model from pretrained model from tensorflow zoo. I download resnet_101_coco, but I see present 50 layers. So I download resnet_50 and meet below error.
`
Restoring model snapshots from /home/ubuntu/project/Action_Recognition/HAKE-Action/Weights/faster_rcnn_resnet50_coco_2018_01_28/model.ckpt
WARNING:tensorflow:From /home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py:118: The name tf.model_variables is deprecated. Please use tf.compat.v1.model_variables instead.

2020-05-08 18:20:31.852477: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Key resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
[[{{node save_1/RestoreV2}}]]
(1) Not found: Key resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
[[{{node save_1/RestoreV2}}]]
[[save_1/RestoreV2/_441]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Key resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
[[node save_1/RestoreV2 (defined at /home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Not found: Key resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
[[node save_1/RestoreV2 (defined at /home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[save_1/RestoreV2/_441]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'save_1/RestoreV2':
File "tools/Train_pasta_HICO_DET.py", line 93, in
train_net(net, Trainval_GT, Trainval_N, output_dir, tb_dir, args.Pos_augment, args.Neg_select, args.Restore_flag, weight, max_iters=args.max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 279, in train_net
sw.train_model(sess, max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 201, in train_model
self.from_snapshot(sess) # load from snapshot
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 141, in from_snapshot
self.saver_restore = tf.train.Saver(saver_t)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1300, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1618, in object_graph_key_mapping
object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 915, in get_tensor
return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "tools/Train_pasta_HICO_DET.py", line 93, in
train_net(net, Trainval_GT, Trainval_N, output_dir, tb_dir, args.Pos_augment, args.Neg_select, args.Restore_flag, weight, max_iters=args.max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 279, in train_net
sw.train_model(sess, max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 201, in train_model
self.from_snapshot(sess) # load from snapshot
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 142, in from_snapshot
self.saver_restore.restore(sess, self.pretrained_model)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1306, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

2 root error(s) found.
(0) Not found: Key resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
[[node save_1/RestoreV2 (defined at /home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Not found: Key resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
[[node save_1/RestoreV2 (defined at /home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[save_1/RestoreV2/_441]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'save_1/RestoreV2':
File "tools/Train_pasta_HICO_DET.py", line 93, in
train_net(net, Trainval_GT, Trainval_N, output_dir, tb_dir, args.Pos_augment, args.Neg_select, args.Restore_flag, weight, max_iters=args.max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 279, in train_net
sw.train_model(sess, max_iters)
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 201, in train_model
self.from_snapshot(sess) # load from snapshot
File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/train_Solver_HICO_DET_pasta.py", line 141, in from_snapshot
self.saver_restore = tf.train.Saver(saver_t)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/act_reg/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

`
Can you check which model should I use with that cmd? Tks you.

The mismatch of parameters.

In line 372 of pasta_HICO_DET.py, the self.vec_attention function is called. And the parameters of this function cotains from fc7_P0 to fc7_P5.

fc7_P_att = self.vec_attention(self.predictions['cls_prob_vec'], fc7_P0, fc7_P1, fc7_P2, fc7_P3, fc7_P4, fc7_P5, fc5_O, initializer, is_training, name='vec_attention')

But in the defination of self.vec_attention, the parametera are fc5_P0 to fc5_P5. Although there is no error, but it's a good choise to fix it up.

def vec_attention(self, cls_prob_vec, fc5_P0, fc5_P1, fc5_P2, fc5_P3, fc5_P4, fc5_P5, fc5_O, initializer, is_training, name): with tf.variable_scope(name) as scope: [] return fc7_P_att

NameError: name 'scores' is not defined in testing

I test after training with cmd:

python tools/Test_pasta_HICO_DET.py --iteration 2000000 --model HICO_DET_model

HICO_DET_model is my model folder.
It returns following error:

Traceback (most recent call last):
  File "tools/Test_pasta_HICO_DET.py", line 60, in <module>
    test_net(sess, net, Test_RCNN, output_file)
  File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/test_Solver_HICO_DET_pasta.py", line 104, in test_net
    im_detect(sess, net, image_id, Test_RCNN, keys, scores, bboxes, hdet, odet)
NameError: name 'scores' is not defined

Does anyone have solution for this error?

Image-level分支scripts/test.py文件有几处笔误

第6行'from obj_mas import prior_info'中的'obj_mask'少写了'k';
第37行'snaps/10v-attention.caffemodel.h5'中的间隔符'-'需要改成'',下载的模型名称中用的就是'';
第42行和37行一样'snaps/language-model.caffemodel.h5'中的间隔符'-'需要改成'_';
第47、50、53、56、59行中的'args.mode'需改为'args.pasta_mode';

我在提供的docker版本上进行了测试,暂时没有出现其他问题。再次感谢作者的工作!

consistency of instruction in testing code and structure folder

After retrain model with HICO_DET, i want to test with

python tools/Test_pasta_HICO_DET.py --num_iteration 2000000 --model pasta_HICO-DET

First, I get

(act_reg) ubuntu@02203cbabcdc:~/project/Action_Recognition/HAKE-Action$ python tools/Test_pasta_HICO_DET.py --num_iteration 2000000 --model pasta_HICO-DET
Traceback (most recent call last):
  File "tools/Test_pasta_HICO_DET.py", line 15, in <module>
    from models.test_Solver_HICO_DET_pasta import test_net
  File "/home/ubuntu/project/Action_Recognition/HAKE-Action/tools/../lib/models/test_Solver_HICO_DET_pasta.py", line 8, in <module>
    from ult.ult import Get_next_sp_with_pose, Generate_part_bbox, Generate_relation_bbox, Generate_action_object
ModuleNotFoundError: No module named 'ult.ult'

I suppose that because you might change file name, so I update from

from ult.ult import Get_next_sp_with_pose, Generate_part_bbox, Generate_relation_bbox, Generate_action_object

in /lib/models/test_Solver_HICO_DET_pasta.py", line 8
to

from ult.ult_HICO_DET  import Get_next_sp_with_pose, Generate_part_bbox, Generate_relation_bbox, Generate_action_object

I copy file from -Result folder to lib/util folder, change
from ult.ult_HICO_DET import obj_range, get_map to
from ult.HICO_DET_utils import obj_range, get_map

so please update your source code to help other.
Please don't name folder start with special character '-Result', it makes me very hard to use with Linux console.
Tks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.