bigredt / no_frills_hoi_det Goto Github PK

Hi @BigRedT ! I'm studying your paper and still have a little question about your "masking" strategy. In your approach, you mask out all the easy negative candidates during both training and testing, and it does reduce the candidate boxes significantly, but have you tried not to mask out these candidates and predict scores for 600 HOI classes (instead of 117 predicates)? Is the mAP performance better or worse compared to the results in your paper? I think it's an interesting trade-off between whether we should fully "trust" the detector or not : )

I want to compare the performance on V-COCO dataset, I read the code and find it basically wrote for HICO-DET, so can I adjust it to V-COCO by modifying several parts of it?

I've been playing around with the HOI candidate files which led to some questions.
Why are you including hoi candidates with low object detection scores?
Why are you including hoi candidates where the human and object are the same bounding box? (this only concerns the case where object=human)

I want to detect actions on my customized dataset using pretrained model on HICO dataset which scripts do I use and what specific setting I need to do as I don't want to download features/dataset/json files etc that is provided in this repository

Hello, thanks for your sharing. I unzip top_boxes_per_hoi.zip but the zip you upload has broken. Could you reload the top_boxes_per_hoi.zip

What is your configurations when using res152 backbone on faster-rcnn?
(I do not see the res152.yml in the pytorch-faster-rcnn/experiments/cfgs)

Thanks for this awesome project 👍

I was wondering if it's possible to evaluate the predicted bouding boxes on the hico-det dataset without looking at the interaction?
I would like to see how different detectors perform on the dataset

Cheers

Thanks for your job!
And I have some questions.
Q1: When I evaluating the model, I only find the code of Default mode,where can I get the KO mode result?
Q2: I just download the res152 faster-rcnn model to generate box and evaluate the connection recall followed your README.md, but I found that the recall results are not as better as your showed in your README.md. Could you give me some advice and tell me why?

Hi,

I came across some problems on Figure3 in your paper. I don't know how you collected the ground truth and corresponding interaction. Could you give me some more details on how to construct the confusion matrix? If possible, could you mind sending me the related code?

Thanks a lot. Look forward to hearing from you.

Best regards.

Many thanks for your great work! I have some questions about your sampling strategy of pos and neg samples.

1. According to README.md, using only the outputs of a pre-trained detector leads to a drop on recall of ground truth HOI labels (only 59% for the labelled case and 77% the unlabelled case) so that even if we ignore the object classes given by the detector, we can use only 77% of the training data to train the model. Did you manually add the missing training samples or just use the outputs from the detector to do the training?

2. In HICO-DET dataset, 'no_interaction' is treated as one kind of 'interaction' in the annotation file. However, most 'no_interaction' instances in the images are not annotated and this may lead to a huge drop when computing mAP. Did you label the negative samples of your training data as 'no_interaction' or 'none of the existing 600 HOI labels'?

Thank you again for your attention!

Hi!

Thanks a lot for the very clear and understandable code and repository. One small comment I can add is maybe this repo can benefit from a warning that extracted data size is huge: Currently it occupies around 700G in my disk. I am not sure if this is the expected behavior?

Traceback (most recent call last): File "/home/mathias/miniconda/envs/no_fills/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/mathias/miniconda/envs/no_fills/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/mathias/no_frills_hoi_det/exp/hoi_classifier/run.py", line 355, in <module> list_exps(globals()) File "/home/mathias/no_frills_hoi_det/exp/experimenter.py", line 16, in list_exps module_globals[args.exp]() File "/home/mathias/no_frills_hoi_det/exp/hoi_classifier/run.py", line 234, in exp_train train.main(exp_const,data_const_train,data_const_val,model_const) File "/home/mathias/no_frills_hoi_det/exp/hoi_classifier/train.py", line 173, in main train_model(model,dataset_train,dataset_val,exp_const) File "/home/mathias/no_frills_hoi_det/exp/hoi_classifier/train.py", line 60, in train_model max_prob = hoi_prob.max().data[0] IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

getting this error when running exp/hoi_classifier/scripts/train.sh. Not sure if I should use tensor.item() to fix it or if there is another fix.

Hello, thank you for releasing the codes for your nice work.
Did you use the Faster RCNN with Resnet-152-FPN backbone or just Faster RCNN with Resnet-152?

Thanks.

The following aguments were specified:

--fp_to_tp_ratio 1000
--imgs_per_batch 1
--rcnn_det_prob True
--verb_given_appearance True
--verb_given_boxes_and_object_label True
--verb_given_human_appearance False
--verb_given_human_pose True
--verb_given_object_appearance False
Saving exp constants ...
Saving data_train constants ...
Saving data_val constants ...
Saving model constants ...
Creating model ...
Writing model.txt ...
Creating data loaders ...
Traceback (most recent call last):
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/run.py", line 356, in
list_exps(globals())
File "/data/cuiyujie/no_frills_hoi_det/exp/experimenter.py", line 16, in list_exps
module_globalsargs.exp
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/run.py", line 235, in exp_train
train.main(exp_const,data_const_train,data_const_val,model_const)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/train.py", line 173, in main
train_model(model,dataset_train,dataset_val,exp_const)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/train.py", line 49, in train_model
prob_vec, factor_scores = model.hoi_classifier(feats)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/models/hoi_classifier_model.py", line 102, in forward
temp_m = module(feats)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/models/verb_given_human_pose.py", line 65, in forward
w.write("self.mlp({}):{}\n".format(in_feat, self.mlp(in_feat)))
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/no_frills_hoi_det/utils/pytorch_layers.py", line 102, in forward
w.write("x:{}\nlayer(x):{}\n\n".format(x, layer(x)))
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 55, in forward
return F.linear(input, self.weight, self.bias)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/functional.py", line 835, in linear
return torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:247
(pytorch030) cuiyujie@amax:/data/cuiyujie/no_frills_hoi_det$ bash exp/hoi_classifier/scripts/train.sh 7
/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters

The following aguments were specified:

--fp_to_tp_ratio 1000
--imgs_per_batch 1
--rcnn_det_prob True
--verb_given_appearance True
--verb_given_boxes_and_object_label True
--verb_given_human_appearance False
--verb_given_human_pose True
--verb_given_object_appearance False
Saving exp constants ...
Saving data_train constants ...
Saving data_val constants ...
Saving model constants ...
Creating model ...
Writing model.txt ...
Creating data loaders ...
Traceback (most recent call last):
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/run.py", line 356, in
list_exps(globals())
File "/data/cuiyujie/no_frills_hoi_det/exp/experimenter.py", line 16, in list_exps
module_globalsargs.exp
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/run.py", line 235, in exp_train
train.main(exp_const,data_const_train,data_const_val,model_const)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/train.py", line 173, in main
train_model(model,dataset_train,dataset_val,exp_const)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/train.py", line 49, in train_model
prob_vec, factor_scores = model.hoi_classifier(feats)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/models/hoi_classifier_model.py", line 110, in forward
factor_scores[factor_name] = module(feats)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/no_frills_hoi_det/exp/hoi_classifier/models/verb_given_human_pose.py", line 67, in forward
factor_scores = self.mlp(in_feat)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/no_frills_hoi_det/utils/pytorch_layers.py", line 107, in forward
x = layer(x)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 55, in forward
return F.linear(input, self.weight, self.bias)
File "/data/cuiyujie/xuyuxiao/anaconda2/envs/pytorch030/lib/python3.6/site-packages/torch/nn/functional.py", line 835, in linear
return torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:247

When executing command "bash exp/hoi_classifier/scripts/train.sh 7", it shows the above error.
Is there something wrong with the network size?

When I'm trying to extract features with f[global_id]['boxes_scores_rpn_ids_hoi_idx'] in many of the cases I'm getting the error:
KeyError: "Unable to open object (object 'HICO_train2015_00000022' doesn't exist)"
I ran a for loop on the training data and counted 7623 missing entries.
Can you explain why there are so many missing entries?

Cheers 👍

I'm having problems understanding why these bounding boxes in the image below don't fit the two persons. I'm extracting the bboxes from the selected_coco_cls_dets file.