oulutan / acam_demo Goto Github PK

why the I3D is cuted on Mixed_4f?

Thanks for wonderful work , I want to train my own data set which have 6 classes , I have spited the videos with various length clips and keep them into each category . I want to annotate and Pre-Process the data set . Can guide me about the annotation process ,what type of pre-processing I need to do before doing the annotations and Which tool I can use to annotate the videos . What post annotations processing I need to done to complete the my training process . Thanks in Advance

Thanks for your outstanding work. I tried the code and it works great. I have one question, is there some way (here or during training) to make the code smaller (as for memory and computation)? Of course the accuracy will be lower but I would like to run it on some embedded device.

I am running the following command

python detect_actions.py --video_path=output.mp4

But I am facing this issue

File "detect_actions.py", line 97, in main
tracker.update_tracker(detection_info, cur_img)
File "/home/user/Downloads/ACAM_Demo-master/object_detection/object_detector.py", line 166, in update_tracker
detection_conf = track.last_detection_confidence
AttributeError: 'Track' object has no attribute 'last_detection_confidence'
Fatal Python error: could not acquire lock for <_io.BufferedReader name=9> at interpreter shutdown, possibly due to daemon threads

Thanks for posting a great demo!

Do you have any recommendations for annotation tools? I'm looking for a tool for annotating some of my videos.

Instructions for updating: Use standard file APIs to check for files with this prefix. frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0 frame_q: 255, obj_q: 0, act_q: 0, vis_q: 0
Hello, the output is caught in such an endless loop. What's wrong with it? In addition, it seems that gpu is not used in this one, because tensorflow in the code is not used at all. Is this true?

Dear
when will you release the training and evaluation codes? Thank you for your great work.

Thank you for very nice work . I want to training my own data set ,which have 4 classes and I have 200 videos for every class . Can anybody Guide me what steps I need to do to train my own data set using this technique .Does I need to do annotation for every video or there is any other way which I need to adopt to train my own data set on this technique. Please give a complete overview for training my own data set. Thank you

@oulutan I have been trying to run multiprocess_detect_actions.py. I am getting the following output?
Could you please have a look? It did not give the output video.

python multiprocess_detect_actions.py --video_path sample_input.mp4
/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
Reading video file sample_input.mp4
H: 720, W: 1280
Running actions every 8 frame
Writing output to ./output_videos/sample_input_output.mp4
Loading object detection model at ./object_detection/weights/ssd_mobilenet_v2_coco_2018_03_29/frozen_inference_graph.pb
/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
2019-02-20 17:03:23.811722: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-20 17:03:24.746611: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Using model soft_attn
Loading weights from ./action_detection/weights/model_ckpt_soft_attn_pooled_cosine_drop_ava-130
frame_q: 0, obj_q: 0, act_q: 0, vis_q: 0
Done!

Hello @oulutan ,

thanks very much for this wonderful repo. I have a few confusions regarding the paper and the implementation itself.

1. Although you mentioned in your paper how your work differs from and improves upon the Actor Conditioned Relation Network (ACRN) paper, I am still confused about the main improvements. I would highly appreciate if you could provide some intuitions about how your work improves upon ACRN.

2. It seems that actor_infos keep increasing and never cleared up, whereas, the tracks corresponding to the actors may have been deleted. Shouldn't you delete the actors from actor_infos whose corresponding tracks are deleted?

Thanks.

Hi.
Thank you for your work,
I confirmed your demo. Very impressive!
I would like to use your method for training, may I ask when the training script will be released?

Thanks.

Hello, first of all let me congratulate you on your work, and thank you for sharing it!

Quick question:

• I see you limit the number of actors to 14 (in both detect_actions.py and multiprocess_detect_actions.py). Just not to increase memory usage or other specific reason?

Thanks

Hello @oulutan ,

As per the demo code, it seems that the inputs to I3d network are the cropped cuboids, one for each detected actor. But, in your paper, you mentioned that the input to I3d network is the whole video. Am I missing something? I would appreciate your response.

Thanks

I followed all the methods which are provided in the documentation. But while running python detect_actions.py, I ran into the following issue. Please help!

Traceback (most recent call last):
  File "detect_actions.py", line 11, in <module>
    import object_detection.object_detector as obj
  File "/home/hs/Spritle/siemens/ACAM_Demo/object_detection/object_detector.py", line 64, in <module>
    from tools.generate_detections import create_box_encoder
ImportError: No module named tools.generate_detections

I have run "multiprocess_detect_actions.py" and it is running on the terminal

Instructions for updating:
Use keras.layers.conv3d instead.
frame_q: 214, obj_q: 0, act_q: 0, vis_q: 0
frame_q: 214, obj_q: 0, act_q: 0, vis_q: 0
frame_q: 214, obj_q: 0, act_q: 0, vis_q: 0
0.383 second per image
frame_q: 214, obj_q: 2, act_q: 0, vis_q: 1
frame_q: 214, obj_q: 5, act_q: 0, vis_q: 4
frame_q: 214, obj_q: 5, act_q: 0, vis_q: 4
frame_q: 214, obj_q: 5, act_q: 0, vis_q: 4
frame_q: 214, obj_q: 5, act_q: 0, vis_q: 4
frame_q: 214, obj_q: 5, act_q: 0, vis_q: 4
frame_q: 214, obj_q: 5, act_q: 0, vis_q: 4

It gives No output opencv window is opened ,neither output video is saved and it get stuck in loop.Any idea?

Thanks For Nice Work . I am testing my videos on the pre-trained weights . I have run the following command

python detect_actions.py --video_path=output.mp4

But After running for some frames, It Give me following error

Instructions for updating:
Use standard file APIs to check for files with this prefix.
frame_cnt: 1
obj det 3.26 seconds
tracker 0.00 seconds
frame_cnt: 2
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 3
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 4
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 5
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 6
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 7
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 8
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 9
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 10
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 11
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 12
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 13
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 14
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 15
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 16
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 17
obj det 0.08 seconds
tracker 0.00 seconds
frame_cnt: 18
obj det 0.11 seconds
tracker 0.00 seconds
frame_cnt: 19
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 20
obj det 0.10 seconds
tracker 0.00 seconds
frame_cnt: 21
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 22
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 23
obj det 0.09 seconds
tracker 0.00 seconds
frame_cnt: 24
obj det 0.08 seconds
tracker 0.19 seconds
frame_cnt: 25
obj det 0.09 seconds
tracker 0.01 seconds
frame_cnt: 26
obj det 0.08 seconds
tracker 0.01 seconds
frame_cnt: 27
obj det 0.09 seconds
tracker 0.01 seconds
frame_cnt: 28
obj det 0.08 seconds
Traceback (most recent call last):
File "detect_actions.py", line 308, in
main()
File "detect_actions.py", line 97, in main
tracker.update_tracker(detection_info, cur_img)
File "/home/user/Downloads/ACAM_Demo-master/object_detection/object_detector.py", line 166, in update_tracker
detection_conf = track.last_detection_confidence
AttributeError: 'Track' object has no attribute 'last_detection_confidence'

Can you guide me about error . Thank you

@oulutan Does the IP camera implementation possible with your implementation? If yes, how it would in terms of performance? Thanks

Dear

I run the demo when using GTX1080Ti，and found it needs a large mount of GPU memory, "CUDA_ERROR_OUT_OF_MEMORY: out of memory" will be got , and a moment later ,it crashed!
 I want to know whether it is like that?

Thank you !

@oulutan Thanks for nice work, I am trying to run the training code but it is giving me this error "faster_rcnn_ava_model_0255000.pth", can u provide it for me. Thanks

Thanks for nice work . I am working on custom data set to train the Model . Please guide me how to annotate and prepare my own data set to train the model . Thank You

when run detect_actions.py

Hi, thanks for your great works,

I would like to detect only two classes on the AVA dataset. Should I do training on the two classes or there is a way on the demo to limit the model with two classes only ?

Thank you for your work, especially for your open code.could you please tell when you will open your training code?

Pytorch code available . Kindly help

when run

python detect_actions.py --video_path sample_input.mp4

@oulutan I tried the detect_actions.py but it seems something went wrong.

 python detect_actions.py --video_path sample_input.mp4 
/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
Loading object detection model at /home/prashantb/Documents/prashant_workspaace/ACAM_Demo/object_detection/weights/ssd_mobilenet_v2_coco_2018_03_29/frozen_inference_graph.pb
2019-02-20 15:40:08.703071: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Reading video file sample_input.mp4
Running actions every 8 frame
/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
Using model soft_attn
Loading weights from ./action_detection/weights/model_ckpt_soft_attn_pooled_cosine_drop_ava-130
frame_cnt: 1
obj det 2.30 seconds
tracker 0.00 seconds
frame_cnt: 2
obj det 0.04 seconds
tracker 0.10 seconds
frame_cnt: 3
obj det 0.04 seconds
tracker 0.02 seconds
frame_cnt: 4
obj det 0.04 seconds
tracker 0.01 seconds
frame_cnt: 5
obj det 0.04 seconds
tracker 0.01 seconds
frame_cnt: 6
obj det 0.04 seconds
tracker 0.01 seconds
frame_cnt: 7
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 8
obj det 0.03 seconds
tracker 0.01 seconds
2019-02-20 15:40:15.682873: W tensorflow/core/framework/allocator.cc:122] Allocation of 353894400 exceeds 10% of system memory.
Person 1
	 stand: 0.201
	 walk: 0.173
	 watch (a person): 0.084
	 carry/hold (an object): 0.051
	 talk to (e.g., self, a person, a group): 0.031
action 2.96 seconds
frame_cnt: 9
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 10
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 11
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 12
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 13
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 14
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 15
obj det 0.04 seconds
tracker 0.01 seconds
frame_cnt: 16
obj det 0.03 seconds
tracker 0.01 seconds
2019-02-20 15:40:18.724569: W tensorflow/core/framework/allocator.cc:122] Allocation of 353894400 exceeds 10% of system memory.
Person 1
	 walk: 0.868
	 watch (a person): 0.176
	 carry/hold (an object): 0.102
	 stand: 0.096
	 talk to (e.g., self, a person, a group): 0.075
action 2.71 seconds
frame_cnt: 17
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 18
obj det 0.04 seconds
tracker 0.01 seconds
frame_cnt: 19
obj det 0.04 seconds
tracker 0.01 seconds
frame_cnt: 20
obj det 0.03 seconds
tracker 0.01 seconds
frame_cnt: 21
obj det 0.03 seconds
tracker 0.01 seconds
Traceback (most recent call last):
  File "detect_actions.py", line 314, in <module>
    main()
  File "detect_actions.py", line 182, in main
    out_img = visualize_detection_results(tracker.frame_history[-16], tracker.active_actors, prob_dict)
  File "detect_actions.py", line 244, in visualize_detection_results
    cv2.rectangle(disp_img, (left,top), (right,bottom), color, 3)
TypeError: only size-1 arrays can be converted to Python scalars
Fatal Python error: could not acquire lock for <_io.BufferedReader name=10> at interpreter shutdown, possibly due to daemon threads

Thread 0x00007fcb45de3700 (most recent call first):
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/imageio_ffmpeg/_parsing.py", line 61 in run
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fcb86a8e700 (most recent call first):
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/subprocess.py", line 1537 in _communicate
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/subprocess.py", line 843 in communicate
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/imageio_ffmpeg/_io.py", line 189 in read_frames
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/imageio/plugins/ffmpeg.py", line 342 in _close
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/imageio/core/format.py", line 252 in close
  File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/imageio/core/format.py", line 241 in __del__
Aborted (core dumped)
```

Thanks for sharing your excellent work! I've not run the code but I am wondering whether the person detector you used is finetuned on AVA. The detectors shown in your paper F RCNN-NAS and F RCNN-Resnet101 achieve high performance, with AP 97.10 and 95.97 on AVA, respectively. But I use a FPN-ResNet101 trained on COCO (43 mAP on 80 categories) and it can only achieve nearly 70 AP on AVA. Finetuning models on AVA can improve the performance to ~90. Finetuning detector is a common practice in many papers solving AVA, like SlowFast, Long Term bank.

@oulutan Thank you very much for sharing your work. This is the first of a kind of action recognition demo I have seen. I am trying to implement your code. I had a few issues while implementing the detect_actions.py. I will be grateful to you if you could give me some hints as a newbie in this domain.
/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec() return _inspect.getargspec(target) Loading object detection model at /home/oytun/work/Conditional_Attention_Maps_Demo/object_detection/weights/ssd_mobilenet_v2_coco_2018_03_29/frozen_inference_graph.pb Traceback (most recent call last): File "detect_actions.py", line 312, in <module> main() File "detect_actions.py", line 52, in main obj_detector = obj.Object_Detector(obj_detection_graph) File "/home/prashantb/Documents/prashant_workspaace/ACAM_Demo/object_detection/object_detector.py", line 12, in __init__ serialized_graph = fid.read() File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 125, in read self._preread_check() File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 85, in _preread_check compat.as_bytes(self.__name), 1024 * 512, status) File "/home/prashantb/anaconda3/envs/demo/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.NotFoundError: /home/oytun/work/Conditional_Attention_Maps_Demo/object_detection/weights/ssd_mobilenet_v2_coco_2018_03_29/frozen_inference_graph.pb; No such file or directory

Thank you for your work, especially for your open code. I really want to use this to my own project, so could you please tell when you will open your training code?