mkocabas / epipolarpose Goto Github PK

When i use four gpu to train this model, one the first gpu, Memory Usage is about 10761MB/12196M, while the other GPU all 3651M/12196M, which waste too much memory usage. Why do not you use GPU balance function (like DataParallelModel in Integral LOSS)? or can you account for the same situation?

Hi, @mkocabas Thanks for sharing your wonderful project! I have a question about triangulation. To my knowledge, both the camera rotation and translation parameters are needed to use the polynomial triangulation algorithm. But it seems your paper did not tell how to derive the camera translation parameters, and your codes do not show how to call the function polynomial_triangulation. Can you share some experience or supplement some demo of calling polynomial_triangulation? Thx a lot!

Hi,

Thanks for your great work. Your paper mentioned that when camera parameters are not available, the body joints can be used as calibration targets. I'm interesting about this, but I didn't find related code in your repo. Can you release the code or explaine more details about it? If I have some synchronized multi-view videos without 3D pose ground-truth, can I use your method to estimate camera parameters?

Looking forward to your reply, thank you!

Hi Muhammad,

Do you fine-tune your 2D pose estimation model on Human3.6M? Or just use a model pre-trained on MPII and keep it fixed?

Thanks,
Umar

When I run demo.py , the error happened.

Traceback (most recent call last):
File "C:/Users/123/Desktop/EpipolarPose-master/EpipolarPose-master/demo.py", line 62, in
ax = fig.add_subplot('122', projection='3d', aspect=1)
File "F:\ana\lib\site-packages\matplotlib\figure.py", line 1414, in add_subplot
a = subplot_class_factory(projection_class)(self, *args, **kwargs)
File "F:\ana\lib\site-packages\matplotlib\axes_subplots.py", line 69, in init
self._axes_class.init(self, fig, self.figbox, **kwargs)
File "F:\ana\lib\site-packages\mpl_toolkits\mplot3d\axes3d.py", line 101, in init
super().init(fig, rect, frameon=True, *args, **kwargs)
File "F:\ana\lib\site-packages\matplotlib\axes_base.py", line 509, in init
self.update(kwargs)
File "F:\ana\lib\site-packages\matplotlib\artist.py", line 974, in update
ret = [_update_property(self, k, v) for k, v in props.items()]
File "F:\ana\lib\site-packages\matplotlib\artist.py", line 974, in
ret = [_update_property(self, k, v) for k, v in props.items()]
File "F:\ana\lib\site-packages\matplotlib\artist.py", line 971, in _update_property
return func(v)
File "F:\ana\lib\site-packages\matplotlib\axes_base.py", line 1281, in set_aspect
'It is not currently possible to manually set the aspect '
NotImplementedError: It is not currently possible to manually set the aspect on 3D axes

Hi, mkocabas!
I can't understand the structure of project. I mean i am not getting that which folder contains which codes? . Like...which folder contains code of 3D pose estimator(integral pose)?,which folder contains code of 2D pose estimator(integral pose)?,and in which folder you are using their output to compute loss...likewise.
Over all, Can you please give brief ideas about which file(.py) is performing what?
That would be very much helpful..
Thank you very much !!.

hi,
Thanks for your works!

Could you be kind to tell me what's the difference of train-fs.pkl and train-ss.pkl ?

can you explain the postprocessing functions like show3Dpose and drawskeleton ? I need to know them to see if the model is problematic or the visualizaition funcitons?

Thanks for your work!
I have tried your demo.ipynb file. It works well when I use pictures from H36M. Speed and accuracy are both perfect. However, I get wrong result if I use my own picture. I think the picture needs some modification before testing. Can you tell me the requirement of the picture? My wrong result as follow.

hi i am getting this error , while running the demo.ipynb in google colab with my image. can you help me for this.

Hello, Thanks for your great work.
I am trying to train human3.6m dataset with your code. I have downloaded the human 3.6m dataset from the website and use your provided script to extract images.But when I train the model, I find the image path in the annotation file(both train-fs.pkl and train-ss.pkl) doesn't match my image path.
For example:
the image path in the annotation file is: 'h36m/images/S1/Photo_1.58860488/000781.jpg'
However, I can only find the path 'h36m/images/S1/TakingPhoto_1.58860488/000781.jpg' ,which is the most similar one to the given path.
I wonder whether you make some changes to the image path when generating annotation files, or can you provide the annotation preparation script?
Thank you very much!

Thanks for opening source codes. However, in the self-supervise setting, I could not find the codes using the model to predict 2D pose and subsequently triangulating 3D joints as the label, so I wonder if you perform the triangulations offline?

And what the parameter cfg.DATASET.TRI means, dose it means using triangulation or not? if so, why it is set false in 'train-ss.yml'?

Looking forward to your reply, thanks.

Hi,

Wonderful work and thanks for sharing the code.

I'm wondering if you have released the implementation of Pose Structure Score PSS and the evaluation script used to calculate scores for different values of K (clustering), because I did not find it in the repo.

It will be very nice if you can share it with us.

Thanks !

dear @mkocabas,
can you send me the result of below?
" You would need Human3.6M data to train or test our model. For Human3.6M data, please download from Human 3.6 M dataset. You would need to create an account to get download permission. After downloading video files, you can run our script to extract images. Then run ln -s <path_to_extracted_h36m_images> ${ROOT}/data/h36m/images to create a soft link to images folder. Currently you can use annotation files we provided in step 4, however we will release the annotation preparation script soon after cleaning and proper testing."

Hi,

I use your pretrained model mpii_heatmap.pth.tar for testing on MPII dataset.
I obtain a very low ap of 5.7 rather than 88.5 as your report.

Hi, thanks for such a great work. I've a question. In your code is it possible that we can use more than one live cameras to estimate human pose? You used H36M dataset which is also based on mutli-views so I'm thinking about it.

Hi mkocabas,

Camera intrinsic and extrinsic are saved in train-xx.pkl files, but could you tell me how do you get those parameters from human3.6m dataset? Thanks:)

I used cameras.py to convert coordinations.
I convert a 2dpoint to camera coordination in camera 0 and 1, and the result camera coordinations are right as annotation describe.But when I convert them to world coordination, those two coordinations are not the same. Have I did something wrong?

I used pickle to load the '.pkl' file,but occurred a problem called ''ModuleNotFoundError: No module named 'lib''. If I create a '.pkl' file, I could load this file successfully.So I think the annotation file downloaded has some problems.

dear,
After I complete training, how to test a video or picture?
Show me please!
Thank you

When I want to unzip any of the pretrained models I get an error. Is something faulty with the files?

I followed the readme to config the environment. When I tried to run the demo.ipynb, I encounted the following errors. It seems that the demo runs using GPUs only. However, I have no access to GPU right now. How can I run the demo using CPU instead of GPUs.

`---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
in
1 model = models.pose3d_resnet.get_pose_net(config, is_train=False)
2 gpus = [int(i) for i in config.GPUS.split(',')]
----> 3 model = torch.nn.DataParallel(model, device_ids=gpus).cuda()
4 print('Created model...')
5

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in cuda(self, device)
263 Module: self
264 """
--> 265 return self._apply(lambda t: t.cuda(device))
266
267 def cpu(self):

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in _apply(self, fn)
191 def _apply(self, fn):
192 for module in self.children():
--> 193 module._apply(fn)
194
195 for param in self._parameters.values():

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in _apply(self, fn)
191 def _apply(self, fn):
192 for module in self.children():
--> 193 module._apply(fn)
194
195 for param in self._parameters.values():

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in _apply(self, fn)
197 # Tensors stored in modules are graph leaves, and we don't
198 # want to create copy nodes, so we have to unpack the data.
--> 199 param.data = fn(param.data)
200 if param._grad is not None:
201 param._grad.data = fn(param._grad.data)

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in (t)
263 Module: self
264 """
--> 265 return self._apply(lambda t: t.cuda(device))
266
267 def cpu(self):

~/.local/lib/python3.5/site-packages/torch/cuda/init.py in _lazy_init()
160 raise RuntimeError(
161 "Cannot re-initialize CUDA in forked subprocess. " + msg)
--> 162 _check_driver()
163 torch._C._cuda_init()
164 _cudart = _load_cudart()

~/.local/lib/python3.5/site-packages/torch/cuda/init.py in _check_driver()
80 Found no NVIDIA driver on your system. Please check that you
81 have an NVIDIA GPU and installed a driver from
---> 82 http://www.nvidia.com/Download/index.aspx""")
83 else:
84 # TODO: directly link to the alternative bin that needs install

AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx`

Hi,

I have a few things to be confirmed after reading your code.
1 Did you compute the PCK by centering the human structure to the root joint (the hip) ?
2 It seems that you transform the 3d pose in camera coordiante system to the world coordinate system, is the world coordinate system actually one of the cameras' coordiante system or the world coordiante system accroding to the extrinsic matrix given by the dataset ?

Thank you

Does it work on ubuntu 18? I have a problem with ubuntu 16 on my laptop.

Hi, thank you for sharing the codes!
I retrained the model in fully supervised setting and got 54.6, 54.9, 55.3, 56.2(hm36_17j). And I valid the best model you provided and got 52.9 which should be 51.8.
Could you help me find out where the problem is? Here is my configuration:
Python 3.7.1
Pytorch 1.3.1
Cuda 10.0.130

Besides, I found some annotation error in S9_SittingDown_15011271, S9_SittingDown_158860488, S9_SittingDown_160457274, 75 images in all. I do not go over all the images, so I do not know if there are other wrong annotations. I think that maybe a reason for the lower performance.

dear,
I mean that testing my picture after training but demo.ipynb is run before training.
thank you so much.

h36m_extract_frames_from_videos.py failed to extract images.
The error is: Could not open file : /data/dataset/h36m_extract/Images/S1/Smoking.55011271/000001.jpg
av_interleaved_write_frame(): Input/output error

Maybe just run python file train.py(python scripts/train.py --cfg experiments/h36m/train-ss.yaml) is wrong, because I did not find the dataloader use the function self_supervision in img_utils.py.
So how to fix it? Thank you!

Hello, I want to load the weights trained by integral human pose repository, on the repository you based your work on . but i will have to change the network configuration to do that. can you guide me how to change your configuration to that of the original repository?

Thanks for your work and the data preparation. Very thanks.

Now, I am re-training your Fully Supervised + resnet50 on H36M, with pre-trained weight (MPII Integral | resnet50 | 88.5 ), using the following codes:
python scripts/train.py --cfg experiments/h36m/train.yaml

Here, I have some questions.
The first one is what is the meaning of 'OCCLUSION' in train.yaml. The original one is 'True'. But I can not train successfully. Thus, I change it to 'false', owing to the lack of related files. Could you tell its effect and whether use it in your provided model? In addition, would you update the related files?

The second one is the poor evaluation result on the validation set. At the first epoch, the value of Validation-hm36_17j is 77.43, 83.9 89.3 (I trained this model three times). At the fifth epoch, the value of that increases to 392.8, 349.4, 287.1. I want to know whether it is normal? Maybe I should wait a long time. Or there is something wrong about it? Just like the turn-off of OCCLUSION?

Thanks very much! :)

hi,
Thanks for your work.

could your share the process of transfering 'joints_3d_cam' coordinates to 'joints_3d' coordinates in your annotation files?

I have downloaded the Human3.6m videos, and want to extract the images as the instructions, but I could not access to the scripts. Could you share in other ways or just upload to the repository? Thanks

Hi mkocabas,

I found you pre-processing code not aligning the same way as in integral pose https://github.com/mks0601/Integral-Human-Pose-Regression-for-3D-Human-Pose-Estimation/blob/master/data/Human36M/Human36M.py#L54

They are trying to get the thorax from left shoulder and right shoulder.
However, in your case, https://github.com/mkocabas/EpipolarPose/blob/master/lib/dataset/JointIntegralDataset.py, you directly have thorax in your H36M set.

May I ask you preprocess in advance or just treat the thorax as Spine1(which is most near thorax) in H36M? Otherwise have you notice any big difference using different thorax handle strategy?

Best,
Zhe

Hi,

Thank you for providing such a great code! I have some questions in the pretrained models on MPII.

I just did a simple verification of the accuracy of pretrained models on MPII (i.e. mpii_integral.pth.tar), and the mAP is 67.3 which is far worse than the claimed 88.5 mAP. I attached the validation config I used for testing the model (I changed the postfix to .txt so that I can upload it to github; the content is the same as a .yaml file)
valid.txt

Can you help me figuring out where I did wrong? I am using python3.7+pytorch1.0+Ubuntu18, so I don't think it's a cudnn issue.

thank you very much!

Hi,

When do you plan to release the annotation preparation script for self-supervised training?

Thank you

Some pictures can't read.
OSError:Fail to read data/h36m/images/S9/Directions_1.54138969/000065.jpg
And I don't have the picturest that numbered 65.
the numbers of the picture is 1,6,11,16,21,26...

Great work Muhammed!!
I just wanted to know about the kind of ground-truth information that is being used during inference. I can see that the ground-truth depth of the root keypoint is used during back-projection, but what about the scale? In the paper, you have mentioned that the "model uses normalized poses as ground truth" for training, how do you recover the scale during inference?
Thanks!

Hi, mkocabas!
I read your paper recently, and I have some questions that why "We assume the first camera as the center of the coordinate system, which means R of the first camera is identity." I have some doubts about this word "identity", does it means "constant"? If not, can you explain it?
Thank you very much!

Can you use camera information like intrinsic parameters in fully supervised training, it seems that you do not use any camera information.

However, you say you use GT intrinsinc camera paremeters in fc-train, i do not found any camera parameters can be used in either training process or valid process.

Btw, i found there is no different between integral loss and fc-train in this paper. Am i right?

sorry， I have solved the problem.

In order to confirm whether the hardware conditions of my laboratory are sufficient, I would like to ask how long does it take to complete the training of the network, how many gpus have been used, and what is the configuration of the GPU.

Thanks for your excellent work! Can you release the PSS code?

Sadly the pretrained models aren't the right size except refiner.pth.tar which appears to be an untrained model now

model.load_state_dict(checkpoint, strict = False) File "C:\Python37\lib\site-packages\torch\nn\modules\module.py", line 1407, in load_state_dict self.__class__.__name__, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for DataParallel: size mismatch for module.layer1.0.conv1.weight: copying a param with shape torch.Size([64, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 1, 1]). size mismatch for module.layer1.1.conv1.weight: copying a param with shape torch.Size([64, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 256, 1, 1]). size mismatch for module.layer1.2.conv1.weight: copying a param with shape torch.Size([64, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 256, 1, 1]). size mismatch for module.layer2.0.conv1.weight: copying a param with shape torch.Size([128, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([128, 256, 1, 1]). size mismatch for module.layer2.0.downsample.0.weight: copying a param with shape torch.Size([128, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([512, 256, 1, 1]). size mismatch for module.layer2.0.downsample.1.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for module.layer2.0.downsample.1.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for module.layer2.0.downsample.1.running_mean: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for module.layer2.0.downsample.1.running_var: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for module.layer2.1.conv1.weight: copying a param with shape torch.Size([128, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([128, 512, 1, 1]). size mismatch for module.layer2.2.conv1.weight: copying a param with shape torch.Size([128, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([128, 512, 1, 1]). size mismatch for module.layer2.3.conv1.weight: copying a param with shape torch.Size([128, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([128, 512, 1, 1]). size mismatch for module.layer3.0.conv1.weight: copying a param with shape torch.Size([256, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([256, 512, 1, 1]). size mismatch for module.layer3.0.downsample.0.weight: copying a param with shape torch.Size([256, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([1024, 512, 1, 1]). size mismatch for module.layer3.0.downsample.1.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([1024]). size mismatch for module.layer3.0.downsample.1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([1024]). size mismatch for module.layer3.0.downsample.1.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([1024]). size mismatch for module.layer3.0.downsample.1.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([1024]). size mismatch for module.layer3.1.conv1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([256, 1024, 1, 1]). size mismatch for module.layer3.2.conv1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([256, 1024, 1, 1]). size mismatch for module.layer3.3.conv1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([256, 1024, 1, 1]). size mismatch for module.layer3.4.conv1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([256, 1024, 1, 1]). size mismatch for module.layer3.5.conv1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([256, 1024, 1, 1]). size mismatch for module.layer4.0.conv1.weight: copying a param with shape torch.Size([512, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([512, 1024, 1, 1]). size mismatch for module.layer4.0.downsample.0.weight: copying a param with shape torch.Size([512, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([2048, 1024, 1, 1]). size mismatch for module.layer4.0.downsample.1.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([2048]). size mismatch for module.layer4.0.downsample.1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([2048]). size mismatch for module.layer4.0.downsample.1.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([2048]). size mismatch for module.layer4.0.downsample.1.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([2048]). size mismatch for module.layer4.1.conv1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([512, 2048, 1, 1]). size mismatch for module.layer4.2.conv1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([512, 2048, 1, 1]). size mismatch for module.deconv_layers.0.weight: copying a param with shape torch.Size([512, 256, 4, 4]) from checkpoint, the shape in current model is torch.Size([2048, 256, 4, 4]).

code-
model = models.pose3d_resnet.get_pose_net(config, is_train=False)
gpus = [int(i) for i in config.GPUS.split(',')]
model = torch.nn.DataParallel(model, device_ids=gpus).cuda()
print('Created model...')

checkpoint = torch.load(config.MODEL.RESUME)
model.load_state_dict(checkpoint)
model.eval()
print('Loaded pretrained weights...')

error-
checkpoint = torch.load(config.MODEL.RESUME)
FileNotFoundError: [Errno 2] No such file or directory: 'models/h36m/fully_supervised.pth.tar'

what should i do plz help me.

dear,
I can not take that data. Can you send me that data?

dear,
When I run python scripts/train.py --cfg experiments/mpii/train.yaml, I met the error like below:

RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 7.93 GiB total capacity; 6.19 GiB already allocated; 340.00 MiB free; 551.04 MiB cached)

Note:
my PC is Hp Z440: Xeon E51650v3, 32G ddr4, GTX1070

i can see this error FileNotFoundError: [Errno 2] No such file or directory: 'data/VOC2012/Annotations'
and i know experiments/h36m/train-ss.yaml have Dataset: Voc : 'data/VOC2012'
then what should i need in data/VOC2012 folder?