cure-lab / smoothnet Goto Github PK

Sorry but I have another question.
I tried to map the groundtruth and predicted points to the original video in MuPoTS-3d dataset. And I found the position in the file
./data/poses/mupots_tposenetrefinenet_3D/groundtruth/mupots_gt_3D_test.npz is a normalized format (Human 3.6M format?), which is described in the paper as: "We compare them on the universal coordinates, where each person is rescaled according to the hip and has a normalized height".
Is there any way to recover absolute pose position or visualize it?

Hi! Nice work, I read the lib/models/smoothnet.py, but I didn't find the code of Motion-aware SmoothNet. So have you already released that code?

Dear author,

Thank you for your amazing work. There is something confuse me about your method.
In this image, you said that your model is trained on 3D position representations only, so if I use a SmoothNet for PARE model (SMPL result), should I input a 3D keypoints postions or 6D rotation matrix for the model?
thank you so much!

I tested human3.6M dataset on FCN estimator by using your testing guideline.

python train_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --dataset_name h36m --estimator fcn --body_representation 3D --slide_window_size 8

Predicted estimator results are the same as paper, including Accel, MPJPE, and PA-MPJPE. However SmoothNet results are different.

In the paper, Accel, MPJPE, and PA-MPJPE are 1.03, 52.72, and 40.92, respectively. In contrast, your checkpoint results are 1.94, 53.29, and 41.38.

Is there any setting that I need to change? Please suggest to me.

Hi, I have an issue when I run demo PARE with SmoothNet.
My command: ! python visualize_smoothnet.py --cfg configs/pw3d_spin_3D.yaml --checkpoint data/checkpoints/pw3d_spin_3D/checkpoints_8.pth.tar --dataset_name pw3d --estimator vibe --body_representation 3D --slide_window_size 32 --visualize_video_id 2 --output_video_path ./visualize

And Output cell gives me the error that "ImportError: Ground-truth data do not exist!"

Can you help solve this problem? Thank you

Seems very useful!
So if I want to refine a 3D keypoints sequences, which checkpoint is good generally?
And since it is trained on 3D positions, why it can be used on a rotation sequences (like SMPL) as well?

I had train a customize pose detector for 12 joints.
Is smoothnet works without retrain the weight or just to input the joints with coordinates like 0,0,0?

作者您好，了解到smoothnet是即插即用的，我想要将smoothnet直接用于我们当前的3D姿态估计预测结果，使用的是NTU数据集，但加载的是h36m_fcn_3D的cfg与模型，效果非常差。我想问一下，smoothnet是与网络模型还有数据集绑定的么，如果我想用smoothnet做3D姿态估计后处理优化，是不是需要重新在自己的数据集和网络上进行训练。

thank you for your work！I just wonder how to use it for actual usage

曾爱玲 大佬您好，针对2D pose滤波，根据图像大小归一化图像序列对应的pose到【-1，1】，smoonthNet滤波后反归一化，整体的棍图看着是平滑没那么抖动了，但是在原视频中可视化，发现贴合到人的骨骼效果并不理想，可以简单分析下可能的原因吗？感谢

Some errors occurred with training on the H36M dataset.
evaluate on dataset: h36m, estimator: hrnet, body representation: 2D
Traceback (most recent call last):
File "SmoothNet/train_smoothnet.py", line 118, in
main(cfg)
File "SmoothNet/train_smoothnet.py", line 105, in main
Trainer(train_dataloader=train_loader,
File "SmoothNet/lib/core/trainer.py", line 68, in run
performance = self.evaluate()
File "SmoothNet/lib/core/trainer.py", line 240, in evaluate
performance.append(self.evaluate_2d(dataset_index,present_dataset))
File "SmoothNet/lib/core/trainer.py", line 199, in evaluate_2d
eval_dict = evaluate_smoothnet_2D(self.model, self.test_dataloader[dataset_index],
TypeError: evaluate_smoothnet_2D() missing 1 required positional argument: 'dataset'

It's a cool job, and it's eye-catching, but can this only be estimated by human body posture? Is there any generalization ability for the 21-point key points of the hand?

Hi, SmoothNet is fantastic! And as the question metioned in the title, can i use SmoothNet for processing the 2D pose keypoints achieved by Google's Mediapipe?

Hello, I am trying to replocate the Motion-aware Smoothnet, but I meet a question about how to define angular velocity in axis-angle representation？

I found that the skeleton of PW3D_SPIN_3D was misplaced when I tried to visualize the pw3d_spin_3d pretrained model. I wonder if there may be some errors in the PW3D_SPIN_3D_JOINTS or PW3D_SPIN_3D_EDGES settings?
As follow:

Hi！Guys！
I like your work very much! However, I have some questions to ask you, that is whether the order and number of key points in the data set during training should be consistent with that during reasoning. If I have 18 key points during training, but only 17 key points during reasoning and the order is different, can this be generalized to my task ?

1)i have no idea how should i genrate data, i saw in redme ,you mentioned two things one is file & second ground truth , also file format is .npz , How should I generate this? what steps should I need to follow?
2) using this repo can I able to do custom yoga pose detection?

Which function should I call and what to input? Thanks.

https://developers.google.com/mediapipe/solutions/vision/hand_landmarker

HI., I saw there has an option, smpl 6d, why it only have 243 and 246

rotation matrix why not be 24 * 3 *3? what's this 6d stands for?

hi, does any pretrained model able to applied to ROMP?

This table is from your paper.

In the paper, you mentioned that "SmoothNet is trained with the pose outputs from SPIN [22]. We test its performance across multiple backbone networks."
But you didn't describe the experimental configuration. So which dataset are you using poses from? And what size is your sliding window here?
Also, did you evaluate the metrics using the test set of the three datasets? Because I run your eval code, but the results of vibe is different from this table.

(data from pw3d_vibe_smpl_test.npz and pw3d_gt_smpl_test.npz)(and the vibe results are not affected by smoothnet pre-trained models)

thank you for your work！I just wonder how to use it for online inference

Hello, the dataset you provided does not include the training set of RLE2D, 3D and VPose3D, can you upload the relevant dataset, thank you very much.

Thanks for your good job! It behaves well on many dataseets. But when I want to train my motion-aware SmoothNet, I meet the question that vibe geet different frames to gt? I noticed that your ASSIT++ data predicted and gt has the same dimension and frames, so how to deal the differernt shape between vibe predicted and gt? Were you delete the redundant frames ？ or other methods? Thanks for your good job again.

Hey, nice work, I have a question that why vibe result presented in your github is different from that of vibe's paper?

I wonder what's the functionality of slide_window_to_sequence(), it seems to average values single axis of the whole slide window. Why not use the single frame rather than the averaged frames for prediction?

Hi,

I'm trying to implement Motion-aware SmoothNet. And it's a little bit confusing about the $T^e$ . Should they output the same same number of $T^e$ or just like the number of input $T, T-1, T-2$?

Besides, for inference pipeline, which frames should be updated in the window? For example, in a 8 frames sequneces, should all frames be updated or just the middle frames?

Thanks.

When I wanna to add my h36m 2D dataset, I found "training_iter = min([len(self.train_dataloader[i]) for i in range(len(self.train_dataloader))])" in the lib.core.trainer.py -> line 100.

I wonder if you want to add different dataset, why use min() not sum()?

Because if min(), the training will take the little dataset only.

Hi, I tested the jitter performance by using serveral pretrained models after normalize the data.
But, the jitter performance is similiar or a little bit worse. Do you have some suggestions ?

Can I directly use the smoothnet to smooth the output of alphapose without any additional training? Thx

Hi, Ailing and Xuan,
Thanks for your great works! I have some questions:

1. The project Github page says, 'Due to the temporal-only network without spatial modelings, SmoothNet is trained on 3D position representations only, and can be tested on 2D, 3D, and 6D representations, respectively.', which means 3D, 2D, and SMPL models are trained with their corresponding 3D keypoints, right? e.g. PARE SMPL model is trained by PARE 3D pred and g.t. During the inference, we only need input PARE SMPL pred.

2. The prediction distribution of PARE is inconsistent with SPIN. Is it unreasonable to initialize PARE's Smoothnet by using SPIN's checkpoint？ See github page table "SMPL Results" row 5.

3. The code in lib/models/smoothnet looks like the base version of SmoothNet (Only position refinement, no velocity and acceleration). Where is the motion-aware SmoothNet?

I may have misunderstood your work because of limited time. I hope your reply.

您好，
如果我想在mmpose中使用smoothnet平滑2D姿态估计器得出的关键点，我应该怎么做呢？mmpose官方的api接口好像就适用于基于滤波器的方法。

I would like to ask what the parameter EVALUATE.ROOT_RELATIVE is used for. The default setting is True. And the predicted value of the network and gt becomes 0 after calculation with this parameter

How to restore the estimation of the smoothnet with the same format of detected, namely, restore the imgname and keypoints while not the visual videos.

Hi, I want to run SmoothNet on an arbitary video, but it seems like a ground truth for that video is required even for the inference phase, right?

I'm sorry to disturb you again. I tried Yolo_ Pose predicted the key points and sent the predicted key points to smoothnet. I used the 2D key point model based on hrnet. In my opinion, the jitter problem has been reduced, but a large position offset problem has been found.
as follow：

great work for 3D task!
I wonder to know how can I convert the model to onnx.

when i run it with smpl，

cmd: python eval_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --checkpoint data/checkpoints/h36m_fcn_3D/checkpoint_32.pth.tar --dataset_name aist --estimator tcmr --body_representation smpl --slide_window_size 32

Q：[WARN] Cannot find rule for <class 'lib.models.smoothnet.SmoothNet'>. Treat it as zero Macs and zero Params.
it is no effect for smoothing?
what happend ?

H36M_HRNET_2D_EDGES

should be [5, 6, 0],please check again

Loading dataset (0)......
#############################################################
You are loading the [training set] of dataset [h36m]
You are using pose esimator [fcn]
The type of the data is [3D]
The frame number is [1559752]
The sequence number is [600]
#############################################################
#############################################################
You are loading the [testing set] of dataset [h36m]
You are using pose esimator [fcn]
The type of the data is [3D]
The frame number is [543344]
The sequence number is [236]
#############################################################

The training process has been stuck in the data loading interface, would like to ask what is probably the problem

Hi, thanks for sharing your work. I have a standard HRNET W-48 model trained on my dataset for pose estimation. Can i use the smoothnet to that model to refine 2D keypoints or heatmap outputs?
Is this a temporal tracking model of keypoints?

None of these config file can using for loading according pretrained models.

A lot of mismatch....

There are some problems in trying to reproduce the pw3d-spin-smpl experiment.

• First, the key name in the ./data/poses/pw3d_spin_smpl/groundtruth is misspelled, and the “pose” is written as “psoe”.
• In addition, I tried to run the ./train_smoothnet.py with default setting from pw3d_ spin_ 3d.yaml to train smpl-based model using oneeuro filter. It is found that although the train loss and acceleration errors have significantly decreased during the training stage, other metrics have become larger like MPJPE, MPVPE and PAMPJPE. I wonder if there is an error in my hyper-parameter setting.
• Besides, does this repository consider adding the slerp filter function mentioned in the paper?

I would like to ask whether this method is feasible if it is used for radar target trajectory filtering, non-image data

Hello, I downloaded your pretrained model. When I tested and trained, there was a problem. data/checkpoints/pw3d_spin_3D/checkpoints_8.pth.tar is not a pretrained model!!!! Can you tell me how to solve it?