Torso not added to final output about ad-nerf HOT 18 CLOSED

I can confirm that it works by using a pretrained torso model, even if the torso in question is very different from the pretrained one.

In my example, I could not get a good initialization even after many tries. Do you maybe have an intuition or explanation on how likely/unlikely a good initialization is? Does it depend on how much of the frame is taken up by the torso?

Feel free to close this issue, as my question is more of a research one.

Hi, in my experience, I can get a good initialization with a single attempt in most cases. In all cases, a good initialization is achieved with no more than 5 attempts.

For bad initialization, the PSNR does not show obvious improvement (usually below 20). For good initialization, the PSNR shows obvious improvement in the first 1000 iterations and usually above 21. I didn't notice the correlation with the torso part rate.

It's strange that you didn't get proper results even with PSNR greater than 25. However, you can always use the pretrained torso model for initialization.

from ad-nerf.

adnerfsub.mp4
@YudongGuo, the output that I generated using the weights that you shared has a lot of flickering between the head and torso. Could this be due to the difference in the tracking code that you mentioned? Or is there any other reason for this. Because I noticed that the demos that you shared do not have this noise.

Hi, this is the generated video with the tracking parameters of pretrained models. So it's because the inconsistence between tracking parameters.

from ad-nerf.

Hi, actually it's a problem of network initialization. We found that in few cases the torso model could not produce proper output due to bad initialization.

One solution is to run the command multiple times until good initialization (in my experience, the psnr is greater than 21 in 1000 iterations for a good initialization). Another solution is to use a pretrained Torso model (e.g. torso model trained from another person).

from ad-nerf.

Oh, ok, this is really useful to know. I see now that in my run PSNR was still below 20 after 10'000 iterations.

Thanks for your help.

from ad-nerf.

I tried several times now with multiple attempts with PSNR over 25 and different input videos and it never worked. Could you maybe provide a pretrained torso model such that I can try if that makes it work/look at how the results are supposed to look?
(The HeadNeRF works very well!)

from ad-nerf.

Hi, the Torso model pretrained on the obama video is here:
https://www.dropbox.com/s/hy99o8jm3iyskur/500000_body.tar?dl=0

It should work.

from ad-nerf.

Thank you, I'll try it out today.

from ad-nerf.

I can confirm that it works by using a pretrained torso model, even if the torso in question is very different from the pretrained one.

In my example, I could not get a good initialization even after many tries. Do you maybe have an intuition or explanation on how likely/unlikely a good initialization is? Does it depend on how much of the frame is taken up by the torso?

Feel free to close this issue, as my question is more of a research one.

from ad-nerf.

Hi @YudongGuo, What is the expected value of PSNR for the training of HeadNeRF? I started a training iteration and completed about 200k iterations but my PSNR value is still at around 19. I used all default settings and commands given in Readme.

from ad-nerf.

Hi @YudongGuo, What is the expected value of PSNR for the training of HeadNeRF? I started a training iteration and completed about 200k iterations but my PSNR value is still at around 19. I used all default settings and commands given in Readme.

Hi, expected PSNR is over 30. And in rare cases (with lower rate than TorsoNeRF) HeadNeRF suffers from bad initialization too. So the advices for training TorsoNeRF above also apply to HeadNeRF.

from ad-nerf.

Hi @YudongGuo, What is the expected value of PSNR for the training of HeadNeRF? I started a training iteration and completed about 200k iterations but my PSNR value is still at around 19. I used all default settings and commands given in Readme.

Hi, expected PSNR is over 30. And in rare cases (with lower rate than TorsoNeRF) HeadNeRF suffers from bad initialization too. So the advices for training TorsoNeRF above also apply to HeadNeRF.

@YudongGuo, could you please provide the HeadNeRF pre-trained weights for the Obama video as well?

from ad-nerf.

Hi @saslamsameja, I have uploaded both pretrained models in pretrained_models/ folder. Note that the code of face tracking for training the uploaded models is slightly different from this repo.

from ad-nerf.

Thanks, @YudongGuo. I have two questions:

• Even though the code for face tracking is different but using these weights for initialization will fix the problem of training, right?
• And do you plan on publishing the changes related to the face tracking as well?

from ad-nerf.

I got a PSNR of around 30 on the 2nd iteration of HeadNerf training, without loading the pre-trained weights. Thanks, @YudongGuo.

from ad-nerf.

Thanks, @YudongGuo. I have two questions:

• Even though the code for face tracking is different but using these weights for initialization will fix the problem of training, right?
• And do you plan on publishing the changes related to the face tracking as well?

Hi, the weights could be used for good initialization. And there were slight differences in differentiable rendering settings and tracking smoothness settings (in general the tracking code in this repo is better).

from ad-nerf.

@YudongGuo, the output that I generated using the weights that you shared has a lot of flickering between the head and torso. Could this be due to the difference in the tracking code that you mentioned? Or is there any other reason for this. Because I noticed that the demos that you shared do not have this noise.

from ad-nerf.

adnerfsub.mp4
@YudongGuo, the output that I generated using the weights that you shared has a lot of flickering between the head and torso. Could this be due to the difference in the tracking code that you mentioned? Or is there any other reason for this. Because I noticed that the demos that you shared do not have this noise.

I think it's because the difference between the tracking parameters. You can finetune the models with current parameters, and I will check this part too.

from ad-nerf.

I also got the same problem with another identity.
Both trainings from scratch and from Obama are tried.
Can I get any advice to converge?

from ad-nerf.