microsoft / deep3dfacereconstruction Goto Github PK

When computing landmark loss, should the coordinates/224? If don't, the loss would be a very big number. And the same issue is that, when computing photometric loss, should RGB/255?

Hi，thanks for your work, I am really interested !
Now I have implement training process according to your paper, but I found some problems during training:

1. Image-level loss is hard to converge，with which the loss is about 40~50 at last , leading the reconstruction result to be mean shape without real texture of input image. (I used skin mask to train, and I want to ask whether there exists any tricks to converge Image-level loss ?)
2. Land-mark loss and image-level loss is hard to balance, which means that if I increase the weight of Land-mark loss, which will lead reconstruction result to have accurate pose and expression but wrong face color. On the contrast, if the weight of image-level loss is increased, the reconstruction result can have accurate face color but wrong pose and expression. (I want to ask How to balance this two losses?)
3. The learning rate is also important during training, which Learning rate strategy is used to achieve good results ?
   Hope for your advice, thank you very much!

What is the format of the landmarks in the .txt file? I was expecting 5 pairs of x,y coordinates but in the example input, it is just 6 integers.

When I use the Python dlib on an image, I get the following points:
points[(715, 266), (667, 269), (551, 270), (599, 270), (637, 359)]
How would I convert this to the expected input format?

netural:

no netural:

maybe i am a chinese?

Hi, I was able to follow #20 and render 2d image after reconstruction. But why does it create the blue-greenish artifacts on the image?

Hey! I run your demo and the results are great!
But your demo only generates meshes without texture, and you mentioned in your readme file that you use tf_mesh_renderer as renderer during training and testing.
Could your kindly release some example code to show me how to render the results with tf_mesh_renderer, since there are not much documents about tf_mesh_renderer?
I'm just getting started with face reconstruction, appreciate your help!

Hi I tried running demo.py and getting this error
Traceback (most recent call last):
File "demo.py", line 76, in
demo()
File "demo.py", line 64, in demo
shape = np.squeeze(face_shape,[0])
File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 1388, in squeeze
return squeeze(axis=axis)
TypeError: 'list' object cannot be interpreted as an integer

Hi,
I have a question that whether the idBase, exBase and texBase you provide are already scaled with their standard deviation respectively (as said in chapter 3 of your paper).

And I am inplementing Regularization in 4.3 of your paper, can you tell me the specific number（or order of magnitude) of ||alpha||^2, ||beta||^2 and ||delta||^2 when the training converged.

Thanks!

When penalizing the texture map variance, are the color values range from 0 to 255 or normalized to 0-1?

I want to get a smaller face region from the 35709 facial vertices, where can I get some information about the index of these vertices?

Hi, Yu,
I am re-implementing this paper and I have finished the photometric loss, landmark loss and regularizations. I find that my network can not regress texture coef(delta) and easily regress illumination coef(gamma). In other word, when the network is fed with a black man, the network thinks that it is the scene illumination that too low not the raw face texture too dark.
I want to ask is this a normal phenomenon?
And are the results posted in your paper rendered with illumination or without illumination that the network estimated?
Thanks!

Hello! Great work!

I have a question: how to get face textures like in this image https://github.com/microsoft/Deep3DFaceReconstruction/blob/master/images/albedo.png?

You have made a 'facemodel_info.mat' in folder 'BFM'. And it contains the 68 landmarks' index. In your .mat, it's 'keypoints'. If I want to use these keypoints, should I minus 1 before use just like what you did to 'tri'.

Hello, thank you very much for sharing your code. I got the reconstructed result based on your code and got the image after the face rendering. How should I paste the rendered 2d image back to the original image?

Hi, can demo code generate face from an image set? Thanks

Hi! Thank you for open sourcing your code! I noticed in your paper that you reported Genova et al's average RMSE result on MICC to be ~1.78, however, Genova et al report (what i assume to be the same evaluation metric) to be ~1.50...is there a reason for this?

If we use Tensorflow version 2.x, it may report that "placeholder" is missed. And using Tensorflow version 1.x could solve this problem.

hi, thanks for sharing this code. when i read the function transferBFM09() in load_data.py.
the chosen indexes were based on BFM_front_idx.mat. why not use select_vertex_id.mat(35709, 1)??? and what is the difference between select_vertex_id.mat and BFM_front_idx.mat(35709,1)

hopes for you reply! thanks

Hi,

Many thanks for your work!

I have a question that why you don't do rotation alignment in image preprocessing, and how can I do that at the same time the model can work properly. I didn't understand how the POS function work, if you can tell me that is better.

Best regards

I have a txt of 68 landmarks. But in program, it need five landmarks. How can I choose these five landmark?

When I plot landmarks_2d on an image in input folder, it produces bad result. Why is this happening?

Hi:

thanks for your contribution.

After split coeff, we can have ruler angle and use it to calculate rotation matrix.

I am confused that, is the angles equals to (roll, pitch, yaw)?

@

where is Supplementary materials mentioned in the article

I just wanna make sure, when training R-Net, after 500k iters, what's the specific number of your losses?

Hello everybody, so im brand new to python, but very motivated to make deep3Dface reconstruction work so i spent two days downloading and installing all the packages, cloning the repositery, installing everything and puting good files in good folders, and then asked myself how the program is running, should i replace the images in the input folder with one of mine ? i don't get it and feel a bit lost, so i ran the demo.py like that and got this message before the window close:

2019-05-08 23:08:10.069653: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
reconstructing...
1
C:\Users\79\Desktop\neuron\Deep3DFaceReconstruction\Deep3DFaceReconstruction\preprocess_img.py:19: FutureWarning: rcond parameter will change to the default of machine precision times max(M, N) where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass rcond=None, to keep using the old, explicitly pass rcond=-1.
k,,,_ = np.linalg.lstsq(A,b)

then got another message but didnt had time to read or copy because windows closed directly.

any help is very welcome, thanks David

How to get the alignment between generated meshes and output images as described in the readme?

In your paper, the photometric loss is a l2 loss with sqrt operation. But when computing gradients, sqrt operation always cause NAN. So, I just want to make sure if the photometric loss is a l2 loss with sqrt? If it is, can you give some suggestion on how to deal with the NAN situation.

In the paper, you said "we use a ResNet-50 network [22] to regress these coefficients by modifying the last fully-connected layer to 239 neurons." , but when I debug your code, and I found the count of output coefficients of R-Net is 257 ? I think that's because the count of lighting coefficients, in your paper it should be 9, but in your code, it's 27. which is right?

thanks for your great projects. i have question about 2D image like Robust in readme.
how to convert the results to 2D image by face_projection and z_buffer?
thank you.

Hi,
I am inplementing landmark loss, I am confused about eqaution (3) in 4.1.2 of your paper. There are two questions.
<1> Is N always equal to 68, even if there are self-occlusion ?
<2> Is q_n and q'_n include z-component when calculating equation (3) ? (I know that you use [8] to detect 3D landmark to get (x,y,z) tuples).

Thanks!

I just want to ask a direct question and looking forward to getting a direct answer. Is your approach better than 2DASL?

In the definition of loss texture, one of the regularization losses, you have mentioned a pre-defined skin region covering cheek, nose, and forehead. And you told readers to see suppl. material. Where can I get this pre-defined skin region?

Hi，Do you retrain the 3DMMasSTN to get a new BFM_UV.mat (35709) file or just pick out 35709 points based on the select_vertex_id.mat from the raw BFM_UV.mat（53490）.

hi， the model can get uv texture like PRNet? thanks~

Hi, I have a question when I try this work. whether does it not work well for Chinese? The result don't like the person who is a Chinese which I input .

Hi

What's the scale and translation value for z-buffer if I want to align the mesh to image? For x,y, the rasterization can be done according to the process_img in preprocess_img.py. I cannot find something similar for z value. Thank you.

when I suppose my image to demo.py ,this happens
To use the future default and silence this warning we advise to pass rcond=None, to keep using the old, explicitly pass rcond=-1. k,_,_,_ = np.linalg.lstsq(A,b) Traceback (most recent call last): File "demo.py", line 76, in <module> demo() File "demo.py", line 55, in demo input_img,lm_new,transform_params = Preprocess(img,lm,lm3D) File "C:\Users\Utente\Desktop\1x\Deep3DFaceReconstruction-master\preprocess_img.py", line 65, in Preprocess img_new,lm_new = process_img(img,lm,t,s) File "C:\Users\Utente\Desktop\1x\Deep3DFaceReconstruction-master\preprocess_img.py", line 46, in process_img img = img[:,:,::-1] IndexError: too many indices for array
matteo.txt
this is the txt input file .

why returns this error?

Projection is a very important procedure since you use 2d info to do weakly supervisation. You set camera posion (0,0,10), and to get a suitable face image, you set fov 12.5936. Actually, it's not precise since each photo is taken at different distance. So, this may cause some shape error. What do you think about this issue, or do you have any solution like 'dynamic projection procedure'.

How to get the evaluation results of Facewarehouse and MICC Florence? And are there differences compared with other methods in evaluation process? Could you provide an evaluation code? Thank you very much~

Hi @YuDeng
Thanks for the library, Can you please tell me how to get the 68 landmarks in 3D instead of 2D ?
I see here:

Thanks.

你好，

请问一下渲染时候使用的相机视角宽度系数fov_y和3d到2d投影时相机focal系数是怎么计算的？因为现在我打算使用kaolin替换tf_mesh_render, 发现使用tf_mesh_render的fov_y系数会导致渲染的结果只看到鼻子。所以我想知道fov_y 和 相机的focal系数之间的关系，如果有相关资料的话，烦请告诉我

祝好！

Hi, thanks for your contribution.

And I have a question, in my own image set, how to get the 5 landmarks for every image?
(obtain by other face detect network like MTCNN??)

Hi, thanks for sharing this awesome work. Just got one question for the data loading code, it seems that you performed the unit conversion by dividing 1e5 for all the base and mean shape. I am wondering is it a normal practice to do that? Cause I see that the location of the vertex in 3dmm is usually large (in the order of 1e5), what is the unit for those coordinates in the 3dmm model?