pinatafarms / dad-3dheads Goto Github PK

Thank you very much for your excellent work!
There are some trouble when I use the 'dad_3dheads_benchmark'. Could you please share the way to generate 'rotation_matrix' and '7_landmarks_3d' for all validation/test images?
Thanks again!

Thanks for your sharing!
We'll appreciate it if you could show us more details of the whole process of annotation(how to build a preliminary mesh before labeling and which modifier is used to optimise the mesh with blender).

Thank you very much for your excellent work and dataset for person face/head analysis. I found that the released DAD-3DHeads only contains dense 3D head mesh shape and two matrices. It does not give the head pose label of each head, such as Euler angles and rotation matrix. This makes it difficult to apply this dataset for head pose estimation task. But this dataset is obviously more comprehensive and challenging than BIWI and 300W-LP. It would be a huge pity if it could not be used to study head pose estimation. Hope you can build and release the DAD-3DHeads-V2 with at least head pose label~

Can i mapping 191 or 445 landmarks to corresponding IDs on FLAME mesh?

How do I build the custom training dataset? I use FaceBuilder for blender to annotate images,but I don't know how to generate the label json file.

Hi, my problem is that when I benchmark, the metrics do not match the expected results for most of the images in the validation set. For example, for the whole validation set when comparing the predictions with the ground truth I get the following results:
nme_reprojection: 41.0635
z5_accuracy: 0.820
pose_error: 0.82040

I don't know why this is happening, because if I draw the landmarks predicted by the model on the images, they are not so far away from the annotations.

Hi,

I encountered an error in the "train.py" and "flame_lightning_model.py" files. The error message I received is as follows:

"TypeError: 'TensorBoardLogger' object is not iterable."

I am currently unable to proceed with my work due to this issue, and I would greatly appreciate your guidance in resolving it. Could you kindly provide some insights on how I can fix this error?

great works!

Can you share checkpoint in others way? the model url is hard to download

Thanks for your excellent work. I have one question about the image preprocessing. Does it have a standard alignment process (e.g. face detection) before sending the test image to the network? I got bad results when the input images have lots of background area.

“./DAD-3DHeads/model_training/train/flame_lightning_model.py", line 247, in on_pretrain_routine_start
for logger in self.logger:
TypeError: 'TensorBoardLogger' object is not iterable

I think something is wrong!!!! the model can detect 68 landmarks but for more landmarks and mesh, generate wrong landmarks

Hello, Tetiana. I'm recently retraining the original project. Different from your implementation in the paper running with 1 RTX A6000, I retrained DAD-3DNet with 3 RTX 3090, but got inferior results with training at least 100 epochs. The batchsize on each GPU is 96. Thus, the total batchsize is 288, which is similar with your given batchsize 256.

Belows are my evaluation results on the DAD-3DHeads val-set.

• Your results (using the released model weight): {'nme_reprojection': 1.9804347670962774, 'z5_accuracy': 0.9569228042646247, 'chamfer': 2.7485585, 'pose_error': 0.12988624173861432}
• My results: {'nme_reprojection': 2.2882715349235956, 'z5_accuracy': 0.9563515915670164, 'chamfer': 2.8907034, 'pose_error': 0.13523285796228382}

Am I missing some important details? Or is it because of inconsistencies in multi-card training? Looking forward to your reply.

Hello,

Excuse me, flame_ mesh_ Is the faces.pt model available? Error reported at runtime!
FileNotFoundError: [Errno 2] No such file or directory: 'model_ training/model/static/flame_ mesh_ faces.pt'

thank you!

how to get true yaw, pitch, roll annotation from current dataset?

the demo script running stuck.

Thank you very much for your excellent work and dataset for person face/head analysis. The Euler angles I get using the rotation matrix are very different from the Euler angles in the predictions obtained in the demo.py, and I find that the predicted Euler angles look a little more accurate, but the Euler angles obtained by the rotation matrix are quite incorrect.
I would appreciate you being able to elaborate on how the Euler angle is calculated in the code, thank you very much.

Hello together,

After getting the dataset I want to try to run the train.py file but after running the code I get this error:

Error executing job with overrides: []
Traceback (most recent call last):
File "train.py", line 44, in run_experiment
train(config)
File "train.py", line 22, in train
dad3d_net = FlameLightningModel(model=model, config=config, train=train_dataset, val=val_dataset)
File "/homes/jonas/DAD-3DHeads/model_training/train/flame_lightning_model.py", line 72, in init
self.iou_metric = SoftIoUMetric(compute_on_step=True)
File "/homes/jonas/DAD-3DHeads/model_training/metrics/iou.py", line 44, in init
super().init(
File "/projects/envs/conda/jonas/envs/py3.8_new/lib/python3.8/site-packages/torchmetrics/metric.py", line 146, in init
raise ValueError(f"Unexpected keyword arguments: {', '.join(kwargs_)}")
ValueError: Unexpected keyword arguments: compute_on_step

these are my packages on my machine:
absl-py==2.0.0
aiohttp==3.8.6
aiosignal==1.3.1
albumentations==1.0.0
antlr4-python3-runtime==4.8
async-timeout==4.0.3
attrs==23.1.0
cachetools==5.3.2
certifi==2023.7.22
charset-normalizer==3.3.2
chumpy==0.70
coloredlogs==15.0.1
fire==0.5.0
frozenlist==1.4.0
fsspec==2023.10.0
google-auth==2.23.4
google-auth-oauthlib==1.0.0
grpcio==1.59.2
humanfriendly==10.0
hydra-core==1.1.0
idna==3.4
imageio==2.32.0
importlib-metadata==6.8.0
importlib-resources==6.1.1
joblib==1.3.2
lazy_loader==0.3
lightning-utilities==0.9.0
Markdown==3.5.1
MarkupSafe==2.1.3
multidict==6.0.4
networkx==3.1
numpy==1.22.0
oauthlib==3.2.2
omegaconf==2.1.2
opencv-python==4.8.1.78
opencv-python-headless==4.8.1.78
packaging==23.2
Pillow==10.0.1
protobuf==4.25.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pyDeprecate==0.3.2
pytorch-lightning==1.6.0
pytorch-ranger==0.1.1
pytorch-toolbelt==0.5.0
pytorchcv==0.0.65
PyWavelets==1.4.1
PyYAML==6.0.1
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.21.0
scikit-learn==1.3.2
scipy==1.10.1
six==1.16.0
smplx==0.1.26
tensorboard==2.14.0
tensorboard-data-server==0.7.2
termcolor==2.3.0
threadpoolctl==3.2.0
tifffile==2023.7.10
timm==0.4.5
torch==1.9.0
torch-optimizer==0.1.0
torchgeometry==0.1.2
torchmetrics==1.2.0
torchvision==0.10.0
tqdm==4.66.1
typing_extensions==4.8.0
urllib3==2.0.7
Werkzeug==3.0.1
yarl==1.9.2
zipp==3.17.0

I also tried on different devices but still go the same result above.

thanks in advance

Hello I'm interested in artificial intelligence face recognition and I'm high school student ,i've filled the google form for the dataset. I haven't gotten any reply yet . my email : [email protected]

need retraining?

Hi there,

Could you please share how you preprocessed ALFW2000 images to your format? From the paper:
"Given an image, we assume the head is detected, loosely cropped, and approximately centered"

Thanks.

Hi, I find that the loss weight of smoothL1 in code is 100 but 1 in paper. I want to know which one is correct.

Besides, you mention that your dataset has no augmentation but in the code, the bbox is offset randomly. I want to know whether you include this offset when the released model was trained.

Thanks,

I use fit_scan.py to generate my custom dataset.

after train 500 epoch, to test image, 2d vertices are close to ok, but 3d vertices are not correct at all.

What is the possible reason for this?

vertices unit is millimeters.
label json is as follows:
{
"vertices": [
[
0.04518344467567696,
0.17775802332478438,
-0.39643736129255214
],
[
0.048658336279237865,
0.1768529119593747,
-0.39521232999733014
],
[
0.04945889770001125,
0.17799078430798437,
-0.39566139249431254
],
[
0.04600878696145669,
0.17851796093268812,
-0.39655851551505694
],
[
0.049867298790216646,
0.18126493109592973,
-0.3838844547280212
],
[
0.05354209709941608,
0.18278256826344497,
-0.3882279945523791
],
........

],
"model_view_matrix": [
[
0.9792482433597487,
0.0477148473832626,
0.19696743694336385,
0.28946250677108765
],
[
-0.0029217173213029147,
0.9751122033902291,
-0.22169270255772225,
-0.300279438495636
],
[
-0.2026433849024943,
0.2165167063730747,
0.9550163215446188,
1.0343986749649048
],
[
0.0,
0.0,
0.0,
1.0
]
],
"projection_matrix": [
[
1373.6173986936753,
0.0,
-252.10671060735615,
0.0
],
[
0.0,
1372.9339350792657,
356.03684183339647,
0.0
],
[
0.0,
0.0,
1,
-0.2
],
[
0.0,
0.0,
1,
0.0
]
]
}

Thanks for your great work!
But I met this problem when I compiled the demo.py：

Traceback (most recent call last):
File "demo.py", line 60, in
Fire(demo)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire
target=component.name)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "demo.py", line 48, in demo
predictor = FaceMeshPredictor.dad_3dnet()
File "/cto_studio/vistring/zhangtianke_code/DAD-3DHeads/predictor.py", line 179, in dad_3dnet
download_model()
File "/cto_studio/vistring/zhangtianke_code/DAD-3DHeads/predictor.py", line 33, in download_model
os.path.join(os.path.expanduser("~"), ".dad_checkpoints", _FILENAME)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/urllib/request.py", line 248, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/urllib/request.py", line 1392, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/home/zhangtianke/anaconda3/envs/DAD-3DHeads/lib/python3.6/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

I can not visit the "https://media.pinatafarm.com/public/research/dad-3dheads/dad_3dheads.trcd"
Can you give me some files?

hi,
how to get UV map and texture map?

thank you!

I have attempted facebuilder with mid poly（4646 vertices）. But I cannot export 5023 vertices as FLAME topology .
Thanks a lot!

Hello,

after experimenting with the repo I attract attention the the generating 3d meshes are upside down. The landmarks and the head mesh seems quite right but the 3d obj mesh are always upside down and are little bit tilted is this intended or did I do a mistake?

You mention on the README that the metafiles are all structured the same. However, the val.json does not contain the attributes for the items. I though I had a corrupted file and redownloaded the dataset zip file, but it is still missing in the new download as well.

Are those attributes for the validation subset available for public?

Hope to hear from you soon.

Thanks for your effort in this repo and dataset !
I believe there is an error in the vertices of the 'right_cheek' - 218 (this is on the middle of the right ear) should be replaced with 2098, according to the symmetry with left_cheek.

In the Appendix, you metioned there was a full video of labeling process in the project webpage, but I can't find it. Can you share it?

Hi,
I've been able to get the demo working and noticed that it outputs an .obj mesh. That's great, now I would like to take the parts of the original image that produced the mesh and texture map the visible components onto the mesh. Is that possible? IIUC there are several steps required: extracting the unprojected image patch, generating UV texture coordinates, and then outputting those to a file format like .mtl.

If the data is already available and I just need to do some wiring, that's great- point me at the right data structures and I can go. If it's a harder problem than I thought, just let me know as well :)

Thanks for your excellent services!

I want to build new face dataset consisted of 445 landmarks, but I don't know how to use this tools for dense landmarks.

Can I use them? If I can, how can I use labeling tool?

hello, i've filled the google form twice for the dataset. yet I haven't gotten any reply. do i need to fill it again?? thank you!