foocker / lsp Goto Github PK

folloing the readme, I finished the steps below:

1. video preprocess
   1. change fps to 60
   2. crop video to (512, 512)
   3. translate video to images
   4. extract audio from croped video
   5. merge audio, images, or video to new video.

but the next step "1. using deca to extract useful information " confused me a lot
how to use deca to extract useful information?
which .py script is need to do this job?
Could someone help me?

when I train the model of Audio2Feature, the loss is very large,it reaches 1000 ,Is this normal?

hi
i get the error like this

warnings.warn(
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
creating the FLAME Decoder
please check model path: /data/lsp-main/otherlib/deca/data/deca_model.tar
Traceback (most recent call last):
File "/data/lsp-main/otherlib/deca/demos/lsp_data_generator.py", line 699, in
main(args)
File "/data/lsp-main/otherlib/deca/demos/lsp_data_generator.py", line 431, in main
deca = DECA(config=deca_cfg, device=device)
File "/data/lsp-main/otherlib/deca/decalib/deca.py", line 50, in init
self._setup_renderer(self.cfg.model)
File "/data/lsp-main/otherlib/deca/decalib/deca.py", line 54, in _setup_renderer
self.render = SRenderY(self.image_size, obj_filename=model_cfg.topology_path, uv_size=model_cfg.uv_size,
File "/data/lsp-main/otherlib/deca/decalib/utils/renderer.py", line 187, in init
verts, uvcoords, faces, uvfaces = load_obj(obj_filename)
File "/data/lsp-main/otherlib/deca/decalib/utils/util.py", line 162, in load_obj
with open(obj_filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/data/lsp-main/otherlib/deca/data/head_template.obj'

when i run python train.py

FileNotFoundError: [Errno 2] No such file or directory: '/root/lihu/lsp/otherlib/deca/hk_fake_8_14_LSP/label/mean_pts_3d_fixed_countour_nose_closemouth.npy'

1、模型支持什么尺寸的视频呢？是任何尺寸都可以吗？
2、random 4 img from imgs有什么作用呢？

In the data folder:

1. what images should I put inside g_sample?
2. What specific files should I put inside 'label' folder?
   (should I put the files that are left out from data folder in original LSP repo?: 3d_fit_data.npz shoulder_points3D.npy
   camera_intrinsic.npy tracked2D_normalized_pts_fix_contour.npy
   change_paras.npz tracked3D_normalized_pts_fix_contour_bk.npy
   mean_pts3d.npy tracked3D_normalized_pts_fix_contour.npy
   normalized_shoulder_points.npy)

File "train.py", line 25, in
main(cfg)
File "train.py", line 15, in main
trainer.fit()
File "/workspace/raid02/edu/LIHU/lsp/lib/solover/trainer.py", line 292, in fit
for step, batch in enumerate(tqdm(self.train_dataloader)):
File "/root/lihu/miniconda3/lib/python3.7/site-packages/tqdm/std.py", line 1195, in iter
for obj in iterable:
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in next
data = self._next_data()
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
return self.collate_fn(data)
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
return collate(batch, collate_fn_map=default_collate_fn_map)
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 143, in collate
return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility.
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 143, in
return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility.
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 120, in collate
return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)
File "/root/lihu/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 163, in collate_tensor_fn
return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [0, 512] at entry 0 and [268, 512] at entry 1

File "/root/lihu/miniconda3/envs/get_lspdata/lib/python3.6/imp.py", line 297, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'standard_rasterize_cuda'

how to solve this problem thanks

---------- Generator networks initialized -------------

initialize network with normal
/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/torch/functional.py:581: UserWarning: stft will soon require the return_complex parameter be given for real inputs, and will further require that return_complex=True in a future PyTorch release. (Triggered internally at /pytorch/aten/src/ATen/native/SpectralOps.cpp:639.)
normalized, onesided, return_complex)
0%| | 0/251 [00:00<?, ?it/s]Training epoch 0
0%| | 0/35 [00:00<?, ?it/s]
0%| | 0/251 [00:00<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 25, in
main(cfg)
File "train.py", line 15, in main
trainer.fit()
File "/media/yulun/12THD1/Wenqing_Projects/lsp/lib/solover/trainer.py", line 314, in fit
for step, batch in enumerate(tqdm(self.train_dataloader)):
File "/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/tqdm/std.py", line 1195, in iter
for obj in iterable:
File "/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 517, in next
data = self._next_data()
File "/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 83, in default_collate
return [default_collate(samples) for samples in transposed]
File "/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 83, in
return [default_collate(samples) for samples in transposed]
File "/home/wenqingwang/anaconda3/envs/lsp2/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [480, 512] at entry 0 and [0, 512] at entry 1

Hi, just to double check, is the video in your "Gallary" section the resulting video from your inference?

File "D:\Preject\DS_Project\lsp-main\lib\datasets\audiovisual_dataset.py", line 68, in init
self.pts3d_mean = np.load(os.path.join(self.data3d_label_dir, 'mean_pts_3d_fixed_countour_nose_closemouth.npy'))
File "C:\Users\Administrator.conda\envs\LSP\lib\site-packages\numpy\lib\npyio.py", line 416, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/home/yourname/lsp/data/hk_fake_8_14\label\mean_pts_3d_fixed_countour_nose_closemouth.npy'

感谢作者的奉献, 非常伟大。

我跟随指引训练了模型, 调节yaml中的参数, 得到了以下的结果

是我在哪个步骤做错了什么, 您有什么好的建议吗

Hi foocker, Thank you for the repository!

If possible, we would greatly appreciate it if you could let us know their order so that we can prepare the required training files. I'm a bit confused.

after bash run.sh
label file have these 3d_fit_data_224.npz 3d_fit_data.npz camera_extrinsics.npy cams.npy img_meta.npy landmarks2d_original.npy mean_pts_3d.npy tforms.npy file
then i python train.py

File "train.py", line 25, in
main(cfg)
File "train.py", line 15, in main
trainer.fit()
File "/workspace/raid02/edu/LIHU/lsp/lib/solover/trainer.py", line 280, in fit
self.prepra_data()
File "/workspace/raid02/edu/LIHU/lsp/lib/solover/trainer.py", line 275, in prepra_data
self.train_dataset = build_dataset(self.cfg)
File "/workspace/raid02/edu/LIHU/lsp/lib/datasets/build_datasets.py", line 7, in build_dataset
dataset = AudioVisualDataset(cfg)
File "/workspace/raid02/edu/LIHU/lsp/lib/datasets/audiovisual_dataset.py", line 68, in init
self.pts3d_mean = np.load(os.path.join(self.data3d_label_dir, 'mean_pts_3d_fixed_countour_nose_closemouth.npy'))
File "/root/lihu/miniconda3/envs/deca/lib/python3.7/site-packages/numpy/lib/npyio.py", line 417, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/root/lihu/lsp/data/hk_fake_8_14/label/mean_pts_3d_fixed_countour_nose_closemouth.npy'

大佬 can you tell me how to get this file

Thank you for sharing the code.

Can the code you provided run in real-time like LSP?

dynamic和static的shape不一致，concat好像也不行

大佬 我按照readme的过程训练完三个模型后为啥landmarks生成的图像是全黑的 是还有其他额外的步骤还是需要调整代码

when i bash run.sh ，until 200steps , the code is finsh ,no error ,but in label file only have camera_extrinsics.npy ，no other npy file

Hi @foocker , After running the pre-processes in order, I run sh run.sh , but it stops at step 200.

['./hk_fake_8_14_imgs/1.png', './hk_fake_8_14_imgs/2.png', './hk_fake_8_14_imgs/3.png', './hk_fake_8_14_imgs/4.png', './hk_fake_8_14_imgs/5.png', './hk_fake_8_14_imgs/6.png', './hk_fake_8_14_imgs/7.png', './hk_fake_8_14_imgs/8.png', './hk_fake_8_14_imgs/9.png', './hk_fake_8_14_imgs/10.png']
creating the FLAME Decoder
trained model found. load /content/lsp/otherlib/deca/data/deca_model.tar
  1% 200/17986 [00:59<1:27:43,  3.38it/s]

It would be appreciated if you could guide me.

Hi foocker, thanks for your repository, can you help me, I am running step 1 video_preprocess.py
1.video preprocessing
1. change fps to 60
2. crop video to (512, 512)
3. translate video to images
4. extract audio from cropped video
5. Merge audio, images, or video to new video.

but in the video_preprocess.py file

`if name == "main":
# vp = './hk_fake_38_half.mp4'
# vp = './croped_hk_fake_38_half_60fps.mp4'
# vp = './data/hk_fake_38/test_mouth_new.mp4'
# vp = './data/hk_fake_38/original_landmarks.mp4'
# vp = './data/hk_fake_38/landmarks3d_fixed_countour_nose.mp4'
# vp = './data/hk_fake_38/test_all_train_add_headpose.mp4'
croped_vp = './data/hk_fake_8_14/croped_half_hk_fake_8_14_60fps.mp4'
dst_dir = '/home/yourname/DECA/DECA/hk_fake_8_14_imgs'

# wav = './data/hk_fake_38/video_audio/audio.wav'
# wav = './data/hk_fake_38/video_audio/Mayun.wav'

# img2video('./imgs', './video/kanghui_5_10.avi')
# kp = parser_json('./kanghui_5_1.json')
# kp = parser_json('./kanghui_5_60fps_crop_512_1.json')
# print(kp)
# lucas_kanade_method_imgs('./kanghui_imgs_512', kp, save_kp_dir='./kp_save_kh')
# check_should2d('./kp_save_kh/shoulder_2D.npy', './kanghui_imgs_512')
# x = np.load('./kp_save_girl/shoulder_2D.npy')
# print(x.shape)

fan = FAN()
# croped_vp = fan.crop_video_letterbox(vp, face_head_scale=1.5)
# fan.video_read_message(croped_vp)
# result_file = fan.change_vd_fps(croped_vp)
# fan.video_read_message(result_file)
# fan.video_read_message(croped_vp)
fan.extract_imgs(croped_vp, dst_dir)

# fan.extract_audio(croped_vp)
# fan.merge_audio_video(vp, wav)
# merge_audio_video('noaudio_croped_hk_fake_38_half_60fps.mp4', 'croped_hk_fake_38_half_60fps.wav')`

It's all commented, can you help me with what step I should follow, thanks for your help.

Can you train an independent person?

Hi,@foocker Can you provide the original resource? I want to train follow your step.thank so much！！

Hello!
I find from lib.solover.trainer import Trainer in train.py.
But the project has no lib package.
Where to find lib package?

data
-- video_name
-- APC
--xx_APC_feature_xxx.npy
--checkpoints
--imgs
--candidates
--g_sample
--label
--3d_fit_data.npz
--mean_pts_3d.npy
--...
--video_audio
these npy data how to get

Is this normal?

hi,thank you for your work. when i run the demo.py ，i get the image. can you help me solve the problem?

when i bash run.sh
get this error
Traceback (most recent call last):
File "demos/lsp_data_generator.py", line 599, in
main(args)
File "demos/lsp_data_generator.py", line 394, in main
deca = DECA(config = deca_cfg, device=device)
File "/workspace/raid02/edu/LIHU/lsp/otherlib/deca/decalib/deca.py", line 49, in init
self._create_model(self.cfg.model)
File "/workspace/raid02/edu/LIHU/lsp/otherlib/deca/decalib/deca.py", line 82, in _create_model
self.flame = FLAME(model_cfg).to(self.device)
File "/workspace/raid02/edu/LIHU/lsp/otherlib/deca/decalib/models/FLAME.py", line 77, in init
lmk_embeddings = np.load(config.flame_lmk_embedding_path, allow_pickle=True, encoding='latin1')
File "/root/lihu/miniconda3/envs/lsp/lib/python3.8/site-packages/numpy/lib/npyio.py", line 405, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/raid02/edu/LIHU/lsp/otherlib/deca/data/landmark_embedding.npy'

how can i get this landmark_embedding.npy file

The error occurs when I run sh run.sh

FileNotFoundError: [Errno 2] No such file or directory: '/content/lsp/otherlib/deca/data/generic_model.pkl'

What is the process of creating the generic_model.pkl?

Thanks a lot