Hello,

How can I use video2template to create a good driving video?

Thanks in advance.

Feature request to select multiple people or a specific person within a picture would be cool

This is an amazing project, awesome work.
I have two issues:

1. My character doesn't close her eyes.
2. I have a wolf character, and there is an error with face detection. it doesn't work with animals?
   

Missing inference code of "Portrait video editing with stitching". Can anyone provide it?

I managed to remove wobble of shoulders by masking driving vid so only the head is visible this had a great effect.

The head on results from certain driving vids can have bad wobble caused by the head enlarging, I imagine this is due to a bad driving vid affecting the alignment crop radius, on another project I was able to set a persistent crop radius based on the most common value that worked rather well what are your thoughts on resolving this?

I don't have the driving vid for this or the settings so if anybody has all the settings checked and the concanated file and a noticable head wobble then please upload

I would like to use the pet effect feature. Do you have plans to make it available?

face detector not detecting animals such as dogs and cats although project pages mentions it might be possible

It really worked. No post processing required. Loved it.

Thanks to all

Jianzhu Guo 1†   Dingyun Zhang 1,2   Xiaoqiang Liu 1   Zhizhou Zhong 1,3   Yuan Zhang 1 
Pengfei Wan 1   Di Zhang 1

Would love to see the audio from reference video is stitched in output video

🐱🐶 Generalization to Animals 🐼🐱
By fine-tuning on animals data, these cute cats, dogs and pandas can be precisely driven by humans

I'm interested in the animal models, but I tried the current code and it says no faces were found. Can you update the animal/pet code to support that? Thanks

Everything was working fine before the update.

D:\AI\LivePortrait\venv\lib\site-packages\torch\nn\modules\conv.py:605: UserWarning: Plan failed with a cudnnException:
CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED
(Triggered internally at ..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
return F.conv3d(
Animating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:02
Concatenating result... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Dump to animations/1111111111111111111--Gradio_3_concat.mp4

Dump to animations/1111111111111111111--Gradio_3.mp4

Traceback (most recent call last):
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\queueing.py", line 532, in process_events
response = await route_utils.call_process_api(
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\blocks.py", line 1928, in process_api
result = await self.call_function(
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\blocks.py", line 1514, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\AI\LivePortrait\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "D:\AI\LivePortrait\venv\lib\site-packages\anyio_backends_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
File "D:\AI\LivePortrait\venv\lib\site-packages\anyio_backends_asyncio.py", line 859, in run
result = context.run(func, *args)
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\utils.py", line 832, in wrapper
response = f(*args, **kwargs)
File "D:\AI\LivePortrait\src\gradio_pipeline.py", line 64, in execute_video
gr.Info("Run successfully!", duration=2)
TypeError: Info() got an unexpected keyword argument 'duration'

Is there a plan to include the sound in the driving video to be included in the Generated Video?
It would be nice if we can have talking avatars

To the developers; GOOD JOB applause emoji

Great work! Not really an issue but was curious if there was a way to get the torso to move a little more naturally within a portrait shot.

伟大的工作，效果很好。
https://github.com/zhaoyun0071/LivePortrait-windows-GUI

V1.2 版本：2024-7-7
1、现在驱动视频支持自动裁剪人脸区域了，默认勾选自动裁剪视频人脸区域即可；

2、修复16代显卡黑图的问题，不勾选半精度即可；

3、生成的视频和驱动视频帧率抱持一致，且带有驱动视频的声音；

4、支持pkl格式的驱动视频导入，pkl格式就是勾选自动裁剪视频人脸区域后自动生成的数据文件，和驱动视频同名同目录。

V1.1 版本：2024-7-5

1、支持批量选择图片和驱动视频。批量生成。

V1.0 版本：2024-7-4

1、基于LivePortrait项目，原汁原味，支持图片模拟驱动视频的表情、五官动作等。

An error was caused by the missing file: src\utils\resources\mask_template.png

Great work! Is it possible to inference on videos? It seems possible based on the demonstration page I viewed.

Excellent work, congratulations!

I wonder if there is an option to remove the head motion completely. In my test I was driving it with a series of still
images and the head started shaking and changing its size.

Thanks in advance

Hey, you guys are just incredibly awesome! 🌟 This is hands down the best project I've come across recently. The test results are flawlessly perfect! 📊💯 I checked out the sections on your homepage about 'Portrait video editing with stitching' and 'Generalization to Animals'. 🐶🎨 Are there any release plans for these features

https://github.com/shadowcz007/comfyui-liveportrait

NVIDIA GeForce GTX 1660 Ti 跑出来的是这样的，是啥情况

as the title describes, can u explain how to make a driving video like the samples in ./assets/examples/driving

我承认可灵刚出的时候说话大声了，现在又来个这个！！！！马上测试！！！！！！！

very nice work but need something simple but big
when human move head the body make move little bit that change will make the model number one

bravo keep going....

I'm trying to create a colab notebook for the gradio version of the project. I have all the pieces together but for some reason when I run the final cell "python app.py" , the cell seem to get stuck and doesn't execute the rest of the script. No error pops up but a Gradio link never generates. Would anyone happen to have an idea what might cause this. Any help would be appreciated. Link to the colab - https://colab.research.google.com/drive/15pX7d8FfCMTty1v0DpgeApIR4VCduyDR?usp=sharing

Thank you for the interesting work.

I found that you used KVQ (CVPR 2023) to filter out the low-qaulity frames, and you are the author of KVQ.
Any chance to access KVQ ? or any other options such as HyperIQA ?

How can I crop a video to prepare it for inference tasks?

Im using Kijai implementation in Comfyui it would be good to mask are of the image to transfer head movement this will be good for hair, hat, helmet, etc.. here is a video example wen the tech only moves the face area and it supposed to move the helmet too.

Good work!

(LivePortrait) PS J:\githubs\LivePortrait> python .\inference.py
ERROR:albumentations.check_version:Error fetching version info
Traceback (most recent call last):
File "I:\anaconda3\envs\LivePortrait\lib\site-packages\albumentations\check_version.py", line 32, in fetch_version_info
data = response.read()
File "I:\anaconda3\envs\LivePortrait\lib\http\client.py", line 476, in read
s = self.safe_read(self.length)
File "I:\anaconda3\envs\LivePortrait\lib\http\client.py", line 626, in safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "I:\anaconda3\envs\LivePortrait\lib\socket.py", line 704, in readinto
return self.sock.recv_into(b)
File "I:\anaconda3\envs\LivePortrait\lib\ssl.py", line 1275, in recv_into
return self.read(nbytes, buffer)
File "I:\anaconda3\envs\LivePortrait\lib\ssl.py", line 1133, in read
return self.sslobj.read(len, buffer)
socket.timeout: The read operation timed out
Traceback (most recent call last):
File "J:\githubs\LivePortrait\inference.py", line 7, in
from src.live_portrait_pipeline import LivePortraitPipeline
File "J:\githubs\LivePortrait\src\live_portrait_pipeline.py", line 20, in
from .utils.cropper import Cropper
File "J:\githubs\LivePortrait\src\utils\cropper.py", line 10, in
from .face_analysis_diy import FaceAnalysisDIY
File "J:\githubs\LivePortrait\src\utils\face_analysis_diy.py", line 9, in
from .dependencies.insightface.app import FaceAnalysis
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface_init.py", line 18, in
from . import app
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\app_init.py", line 2, in
from .mask_renderer import *
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\app\mask_renderer.py", line 8, in
from ..thirdparty import face3d
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d_init.py", line 3, in
from . import mesh
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d\mesh_init.py", line 9, in
from .cython import mesh_core_cython
ImportError: cannot import name 'mesh_core_cython' from 'src.utils.dependencies.insightface.thirdparty.face3d.mesh.cython' (unknown location)

anyone has this error ? how to fixed ?

I followed the setup stages in the README (pip installations and downloading models from the Drive folder), and then tried to run inference.py but got this error message :

[07:53:17] Load appearance_feature_extractor done.
live_portrait_wrapper.py:29
Traceback (most recent call last):
  File "/efs/user_folders/yossib/Dev/LivePortrait/inference.py", line 33, in <module>
    main()
  File "/efs/user_folders/yossib/Dev/LivePortrait/inference.py", line 23, in main
    live_portrait_pipeline = LivePortraitPipeline(
  File "/efs/user_folders/yossib/Dev/LivePortrait/src/live_portrait_pipeline.py", line 38, in __init__
    self.live_portrait_wrapper: LivePortraitWrapper = LivePortraitWrapper(cfg=inference_cfg)
  File "/efs/user_folders/yossib/Dev/LivePortrait/src/live_portrait_wrapper.py", line 31, in __init__
    self.motion_extractor = load_model(cfg.checkpoint_M, model_config, cfg.device_id, 'motion_extractor')
  File "/efs/user_folders/yossib/Dev/LivePortrait/src/utils/helper.py", line 132, in load_model
    model.load_state_dict(torch.load(ckpt_path, map_location=lambda storage, loc: storage))
  File "/opt/conda/envs/LivePortrait/lib/python3.9/site-packages/torch/serialization.py", line 1004, in load
    with _open_zipfile_reader(opened_file) as opened_zipfile:
  File "/opt/conda/envs/LivePortrait/lib/python3.9/site-packages/torch/serialization.py", line 456, in __init__
    super().__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

This is how my models folder looks like :

(LivePortrait) ~/Dev/LivePortrait$ find pretrained_weights/
pretrained_weights/
pretrained_weights/insightface
pretrained_weights/insightface/models
pretrained_weights/insightface/models/buffalo_l
pretrained_weights/insightface/models/buffalo_l/det_10g.onnx
pretrained_weights/insightface/models/buffalo_l/2d106det.onnx
pretrained_weights/liveportrait
pretrained_weights/liveportrait/base_models
pretrained_weights/liveportrait/base_models/appearance_feature_extractor.pth
pretrained_weights/liveportrait/base_models/spade_generator.pth
pretrained_weights/liveportrait/base_models/motion_extractor.pth
pretrained_weights/liveportrait/base_models/warping_module.pth
pretrained_weights/liveportrait/landmark.onnx
pretrained_weights/liveportrait/retargeting_models
pretrained_weights/liveportrait/retargeting_models/stitching_retargeting_module.pth
pretrained_weights/.gitkeep

First of all, thank you for this great work. When I tried it in my pixel art-based images, it produced very successful results. However, I have one question: Is it possible to make it not apply filtering during resizing and perform a nearest neighbor style resize? I changed the interpolation in the cv2 resize parts, but the result did not change. I think the model is doing this smoothing internally during resize. Do you have any suggestions for this?

步骤 1：安装Flask
首先，你需要安装Flask。可以通过pip来安装：
pip install Flask
步骤 2：创建Flask应用
创建一个新的Python脚本，例如app1.py，并添加以下代码：
`from flask import Flask, request, jsonify
import subprocess
import os

app = Flask(name)

@app.route('/run-inference', methods=['POST'])
def run_inference():
# 获取JSON请求中的数据
data = request.json
source_image = data.get('source_image')
driving_video = data.get('driving_video')

if not source_image or not driving_video:
    return jsonify({"error": "Missing source_image or driving_video parameter"}), 400

# 构建命令
command = f'python inference.py -s {source_image} -d {driving_video}'

# 运行命令
try:
    subprocess.run(command, check=True, shell=True, cwd='C:/ALLWEBUI/liveportarit/LivePortrait')
    return jsonify({"message": "Inference completed successfully"}), 200
except subprocess.CalledProcessError as e:
    return jsonify({"error": str(e)}), 500

if name == 'main':
app.run(host='0.0.0.0', port=5000)
步骤 3：运行Flask应用 在终端中运行Flask应用：python app1.py
`

步骤 4：通过API调用
现在，你可以通过API来调用这个命令。例如，使用curl命令：
`import requests

url = 'http://localhost:5000/run-inference'
data = {
'source_image': 'Your image file here',
'driving_video': 'assets/examples/driving/d13(1).mp4'
}

response = requests.post(url, json=data)

print(response.json())
`

How can I use it locally with CPU?

I have not been able to find parameter to run it from CPU and I receive the message: RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

Thanks

Hello,

I have noticed in the latest update in argument_configs.py, the gradio section, the Share is set to True. Shouldn't it be False?

Thanks!

Hello! The installation went well, but an error is displayed at startup:

ImportError: cannot import name 'mesh_core_cython' from 'src.utils.dependencies.insightface.thirdparty.face3d.mesh.cython' (unknown location)

How to fine tune, refer to the demonstration of fine-tuning animal faces in the paper

I've gone through the steps on the main page to install. Setup environment, install requirements and gradio as well as the models. I get the same error whether I use a conda env or python one.
When running python inference.py or python app.py I get the following:

(venv) I:\zzzz_stablediffusion-other\zz_LP\LivePortrait>python app.py
INFO:albumentations.check_version:A new version of Albumentations is available: 1.4.11 (you have 1.4.10). Upgrade using: pip install --upgrade albumentations
Traceback (most recent call last):
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\app.py", line 11, in <module>
    from src.gradio_pipeline import GradioPipeline
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\gradio_pipeline.py", line 8, in <module>
    from .live_portrait_pipeline import LivePortraitPipeline
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\live_portrait_pipeline.py", line 20, in <module>
    from .utils.cropper import Cropper
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\cropper.py", line 11, in <module>
    from .face_analysis_diy import FaceAnalysisDIY
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\face_analysis_diy.py", line 9, in <module>
    from .dependencies.insightface.app import FaceAnalysis
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\__init__.py", line 18, in <module>
    from . import app
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\app\__init__.py", line 2, in <module>
    from .mask_renderer import *
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\app\mask_renderer.py", line 8, in <module>
    from ..thirdparty import face3d
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d\__init__.py", line 3, in <module>
    from . import mesh
  File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d\mesh\__init__.py", line 9, in <module>
    from .cython import mesh_core_cython
ImportError: cannot import name 'mesh_core_cython' from 'src.utils.dependencies.insightface.thirdparty.face3d.mesh.cython' (unknown location)

Great Work!!! Could you give more instructions on how to reproduce the demos in the project page.

I am getting the following when performing an animate. I have done a 'git pull' to make sure I have latest files and re-ran 'pip install -r requirements.txt' after doing so.

I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\venv\lib\site-packages\torch\nn
\modules\conv.py:605: UserWarning: Plan failed with a cudnnException:
CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed
cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at
..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
  return F.conv3d(
I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\venv\lib\site-packages\torch\nn
\modules\conv.py:456: UserWarning: Plan failed with a cudnnException:
CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed
cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at
..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
  return F.conv2d(input, weight, bias, self.stride,
Animating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:29
Concatenating result... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Dump to animations/s10--d5_concat.mp4

writing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--[swscaler @ 000001adabad0000] Warning: data is not aligned! This can lead to a speed loss
Dump to animations/s10--d5.mp4

re-targeting in the gradio seems to show correct results in the example image it produces but does not change the video

hi,gays,
(D:\work\Git\LivePortrait\LivePortrait) D:\work\Git\LivePortrait>python app.py
[09:06:26] Load appearance_feature_extractor done. live_portrait_wrapper.py:29
Load motion_extractor done. live_portrait_wrapper.py:32
Load warping_module done. live_portrait_wrapper.py:35
[09:06:27] Load spade_generator done. live_portrait_wrapper.py:38
Load stitching_retargeting_module done. live_portrait_wrapper.py:42
[09:06:31] LandmarkRunner warmup time: 1.211s landmark_runner.py:89
Traceback (most recent call last):
File "D:\work\Git\LivePortrait\app.py", line 28, in
gradio_pipeline = GradioPipeline(
File "D:\work\Git\LivePortrait\src\gradio_pipeline.py", line 26, in init
super().init(inference_cfg, crop_cfg)
File "D:\work\Git\LivePortrait\src\live_portrait_pipeline.py", line 39, in init
self.cropper = Cropper(crop_cfg=crop_cfg)
File "D:\work\Git\LivePortrait\src\utils\cropper.py", line 44, in init
self.face_analysis_wrapper = FaceAnalysisDIY(
File "D:\work\Git\LivePortrait\src\utils\face_analysis_diy.py", line 37, in init
super().init(name=name, root=root, allowed_modules=allowed_modules, **kwargs)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\app\face_analysis.py", line 33, in init
model = model_zoo.get_model(onnx_file, **kwargs)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\model_zoo\model_zoo.py", line 96, in get_model
model = router.get_model(providers=providers, provider_options=provider_options)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\model_zoo\model_zoo.py", line 40, in get_model
session = PickableInferenceSession(self.onnx_file, **kwargs)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\model_zoo\model_zoo.py", line 25, in init
super().init(model_path, **kwargs)
File "D:\work\Git\LivePortrait\LivePortrait\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "D:\work\Git\LivePortrait\LivePortrait\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 472, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from D:\work\Git\LivePortrait\src\utils../../pretrained_weights/insightface\models\buffalo_l\det_10g.onnx failed:Protobuf parsing failed.

Just wondering if there is any hope of having this project be used to create talking avatar that is audio driven. Im having fun with this proect but it would be nice to have talking heads

I am working on a 1-click installers (all ready for Windows, RunPod, Massed Compute & Kaggle (works amazing on free Kaggle account)) and a tutorial video and flag_eye_retargeting is not working

I have made a lot of changes and implemented the logic but here comparison

Default settings as below

When target eyes-open ratio is 30% more of original and target lip-open ratio is 80% more as below

Thanks for the project ❤️ I made a jupyter notebook 🥳 I hope you like it.

https://github.com/camenduru/LivePortrait-jupyter

Excellent work guys! Loved reading the paper. Are there any plans to release the training code and/or the dataset?

pip install -r requirements.txt

ERROR: Ignored the following versions that require a different python version: 1.14.0 Requires-Python >=3.10; 1.14.0rc1 Requires-Python >=3.10; 1.14.0rc2 Requires-Python >=3.10
ERROR: Could not find a version that satisfies the requirement onnxruntime-gpu==1.18.0 (from versions: none)
ERROR: No matching distribution found for onnxruntime-gpu==1.18.0

that input image + input video (both auto cropped and processed) and generates output video

ty