anandpawara / real_time_image_animation Goto Github PK

View Code? Open in Web Editor NEW

3.2K 94.0 520.0 22.13 MB

The Project is real time application in opencv using first order model

License: GNU General Public License v3.0

Python 53.17% Jupyter Notebook 46.83%

real_time_image_animation's Introduction

Real time Image Animation

The Project is real time application in opencv using first order model

Steps to setup

Step 1: Create virtual environment

Python version : python v3.7.3 or higher

create virual environment : pip install virtualenv

activate virtual environment : virtualenv env

Step 2: Activate virtual environment

For windows : env/Script/activate

For Linux : source env/bin/activate

Step 3 : Install required modules

Install modules : pip install -r requirements.txt

Install pytorch and torchvision : pip install torch===1.0.0 torchvision===0.2.1 -f https://download.pytorch.org/whl/cu100/torch_stable.html

Step 4 : Download cascade file ,weights and model and save in folder named extract

gdown --id 1wCzJP1XJNB04vEORZvPjNz6drkXm5AUK The file is also availible via direct link on Google's Drive: https://drive.google.com/uc?id=1wCzJP1XJNB04vEORZvPjNz6drkXm5AUK

On Linux machine : unzip checkpoints.zip

If on windows platfrom unzip checkpoints.zip using unzipping software like 7zip.

Delete zip file : rm checkpoints.zip

Step 5 : Run the project

Run application from live camera : python image_animation.py -i path_to_input_file -c path_to_checkpoint

Example : python .\image_animation.py -i .\Inputs\Monalisa.png -c .\checkpoints\vox-cpk.pth.tar

Run application from video file : python image_animation.py -i path_to_input_file -c path_to_checkpoint -v path_to_video_file

Example : python .\image_animation.py -i .\Inputs\Monalisa.png -c .\checkpoints\vox-cpk.pth.tar -v .\video_input\test1.mp4

TODO:

Tkinter version

Need work on face alignments

Future plans adding deepfake voice and merging with video

Credits

@InProceedings{Siarohin_2019_NeurIPS,
  author={Siarohin, Aliaksandr and Lathuilière, Stéphane and Tulyakov, Sergey and Ricci, Elisa and Sebe, Nicu},
  title={First Order Motion Model for Image Animation},
  booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},
  month = {December},
  year = {2019},
  url = {https://github.com/AliaksandrSiarohin/first-order-model}
}

Original Project
- AliaksandrSiarohin
If you like this project give your support to original author of this project by giving github star to author's project
video explanation to the project
- Video explanation by original author
- Two min papers
try project on google colab
- youtube link
- link to colab version
For any valueable feedback feel free to contact me on linkedin

real_time_image_animation's People

Contributors

Stargazers

Watchers

Forkers

ml-algorhythms rauniyar01 digitiqman deeptiagl huguensjean frh02 tomarskt ismail-30 devarshi87 gbernal akhil08agrawal balajimcr cr4shov3rrid3 saikapalayam parasdhiman111 burakakrishna rasata sujitrulz satoshirobatofujimoto bygreencn stevenyesz pankhuri25 labimage rakshithracharya lyuji282 surendranaidu endpress maozhiqiang gpsbird jaedukseo aihill ashamsu12 peternara dejan-stankovic tspannhw murilo bssrdf ryanmcgary aleksimyllyoja jdefrancesco longjohncoder rskvazh tisma r-build dorucioclea jwoods1 avinash273 victor9000 abhijitdalavi colbygk prakharpatidar meikuam lesliesibbs sbrichardson arthur1234 utkarsh-shukla12 heroic wottan32 felixzhang7 deepchatterjeevns onceinlifetime hasmeed ravitale heisenberg01010 simonjuleseric2 tenfleques cy-dev-tex imvickykumar999 dr4ik saikunocoda jrinconcol hadryan 0x7an sycomix saikst kanchanbhale waseemdev serafim-end kakaxi6822 peregrin72 cprakashagr smhtkn dwsyoyo m1369 noobartint srinivasanbigdata ashahrior raveikant filipe-monteiro gchenfly priestd09 prashant-sk dhstone ktchani zuiwanting poojahagone321 hhy5277 kamyu104 9trocode tchigher

real_time_image_animation's Issues

Requirements.txt gives error on mac

Line 48 on Requirements.txt gives the following error on Mac:

ERROR: Could not find a version that satisfies the requirement pywin32==227 (from -r requirements.txt (line 48)) (from versions: none)
ERROR: No matching distribution found for pywin32==227 (from -r requirements.txt (line 48))

How to use in CPU?

I change the sentence 'cpu=False' to 'cpu=True' in Line 40 file 'image_anmation.py', and change the function 'load_checkpoints(config_path, checkpoint_path, cpu=False)' to 'load_checkpoints(config_path, checkpoint_path, cpu=True)' in Line 24 file 'demo.py'.
But the output video file, only 54KB, and also it can't play.
Can use in CPU? How?

如何转成onnx模型呢？

How to run?

(rt) root@vultr:~/AIGC/Real_Time_Image_Animation# python ./image_animation.py -i ./Inputs/Monalisa.png -c ./checkpoints/vox-cpk.pth.tar -v ./demo/videos/0.mp4
/root/AIGC/Real_Time_Image_Animation/demo.py:2: UserWarning: 
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was *originally* set to 'TkAgg' by the following code:
  File "./image_animation.py", line 4, in <module>
    from animate import normalize_kp
  File "/root/AIGC/Real_Time_Image_Animation/animate.py", line 7, in <module>
    from frames_dataset import PairedDataset
  File "/root/AIGC/Real_Time_Image_Animation/frames_dataset.py", line 2, in <module>
    from skimage import io, img_as_float32
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/io/__init__.py", line 15, in <module>
    reset_plugins()
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/io/manage_plugins.py", line 95, in reset_plugins
    _load_preferred_plugins()
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/io/manage_plugins.py", line 75, in _load_preferred_plugins
    _set_plugin(p_type, preferred_plugins['all'])
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/io/manage_plugins.py", line 87, in _set_plugin
    use_plugin(plugin, kind=plugin_type)
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/io/manage_plugins.py", line 258, in use_plugin
    _load(name)
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/io/manage_plugins.py", line 302, in _load
    fromlist=[modname])
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/io/_plugins/matplotlib_plugin.py", line 4, in <module>
    import matplotlib.pyplot as plt
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/matplotlib/pyplot.py", line 71, in <module>
    from matplotlib.backends import pylab_setup
  File "/root/miniconda3/envs/rt/lib/python3.7/site-packages/matplotlib/backends/__init__.py", line 16, in <module>
    line for line in traceback.format_stack()


  matplotlib.use('Agg')
[INFO] loading source image and checkpoint...
./image_animation.py:29: DeprecationWarning: Starting with ImageIO v3 the behavior of this function will switch to that of iio.v3.imread. To keep the current behavior (and make this warning disappear) use `import imageio.v2 as imageio` or call `imageio.v2.imread` directly.
  source_image = imageio.imread(source_path)
/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/transform/_warps.py:105: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
  warn("The default mode, 'constant', will be changed to 'reflect' in "
/root/miniconda3/envs/rt/lib/python3.7/site-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
  warn("Anti-aliasing will be enabled by default in skimage 0.15 to "
/root/AIGC/Real_Time_Image_Animation/demo.py:27: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(f)
/root/miniconda3/envs/rt/lib/python3.7/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2894.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[INFO] Loading video from the given path
/root/miniconda3/envs/rt/lib/python3.7/site-packages/torch/nn/functional.py:4216: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  "Default grid_sample and affine_grid behavior has changed "
/root/miniconda3/envs/rt/lib/python3.7/site-packages/torch/nn/functional.py:1960: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
  warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
: cannot connect to X server

First

why I can only get a 12frame/s output video?

I input a 25frame/s video, but the output is only 12frame/s. I don't know if I do anything wrong?

Output is really weird and unexpected

Lovely project but when trying to run it the output was unexpected.

I got some user warnings but similar to the ones in the jupyter notebook shared in the repo.

All the outputs look like this:

Any idea of what I did wrong?

add the reference code

since the main modules are almost the same with https://github.com/AliaksandrSiarohin/first-order-model
might consider to add the reference code in README.md?

Dependency error - ipykernel

Hi
I got this dependency error below.

(env) everton@tromso:~/Real_Time_Image_Animation$ python --version
Python 2.7.17
(env) everton@tromso:~/Real_Time_Image_Animation$ pip --version
pip 20.1.1 from /home/everton/Real_Time_Image_Animation/env/lib/python2.7/site-packages/pip (python 2.7)
(env) everton@tromso:~/Real_Time_Image_Animation$ pip install -r requirements.txt
...

ERROR: Could not find a version that satisfies the requirement ipykernel==5.2.1 (from -r requirements.txt (line 14)) (from versions: 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.2.0, 4.2.1, 4.2.2, 4.3.0, 4.3.1, 4.4.0, 4.4.1, 4.5.0, 4.5.1, 4.5.2, 4.6.0, 4.6.1, 4.7.0, 4.8.0, 4.8.1, 4.8.2, 4.9.0, 4.10.0, 4.10.1)
ERROR: No matching distribution found for ipykernel==5.2.1 (from -r requirements.txt (line 14))

issue AssertionError: Torch not compiled with CUDA enabled

I've faced this in OSx - System Version: macOS 10.14.6 (18G103)

File "/Users//projects/Real_Time_Image_Animation/env/lib/python3.7/site-packages/torch/cuda/init.py", line 161, in _lazy_init
_check_driver()
File "/Users//projects/Real_Time_Image_Animation/env/lib/python3.7/site-packages/torch/cuda/init.py", line 75, in _check_driver
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Is the any ability to use CPU, instead of trying to use my non-existent GPUs ???

Permission denied for weights download

Getting an error when downloading weight file:

(venv) kyles-mbp:Real_Time_Image_Animation kylemathewson$ gdown --id 1wCzJP1XJNB04vEORZvPjNz6drkXm5AUK
Permission denied: https://drive.google.com/uc?id=1wCzJP1XJNB04vEORZvPjNz6drkXm5AUK
Maybe you need to change permission over 'Anyone with the link'?

virtualenv 版本

virtualenv 20.24.7 requires importlib-metadata>=6.6; python_version < "3.8", but you have importlib-metadata 1.6.0 which is incompatible.

Illegal instruction (core dumped)

I'm using python 3.7.3, and I followed all the steps as mentioned, and it's giving "Illegal instruction (core dumped)" while running.

AMD GPU

Just to say that while I understand most people who are deep into this field know that CUDA won't work with AMD GPUs, not everyone doe (me included).

It might be worth popping into the readme if there's a way to use this code on a machine with an AMD GPU, or that there;s no point starting if that's what you've got.

the google drive file is not downloading

gdown --id 1wCzJP1XJNB04vEORZvPjNz6drkXm5AUK this command is not working and also the link of google drive is not also working if somebody downloaded it already then please send me the link /reply this thread . (very urgent)

centos install pywin32 error

Readme has a typo

Doesn't work with python 3.8.x. only with python 3.7.x

It stated in readme that python v3.7.3 or higher is required.
However, torch 1.0.0 works only with python up to 3.7.x
If your virtual enviroment is python 3.8.* torch installation will exit with an error if yoy follow instructions in readme.

how to reduce distortion in the target image

Hi,
Tried out this exciting project but on my system the target animated images show lot of distortion & deformation. Maybe I am doing something something wrong. Could the authors please provide guidance if anything can be done to reduce the distortion & deformation ?

-Thanks

什么问题

Link is down

The link to google drive could not be found.
(Step 4: Download cascade file ,weights and model and save in folder named extract)

Lots of runtime errors

hi
i always use python in django
And I do not have much information about artificial intelligence and image processing
And I installed all the modules and deleted the files you mentioned
But while running, I encountered these errors
I really need this project
please help
erros

PS D:\Programmer\python\Real_Time_Image_Animation-master> python .\image_animation.py -i .\Inputs\Monalisa.png -c .\checkpoints\vox-cpk.pth.tar
[INFO] loading source image and checkpoint...
C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\skimage\transform_warps.py:105: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
warn("The default mode, 'constant', will be changed to 'reflect' in "
C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\skimage\transform_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
warn("Anti-aliasing will be enabled by default in skimage 0.15 to "
D:\Programmer\python\Real_Time_Image_Animation-master\demo.py:27: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
Traceback (most recent call last):
File ".\image_animation.py", line 32, in
generator, kp_detector = load_checkpoints(config_path='config/vox-256.yaml', checkpoint_path=checkpoint_path)
File "D:\Programmer\python\Real_Time_Image_Animation-master\demo.py", line 32, in load_checkpoints
generator.cuda()
File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 491, in cuda
return self._apply(lambda t: t.cuda(device))
File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
module._apply(fn)
File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
module._apply(fn)
File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
module._apply(fn)
[Previous line repeated 3 more times]
File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 409, in _apply
param_applied = fn(param)
File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 491, in
return self.apply(lambda t: t.cuda(device))
File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda_init.py", line 164, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Checkpoint. Zip

How can i get the zip folder for 4th step

Consider PySimpleGUI?

Hiya

I saw you have tkinter as a to-do.

I am planning on using PySimpleGUI to add a front-end to your work.

I've done a YOLO demo, a colorizer and REALLY want to add your stuff. There are a number of OpenCV demos posted in the Demos section that shows how to integrate PySimpleGUI with OpenCV.

It's really easy to add OpenCV to a GUI using PySimpleGUI. Adding a camera is shown in both projects above and there's a 4-line version the demos too. It reads and displays the camera. My hope is that I can simply hook into your software and take the opencv output and route it to one of my windows and that's all that's required.

I've wanted to do something exactly like you've done for a long long time. Thank you for publishing it.

GPU dedicated memory

Could you please tell me the requirements of GPU.
for running first-order-model I use Geoforce 1080 6GB, it consumes 3.8GB of GPU memory.
what about your program...please tell me about your experience.