williamyang1991 / fresco Goto Github PK

View Code? Open in Web Editor NEW

613.0 613.0 60.0 10.19 MB

[CVPR 2024] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation

License: Other

Python 5.85% Jupyter Notebook 89.07% MATLAB 0.01% Shell 0.15% Batchfile 0.03% C 3.03% C++ 1.35% Cuda 0.51%

controlnet diffusion video-processing

fresco's People

Contributors

Stargazers

Watchers

fresco's Issues

imageio version?

Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 1437, in process_api
result = await self.call_function(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 1109, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\utils.py", line 650, in wrapper
response = f(*args, **kwargs)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Desktop\FRESCO\webUI.py", line 312, in process1
writer.append_data(curImg)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\core\format.py", line 590, in append_data
return self._append_data(im, total_meta)
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\plugins\ffmpeg.py", line 587, in _append_data
self._initialize()
File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\plugins\ffmpeg.py", line 630, in _initialize
self._write_gen = self._ffmpeg_api.write_frames(
TypeError: write_frames() got an unexpected keyword argument 'audio_path'

python video_blend.py Cannot run

I ran the command below
"python video_blend.py ./output/music/ --key keys --key_ind 0 26 36 54 72 87 107 122 132 159 173 183 194 206 217 229 239 --output ./output/music/blend.mp4 --fps 24 --n_proc 4 -ps"
which is get by command "python run_fresco.py ./config/config_music.yaml"
But I get the error output below:

/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
self.stylized_imgs[i - 1]: ./output/music/out_194/0194.jpg
self.stylized_imgs[i - 1]: ./output/music/out_194/0195.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_194/0195.jpg'): can't open/read file: check file path/integrity
Process Process-4:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
self.stylized_imgs[i - 1]: ./output/music/out_72/0072.jpg
self.stylized_imgs[i - 1]: ./output/music/out_72/0073.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_72/0073.jpg'): can't open/read file: check file path/integrity
Process Process-2:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
self.stylized_imgs[i - 1]: ./output/music/out_0/0000.jpg
self.stylized_imgs[i - 1]: ./output/music/out_0/0001.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
self.stylized_imgs[i - 1]: ./output/music/out_132/0132.jpg
self.stylized_imgs[i - 1]: ./output/music/out_132/0133.jpg
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/output/music/out_132/0133.jpg'): can't open/read file: check file path/integrity
Process Process-3:
Traceback (most recent call last):
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 315, in bootstrap
self.run()
File "/alg_vepfs/public/miniconda_dirs/envs/chm/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 118, in process_sequences
process_one_sequence(i, video_sequence)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 104, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/blender/guide.py", line 100, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 9.930143356323242
run_ebsynth done!
video_sequence.get_key_img(i): ./output/music/keys/0000.png
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0012.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0013.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0014.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0015.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('./output/music/out_0/0016.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0017.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0018.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0019.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0020.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0021.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0022.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0023.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0024.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_0/0025.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0026.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0012.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0013.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0014.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0015.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0016.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0017.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0018.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0019.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0020.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0021.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0022.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0023.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0024.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/music/out_26/0025.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 319, in
main(args)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 279, in main
process_seq(video_sequence, i, blend_histogram, blend_gradient)
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 209, in process_seq
dist1s.append(load_error(bin_a, img_shape))
File "/alg_vepfs/private/panfayu/chm/xiaozaza/FRESCO/video_blend.py", line 168, in load_error
with open(bin_path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: './output/music/out_0/0001.bin'

I am extremely confused.....................
Since I get the output images by official command, i didn't get corresponding images as supposed.

Error in saving the video

Getting error on selecting Run All
using the given example dog video

python video_blend.py output/dog --key keys --key_ind 0 11 23 49 72 99  --output output/dog/blend.mp4 --fps 24 --n_proc 4 -ps
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspace/FRESCO/venv/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_49/0050.jpg'): can't open/read file: check file path/integrity
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_23/0024.jpg'): can't open/read file: check file path/integrity
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_72/0073.jpg'): can't open/read file: check file path/integrity
Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/FRESCO/video_blend.py", line 111, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/workspace/FRESCO/video_blend.py", line 98, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
  File "/workspace/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
  File "/workspace/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 118.52607154846191
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_0/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/dog/out_11/0010.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
  File "/workspace/FRESCO/video_blend.py", line 308, in <module>
    main(args)
  File "/workspace/FRESCO/video_blend.py", line 268, in main
    process_seq(video_sequence, i, blend_histogram, blend_gradient)
  File "/workspace/FRESCO/video_blend.py", line 200, in process_seq
    dist1s.append(load_error(bin_a, img_shape))
  File "/workspace/FRESCO/video_blend.py", line 161, in load_error
    with open(bin_path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'output/dog/out_0/0001.bin'
Traceback (most recent call last):
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/queueing.py", line 388, in call_prediction
    output = await route_utils.call_process_api(
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/route_utils.py", line 219, in call_process_api
    output = await app.get_blocks().process_api(
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1440, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1341, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/video.py", line 281, in postprocess
    processed_files = (self._format_video(y), None)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/video.py", line 355, in _format_video
    video = self.make_temp_copy_if_needed(video)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/base.py", line 226, in make_temp_copy_if_needed
    temp_dir = self.hash_file(file_path)
  File "/workspace/FRESCO/venv/lib/python3.10/site-packages/gradio/components/base.py", line 190, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/dog/blend.mp4'```

Got RuntimeError:: element 0 of tensors does not require grad and does not have a grad_fn

video to video translation...
processing batch [1/7] with 8 frames
FutureWarning: _encode_prompt() is deprecated and it will be removed in a future version. Use encode_prompt() instead. Also, be aware that the output format changed from a concatenated tensor to a tuple.

0%| | 0/15 [00:00<?, ?it/s]
0%| | 0/15 [00:00<?, ?it/s]
!!! Exception during processing !!!
Traceback (most recent call last):
File "/data/comfy-ui/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/data/comfy-ui/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/data/comfy-ui/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/nodes.py", line 68, in stylize
FrescoService().stylize(video_file_path,output_dir,prompt)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco_service.py", line 39, in stylize
keys = run_keyframe_translation(config)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/run_fresco.py", line 239, in run_keyframe_translation
latents = inference(pipe, controlnet, frescoProc,
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/pipe_FRESCO.py", line 202, in inference
noise_pred = pipe.unet(
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 778, in forward
sample = optimize_feature(sample, flows, occs, correlation_matrix,
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 487, in optimize_feature
optimizer.step(closure)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper
out = func(*args, **kwargs)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 76, in _use_grad
ret = func(self, *args, **kwargs)
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/adam.py", line 143, in step
loss = closure()
File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 480, in closure
loss.backward()
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

ImportError: cannot import name 'logger' from 'diffusers.models.unet_2d_condition' (/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py)

2024-03-23 09:12:10.007036: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-23 09:12:10.007276: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-23 09:12:10.144748: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-23 09:12:13.261790: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py:20: FutureWarning: UNet2DConditionOutput is deprecated and will be removed in version 0.29. Importing UNet2DConditionOutput from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionOutput, instead.
deprecate("UNet2DConditionOutput", "0.29", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py:25: FutureWarning: UNet2DConditionModel is deprecated and will be removed in version 0.29. Importing UNet2DConditionModel from diffusers.models.unet_2d_condition is deprecated and this will be removed in a future version. Please use from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel, instead.
deprecate("UNet2DConditionModel", "0.29", deprecation_message)
Traceback (most recent call last):
File "/content/FRESCO/webUI.py", line 25, in
from src.free_lunch_utils import apply_freeu
File "/content/FRESCO/src/free_lunch_utils.py", line 6, in
from diffusers.models.unet_2d_condition import logger as logger2d
ImportError: cannot import name 'logger' from 'diffusers.models.unet_2d_condition' (/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py)

No video output after run the script

Hi:
First of all, this project is great.

After run the script

python run_fresco.py ./config/config_music.yaml

Everythings seems going well but there is no video output.
Is it necessary to manually execute the script below in order to generate the video?

python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75  --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps

The full log list below:


(venv) G:\AI\FRESCO>python run_fresco.py ./config/config_music.yaml
====================================================================================================
loading configuration...
batch_size: 8
cond_scale: 0.7
controlnet_type: hed
end_opt_step: 15
file_path: ./data/car-turn.mp4
gmflow_path: ./model/gmflow_sintel-0c07dcb3.pth
max_process: 4
maxinterv: 5
mininterv: 5
num_inference_steps: 20
num_warmup_steps: 5
prompt: a red car turns in the winter
run_ebsynth: False
save_path: ./output/car-turn/
sd_path: SG161222/Realistic_Vision_V2.0
seed: 0
sod_path: ./model/epoch_resnet.pth
use_controlnet: True
use_freeu: False
use_salinecy: True

====================================================================================================
creating models...
create optical flow estimation model successfully!
create saliency detection model successfully!
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 920/920 [00:00<?, ?B/s]
diffusion_pytorch_model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.45G/1.45G [01:49<00:00, 13.2MB/s]
create controlnet model-hed successfully!
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 547/547 [00:00<?, ?B/s]
diffusion_pytorch_model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 335M/335M [00:26<00:00, 12.8MB/s]
model_index.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 577/577 [00:00<00:00, 576kB/s]
safety_checker\model.safetensors not found
scheduler/scheduler_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 341/341 [00:00<?, ?B/s]
text_encoder/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 612/612 [00:00<?, ?B/s]
(…)ature_extractor/preprocessor_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 518/518 [00:00<00:00, 518kB/s]
tokenizer/special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 472/472 [00:00<00:00, 472kB/s]]
safety_checker/config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.89k/4.89k [00:00<?, ?B/s]
unet/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 901/901 [00:00<?, ?B/s]
tokenizer/tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 806/806 [00:00<?, ?B/s]
vae/config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 548/548 [00:00<?, ?B/s]
tokenizer/merges.txt: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 793kB/s]
tokenizer/vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.06M/1.06M [00:00<00:00, 1.18MB/s]
text_encoder/pytorch_model.bin: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 492M/492M [00:48<00:00, 10.1MB/s]
safety_checker/pytorch_model.bin: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.22G/1.22G [01:38<00:00, 12.4MB/s]
unet/diffusion_pytorch_model.bin: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.44G/3.44G [04:35<00:00, 12.5MB/s]
Fetching 14 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [04:38<00:00, 19.86s/it]
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:03<00:00,  2.05it/s]
create diffusion model SG161222/Realistic_Vision_V2.0 successfully!

====================================================================================================
key frame selection for "./data/car-turn.mp4"...
processing 3 batches:
keyframe indexes [[0, 5, 10, 15, 20, 25, 30, 35], [40, 45, 50, 55, 60], [65, 70, 75]]

====================================================================================================
video to video translation...
processing batch [1/3] with 8 frames
G:\AI\FRESCO\venv\lib\site-packages\torch\functional.py:507: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:3550.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
G:\AI\FRESCO\venv\lib\site-packages\diffusers\models\attention_processor.py:1117: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
  hidden_states = F.scaled_dot_product_attention(
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [06:36<00:00, 26.40s/it]
processing batch [2/3] with 5 frames
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [03:35<00:00, 14.35s/it]
processing batch [3/3] with 3 frames
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:26<00:00,  1.80s/it]

====================================================================================================
to translate full video with ebsynth, install ebsynth and run:

python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps


====================================================================================================
Done

视频大于1M ，24GPU会爆吗？还是只有我是这样的？

Traceback (most recent call last):
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\blocks.py", line 1437, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\blocks.py", line 1109, in call_function
prediction = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\gradio\utils.py", line 650, in wrapper
response = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\webUI.py", line 159, in process
keypath = process1(*args)
^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\webUI.py", line 280, in process1
latents = inference(global_state.pipe, global_state.controlnet, global_state.frescoProc,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\pipe_FRESCO.py", line 201, in inference
noise_pred = pipe.unet(
^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\diffusion_hacked.py", line 787, in forward
sample = upsample_block(
^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\free_lunch_utils.py", line 346, in forward
hidden_states = attn(
^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\diffusers\models\transformer_2d.py", line 292, in forward
hidden_states = block(
^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\diffusers\models\attention.py", line 155, in forward
attn_output = self.attn1(
^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 322, in forward
return self.processor(
^^^^^^^^^^^^^^^
File "D:\python_project\Rerender_A_Video\src\diffusion_hacked.py", line 281, in call
query = F.scaled_dot_product_attention(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.66 GiB (GPU 0; 23.99 GiB total capacity; 15.29 GiB already allocated; 2.85 GiB free; 17.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Does it support LCM models?

....

3060 12G GPU can run normally, but the speed is too slow. I hope it can be optimized.

Number of frames in a batch 3

Error when running python webUI.py

(diffusers) PS D:\FRESCO> python webUI.py
safety_checker\model.safetensors not found
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]D:\miniconda3\envs\diffusers\lib\site-packages\torch_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:04<00:00, 1.55it/s]
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
safety_checker\model.safetensors not found
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]D:\miniconda3\envs\diffusers\lib\site-packages\torch_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Loading pipeline components...: 29%|██████████████▊ | 2/7 [00:03<00:08, 1.67s/it]D:\miniconda3\envs\diffusers\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
warnings.warn(
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:04<00:00, 1.54it/s]
Traceback (most recent call last):
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\blocks.py", line 1437, in process_api
result = await self.call_function(
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\blocks.py", line 1109, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\miniconda3\envs\diffusers\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "D:\miniconda3\envs\diffusers\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "D:\miniconda3\envs\diffusers\lib\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "D:\miniconda3\envs\diffusers\lib\site-packages\gradio\utils.py", line 650, in wrapper
response = f(*args, **kwargs)
File "D:\miniconda3\envs\diffusers\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "webUI.py", line 159, in process
keypath = process1(*args)
File "D:\miniconda3\envs\diffusers\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "webUI.py", line 172, in process1
global_state.update_sd_model(sd_model)
File "webUI.py", line 149, in update_sd_model
self.pipe.scheduler = DDPMScheduler.from_config(pipe.scheduler.config)
NameError: name 'pipe' is not defined

Meaning of unet_chunk_size in FRESCOAttnProcessor2_0 codes？

The 'video_len' in the source code is equal to 'batch_size // unet_chunk_size'. But if 8 frames are fed in at once, isn't 'batch_size' equal to 'video_len'?

        '''for efficient cross-frame attention'''
        if self.controller and self.controller.use_cfattn and (not crossattn):
            video_length = key.size()[0] // self.unet_chunk_size  # BC // C = B
            former_frame_index = [0] * video_length  # pervious frame index
            attn_mask = None  # get attn_mask from controller
            if self.controller.attn_mask is not None:
                for m in self.controller.attn_mask:
                    if m.shape[1] == key.shape[1]:
                        attn_mask = m
            # get key and value in former_frame_index range
            # BC * HW * 8D --> B * C * HW * 8D
            key = rearrange(key, "(b f) d c -> b f d c", f=video_length)
            # B * C * HW * 8D --> B * C * HW * 8D
            if attn_mask is None:
                key = key[:, former_frame_index]
            else:
                key = repeat(key[:, attn_mask], "b d c -> b f d c", f=video_length)
            # B * C * HW * 8D --> BC * HW * 8D 
            key = rearrange(key, "b f d c -> (b f) d c").detach()
            value = rearrange(value, "(b f) d c -> b f d c", f=video_length)
            if attn_mask is None:
                value = value[:, former_frame_index]
            else:
                value = repeat(value[:, attn_mask], "b d c -> b f d c", f=video_length)              
            value = rearrange(value, "b f d c -> (b f) d c").detach()
        
        # BC * HW * 8D --> BC * HW * 8 * D --> BC * 8 * HW * D
        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
        # BC * 8 * HW2 * D
        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
        # BC * 8 * HW2 * D2
        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)

pull request requirements.txt

sorry still havent figured out pull requests
in requirements.txt

pip install torchvision=0.16.0
should be
pip install torchvision==0.16.0

im on windows annaconda so maybe its different on other systems

Enable support to use lightning model

https://huggingface.co/ByteDance/SDXL-Lightning

[Run Key Frames] sucess, but [Run Propagation (Ebsynth)] failed: FileNotFoundError: [Errno 2] No such file or directory: 'output/480x480/blend.mp4'

Running on local URL: http://127.0.0.1:7860/
Running on public URL: https://26363563e97e3055dc.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
100% 15/15 [01:10<00:00, 4.71s/it]
100% 15/15 [00:45<00:00, 3.01s/it]
python video_blend.py output/3 seconds --key keys --key_ind 0 12 18 27 32 52 57 64 74 79 90 --output output/3 seconds/blend.mp4 --fps 30 --n_proc 4 -ps
usage: video_blend.py [-h] [--output OUTPUT] [--fps FPS] [--key_ind KEY_IND [KEY_IND ...]]
[--key KEY] [--n_proc N_PROC] [-ps] [-ne] [-tmp]
name
video_blend.py: error: unrecognized arguments: seconds seconds/blend.mp4
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1440, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1341, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 281, in postprocess
processed_files = (self.format_video(y), None)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 355, in format_video
video = self.make_temp_copy_if_needed(video)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 226, in make_temp_copy_if_needed
temp_dir = self.hash_file(file_path)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 190, in hash_file
with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/3 seconds/blend.mp4'
/content/drive/MyDrive/video_gen/FRESCO/src/free_lunch_utils.py:38: UserWarning: ComplexHalf support is experimental and many operators don't support it yet. (Triggered internally at ../aten/src/ATen/EmptyTensor.cpp:31.)
x_freq = fft.fftn(x, dim=(-2, -1))
100% 15/15 [00:29<00:00, 1.96s/it]
100% 15/15 [00:19<00:00, 1.32s/it]
python video_blend.py output/480x480 --key keys --key_ind 0 12 18 27 32 52 57 64 74 79 90 --output output/480x480/blend.mp4 --fps 30 --n_proc 4 -ps
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_74/0075.jpg'): can't open/read file: check file path/integrity
Process Process-4:
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_57/0058.jpg'): can't open/read file: check file path/integrity
Process Process-3:
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_27/0028.jpg'): can't open/read file: check file path/integrity
Process Process-2:
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self.target(*self.args, **self.kwargs)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 111, in process_sequences
process_one_sequence(i, video_sequence)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 98, in process_one_sequence
cmd += ' ' + g.get_cmd(j, w)
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
File "/content/drive/MyDrive/video_gen/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
if len(img.shape) == 2:
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 8.934192419052124
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_0/0011.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0012.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0006.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread('output/480x480/out_12/0007.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0008.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0009.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0010.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('output/480x480/out_12/0011.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 308, in
main(args)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 268, in main
process_seq(video_sequence, i, blend_histogram, blend_gradient)
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 200, in process_seq
dist1s.append(load_error(bin_a, img_shape))
File "/content/drive/MyDrive/video_gen/FRESCO/video_blend.py", line 161, in load_error
with open(bin_path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'output/480x480/out_0/0001.bin'
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/queueing.py", line 388, in call_prediction
output = await route_utils.call_process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/route_utils.py", line 219, in call_process_api
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1440, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1341, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 281, in postprocess
processed_files = (self._format_video(y), None)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/video.py", line 355, in _format_video
video = self.make_temp_copy_if_needed(video)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 226, in make_temp_copy_if_needed
temp_dir = self.hash_file(file_path)
File "/usr/local/lib/python3.10/dist-packages/gradio/components/base.py", line 190, in hash_file
with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/480x480/blend.mp4'

run on Colab with A100 GPU and 40G GPU RAM still got Error

run webUI.py on Colab with A100 GPU and 40G GPU RAM still got Error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.25 GiB (GPU 0; 39.56 GiB total capacity; 21.83 GiB already allocated; 10.45 GiB free; 22.58 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Video blend too slow: 18 vCPU / RTX 3090(24GB)

Just run the demo below and generate 1MB video, cost me almost 10min.
Could it be optimized?

python run_fresco.py ./config/config_music.yaml

config_music.yaml:
run_ebsynth: True

notebook doesn't run

Notebooks are a great tool to help people walk through a demo of something step by step. The notebook provided has some prerequisites listed at the top that have to be manually figured out. It'd be great if the notebook was standalone and had cells for all the steps required to run.

Is there any way to speed it up?

Great work. Thank you very much.
And Is there any way to speed it up?
3090 GPU, deal with about 100frames,need 40 min+ ,got a 3 sec video.
I'm not sure if it's my setup or something?

blender not found

how to install blender package in video_blend.py file? I use "pip install blender", when import blender, has "ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory"

I have a 1.5 model .safetensor file downloaded from civitai. How to use it?

I have a 1.5 model .safetensor file downloaded from civitai. How to use it? I tried to change the sd_model to include path to the file but it complains that it cannot find model_index.json

can not find target file while executing process_seq(video_sequence, i, blend_histogram, blend_gradient) in video_blend.py

After I configured parameter "run_ebsynth" as True to generate video occurred the following error

I am wondering is the files were not generated or is just given the wrong target file path ？

And why the target file is not there (and there are no *.bin files generated in output directory):
./output/boxer-punching-towards-camera/out_0/0001.jpg
./output/boxer-punching-towards-camera/out_2/0001.jpg

is there something wrong in code or the related file is not generated correctly?

this is the parameter config in yaml file

this is the error log

Out of memory: RTX 4090 / 24GB

Hi no matter what movie size I choose -- 5fps, 640x480 I get this error below. nvtop shows that the webUI triggers pre-allocation of 21.5GB but then ... it's not used?

/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 0%| | 0/15 [00:01<?, ?it/s] Traceback (most recent call last): File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/queueing.py", line 388, in call_prediction output = await route_utils.call_process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/route_utils.py", line 219, in call_process_api output = await app.get_blocks().process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/blocks.py", line 1437, in process_api result = await self.call_function( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/blocks.py", line 1109, in call_function prediction = await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/gradio/utils.py", line 650, in wrapper response = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/webUI.py", line 159, in process keypath = process1(*args) ^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/webUI.py", line 280, in process1 latents = inference(global_state.pipe, global_state.controlnet, global_state.frescoProc, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/pipe_FRESCO.py", line 201, in inference noise_pred = pipe.unet( ^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/diffusion_hacked.py", line 776, in forward sample = optimize_feature(sample, flows, occs, correlation_matrix, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/diffusion_hacked.py", line 485, in optimize_feature optimizer.step(closure) File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/optim/optimizer.py", line 373, in wrapper out = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/optim/optimizer.py", line 76, in _use_grad ret = func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/optim/adam.py", line 143, in step loss = closure() ^^^^^^^^^ File "/home/julian/sambashare/expts/FRESCO/src/diffusion_hacked.py", line 478, in closure loss.backward() File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/autograd/__init__.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.06 GiB. GPU 0 has a total capacty of 23.65 GiB of which 525.94 MiB is free. Process 15481 has 1.27 GiB memory in use. Including non-PyTorch memory, this process has 21.00 GiB memory in use. Of the allocated memory 20.11 GiB is allocated by PyTorch, and 416.71 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

add --listen

Can you add the --listen startup parameter to enable access within the LAN?

修改端口ip

Running on local URL: http://127.0.0.1:7860 哪里修改ip和端口呢

Which directory should the SD1.5 model be placed in?

Hi:

After run the script

python run_fresco.py ./config/config_music.yaml

But I didn't found any SD1.5 model in the project.

Is it necessary to manually download the SD1.5 model?
Which directory should the SD1.5 model be placed in?

BTW : the default path on windows 10 is C:\Users\xxxx\.cache\huggingface\hub

(FRESCO) gruntus FRESCO % python ./run_fresco.py ./config/config_carturn.yaml
====================================================================================================
loading configuration...
batch_size: 8
cond_scale: 0.7
controlnet_type: hed
end_opt_step: 15
file_path: ./data/car-turn.mp4
gmflow_path: ./model/gmflow_sintel-0c07dcb3.pth
max_process: 4
maxinterv: 5
mininterv: 5
num_inference_steps: 20
num_warmup_steps: 5
prompt: a red car turns in the winter
run_ebsynth: False
save_path: ./output/car-turn/
sd_path: SG161222/Realistic_Vision_V2.0
seed: 0
sod_path: ./model/epoch_resnet.pth
use_controlnet: True
use_freeu: False
use_salinecy: True

====================================================================================================
creating models...
create optical flow estimation model successfully!
create saliency detection model successfully!
create controlnet model-hed successfully!
unet/diffusion_pytorch_model.safetensors not found
Loading pipeline components...:  57%|████████▌      | 4/7 [00:01<00:01,  2.57it/s]/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading pipeline components...: 100%|███████████████| 7/7 [00:01<00:00,  3.77it/s]
create diffusion model SG161222/Realistic_Vision_V2.0 successfully!

====================================================================================================
key frame selection for "./data/car-turn.mp4"...
processing 3 batches:
keyframe indexes [[0, 5, 10, 15, 20, 25, 30, 35], [40, 45, 50, 55, 60], [65, 70, 75]]

====================================================================================================
video to video translation...
processing batch [1/3] with 8 frames
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
100%|█████████████████████████████████████████████| 15/15 [00:26<00:00,  1.73s/it]
processing batch [2/3] with 5 frames
100%|█████████████████████████████████████████████| 15/15 [00:22<00:00,  1.51s/it]
processing batch [3/3] with 3 frames
100%|█████████████████████████████████████████████| 15/15 [00:16<00:00,  1.11s/it]

====================================================================================================
to translate full video with ebsynth, install ebsynth and run:

python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps


====================================================================================================
Done
(FRESCO) gruntus FRESCO % python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75  --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([0, 1, 2, 3], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([4, 5, 6, 7], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([8, 9, 10, 11],
           <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([12, 13, 14], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_20/0021.jpg'): can't open/read file: check file path/integrity
Process Process-2:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_40/0041.jpg'): can't open/read file: check file path/integrity
Process Process-3:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_60/0061.jpg'): can't open/read file: check file path/integrity
Process Process-4:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 12.073577880859375
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0004.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 311, in <module>
    main(args)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 271, in main
    process_seq(video_sequence, i, blend_histogram, blend_gradient)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 203, in process_seq
    dist1s.append(load_error(bin_a, img_shape))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 164, in load_error
    with open(bin_path, 'rb') as fp:
         ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './output/car-turn/out_0/0001.bin'
``

williamyang1991 / fresco Goto Github PK

fresco's People

Contributors

Stargazers

Watchers

Forkers

fresco's Issues

Recommend Projects

Recommend Topics

Recommend Org