thecooltechguy / comfyui-stable-video-diffusion Goto Github PK

I'm trying to get the same output results on the new built-in SVD nodes.

The differences:
CoolTechGuy node set could set the decode_T, but the new nodes don't have decode_t - where would you set this on the new nodes?

The new nodes uses the Ksampler and sets Euler and Karras as the sampler and scheduler. For CoolTechGuy SVD nodes, what are the default sampler and scheduler? (what would be the closest pick? is it Eular and ddim_uniform?)

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\nodes.py", line 56, in load_svd_model
self.svd_model = load_model(
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\svd.py", line 78, in load_model
model = instantiate_from_config(config.model).to(device).eval()
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\util.py", line 186, in get_obj_from_str
return getattr(importlib.import_module(module, package=None), cls)
File "importlib_init_.py", line 126, in import_module
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in load_unlocked
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm_init.py", line 1, in
from .models import AutoencodingEngine, DiffusionEngine
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\models_init.py", line 1, in
from .autoencoder import AutoencodingEngine
File "C:\Users\stable\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\models\autoencoder.py", line 8, in
import pytorch_lightning as pl
ModuleNotFoundError: No module named 'pytorch_lightning'

When loading the graph, the following node types were not found:
SVDDecoder
SVDModelLoader
SVDSampler
Nodes that have failed to load will show as red on the graph.

If the path has a space, its not working correctly.

The solution is to add quotation marks here:

log:
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "C:\ai\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\ai\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\ai\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\nodes.py", line 55, in load_svd_model
self.svd_model = load_model(
File "C:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\svd.py", line 79, in load_model
model = instantiate_from_config(config.model).to(device).eval()
File "C:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "C:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\util.py", line 186, in get_obj_from_str
return getattr(importlib.import_module(module, package=None), cls)
File "importlib_init_.py", line 126, in import_module
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in load_unlocked
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "C:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm_init.py", line 1, in
from .models import AutoencodingEngine, DiffusionEngine
File "C:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\models_init.py", line 1, in
from .autoencoder import AutoencodingEngine
File "C:\ai\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\models\autoencoder.py", line 8, in
import pytorch_lightning as pl
ModuleNotFoundError: No module named 'pytorch_lightning'

Prompt executed in 0.12 seconds

prolly a dumb issue but like how do i fix it?

I am in google colab and I can not run well SVD, although in this other colab free if you can run but it has the watermark stability filter and nsfw that bothers a lot, maybe you can get things from here to reduce vram usage, in my little knowledge with models I think a good option is to load the model in float16, but I better leave to those who know to do it.

colab mentioned: https://colab.research.google.com/github/mkshing/notebooks/blob/main/stable_video_diffusion_img2vid.ipynb

failed to import... nodes will stay red after trying to reinstall 3 times.

Everytime I start comfyUI this plugin tries to download all 4 models.

I want to use just svd_xt and I downloaded it manually.

I don't want to install the other models in order to save some disk space and not wait for the download.

As a workaround I ctrl-C all the downloads and remove the downloaded parts everytime when I'm starting comfyUI.

I suggest to start the downloads only if there is not at least one model file in the models/svd directory.

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\comfy_extras\nodes_video_model.py", line 45, in encode
output = clip_vision.encode_image(init_image)
^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'encode_image'

any idea why this is happening?

Why the forced model download? If it was a small file then it wouldn't matter, but since it's 9Gb and a lot of people don't have superfast internet it should be optional.

A lot of people also have multiple installations for image generation, often with shared folders, a forced download to a forced path isn't necessary.

The SVD sampler is giving the error:

Error occurred when executing SVDSampler:

Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated : 15.96 GiB
Requested : 2.26 GiB
Device limit : 23.65 GiB
Free (according to CUDA): 25.19 MiB
PyTorch limit (set by user-supplied memory fraction)
: 17179869184.00 GiB

I'm confused; I have 24GBs of VRAM, and yet it's saying that is' running out of VRAM beyond 16GBs. Am I doing something wrong? This is the only process running on my GPU.

How to fix it？！！！

`The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\DProjects\AI\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\DProjects\AI\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\DProjects\AI\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\nodes.py", line 55, in load_svd_model
self.svd_model = load_model(
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\svd.py", line 79, in load_model
model = instantiate_from_config(config.model).to(device).eval()
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\models\diffusion.py", line 59, in init
self.conditioner = instantiate_from_config(
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\modules\encoders\modules.py", line 79, in init
embedder = instantiate_from_config(embconfig)
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\modules\encoders\modules.py", line 1039, in init
self.open_clip = instantiate_from_config(open_clip_embedding_config)
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "D:\DProjects\AI\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\modules\encoders\modules.py", line 591, in init
model, _, _ = open_clip.create_model_and_transforms(
File "D:\DProjects\AI\ComfyUI\venv\lib\site-packages\open_clip\factory.py", line 382, in create_model_and_transforms
model = create_model(
File "D:\DProjects\AI\ComfyUI\venv\lib\site-packages\open_clip\factory.py", line 281, in create_model
checkpoint_path = download_pretrained(pretrained_cfg, cache_dir=cache_dir)
File "D:\DProjects\AI\ComfyUI\venv\lib\site-packages\open_clip\pretrained.py", line 552, in download_pretrained
target = download_pretrained_from_hf(model_id, cache_dir=cache_dir)
File "D:\DProjects\AI\ComfyUI\venv\lib\site-packages\open_clip\pretrained.py", line 522, in download_pretrained_from_hf
cached_file = hf_hub_download(model_id, filename, revision=revision, cache_dir=cache_dir)
File "D:\DProjects\AI\ComfyUI\venv\lib\site-packages\huggingface_hub\utils_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "D:\DProjects\AI\ComfyUI\venv\lib\site-packages\huggingface_hub\file_download.py", line 1371, in hf_hub_download
raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.`

Thanks for creating this node so quickly!

I'm running on an A100 with 40gb VRAM, but seeing this:

This is with normal SVD. SVD_xt would OOM before even running inference.

When I used the official workflow, I encountered this error and was stuck at the SVDS step. Nothing changed for an hour.

E:\comfyui\new_ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils[checkpoint.py:429](https://checkpoint.py:429/): UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.

warnings.warn(

E:\comfyui\new_ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils[checkpoint.py:61](https://checkpoint.py:61/): UserWarning: None of the inputs have requires_grad=True. Gradients will be None

warnings.warn(

what should I do?

ComfyUI import the custom node suite just fine, but the SVD Model Loader node fails to appear in the menu due to the following error:

[ERROR] An error occurred while retrieving information for the 'SVDModelLoader' node.
Traceback (most recent call last):
File "xyz/ComfyUI/server.py", line 418, in get_object_info
out[x] = node_info(x)
^^^^^^^^^^^^
File "xyz/ComfyUI/server.py", line 396, in node_info
info['input'] = obj_class.INPUT_TYPES()
^^^^^^^^^^^^^^^^^^^^^^^
File "xyz/ComfyUI/custom_nodes/ComfyUI-Stable-Video-Diffusion/nodes.py", line 28, in INPUT_TYPES
"default" : checkpoints[0],
~~~~~~~~~~~^^^
IndexError: list index out of range

Thank you

UPDATE

I figured it out: the error (and the node loading failure) occurs if ComfyUI is started before the models are placed in the indicated folders.

Throws in console:
** ComfyUI start up time: 2023-11-23 19:35:05.661688

Prestartup times for custom nodes:
0.0 seconds: D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\rgthree-comfy
0.0 seconds: D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager

Traceback (most recent call last):
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\main.py", line 72, in
import execution
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 12, in
import nodes
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\nodes.py", line 20, in
import comfy.diffusers_load
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\diffusers_load.py", line 4, in
import comfy.sd
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 5, in
from comfy import model_management
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\model_management.py", line 114, in
total_vram = get_total_memory(get_torch_device()) / (1024 * 1024)
^^^^^^^^^^^^^^^^^^
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\ComfyUI\comfy\model_management.py", line 83, in get_torch_device
return torch.device(torch.cuda.current_device())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda_init_.py", line 769, in current_device
lazy_init()
File "D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda_init.py", line 289, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

D:\WORK\conda_envs\ComfyUI\ComfyUI_windows_portable>pause
Press any key to continue . . .

Any way to mitigate this without a full re-install of Comfy ?

Hello, when trying to run first test render I have this error:

Error occurred when executing SVDModelLoader:

No module named 'pytorch_lightning'

File "C:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\nodes.py", line 55, in load_svd_model
self.svd_model = load_model(
File "C:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\svd.py", line 79, in load_model
model = instantiate_from_config(config.model).to(device).eval()
File "C:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\util.py", line 175, in instantiate_from_config
return get_obj_from_str(config["target"])(**config.get("params", dict()))
File "C:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\util.py", line 186, in get_obj_from_str
return getattr(importlib.import_module(module, package=None), cls)
File "importlib_init_.py", line 126, in import_module
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in load_unlocked
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "C:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm_init.py", line 1, in
from .models import AutoencodingEngine, DiffusionEngine
File "C:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\models_init.py", line 1, in
from .autoencoder import AutoencodingEngine
File "C:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion\libs\sgm\models\autoencoder.py", line 8, in
import pytorch_lightning as pl

Hi.........I'm not a coder, I just know modify some simple codes in .dat file.
I got these errors.
How can I solve this?
Thank you!

got prompt
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
Initialized embedder #0: FrozenOpenCLIPImagePredictionEmbedder with 683800065 params. Trainable: False
Initialized embedder #1: ConcatTimestepEmbedderND with 0 params. Trainable: False
Initialized embedder #2: ConcatTimestepEmbedderND with 0 params. Trainable: False
Initialized embedder #3: VideoPredictionEmbedderWithEncoder with 83653863 params. Trainable: False
Initialized embedder #4: ConcatTimestepEmbedderND with 0 params. Trainable: False

D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable>set "TRANSFORMERS_CACHE=D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable_cache\huggingface\transformers"

D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable>set "HF_HOME=D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable_cache\huggingface"

D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable>set "XDG_CACHE_HOME=D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable_cache"

D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable>set "HF_DATASETS_CACHE=D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable_cache\huggingface\datasets"

D:\AI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable>pause
Press any key to continue . . .

hi
After following everything and downloading the model, this appeared

An error message appeared during the second step of installation.
this step:cd ComfyUI-Stable-Video-Diffusion/
pip install -r requirements.txt