Doing animatediff generate --device cpu does not work. Generating with cuda does, and the program works fine otherwise, but cpu support does not appear to be functional.

╭──────────────────────────── Traceback (most recent call last) ─────────────────────────────╮
│/animatediff-cli/src/animatediff/cli.py:154 in     │
│ generate                                                                                   │
│                                                                                            │
│   151 │   set_diffusers_verbosity_error()                                                  │
│   152 │                                                                                    │
│   153 │   device = torch.device(device)                                                    │
│ ❱ 154 │   device_info = torch.cuda.get_device_properties(device)                           │
│   155 │                                                                                    │
│   156 │   logger.info(device_info_str(device_info))                                        │
│   157 │   has_bf16 = torch.cuda.is_bf16_supported()                                        │
│                                                                                            │
│ /animatediff-cli/venv/lib/python3.10/site-packages │
│ /torch/cuda/__init__.py:396 in get_device_properties                                       │
│                                                                                            │
│    393 │   │   _CudaDeviceProperties: the properties of the device                         │
│    394 │   """                                                                             │
│    395 │   _lazy_init()  # will define _get_device_properties                              │
│ ❱  396 │   device = _get_device_index(device, optional=True)                               │
│    397 │   if device < 0 or device >= device_count():                                      │
│    398 │   │   raise AssertionError("Invalid device id")                                   │
│    399 │   return _get_device_properties(device)  # type: ignore[name-defined]             │
│                                                                                            │
│ /animatediff-cli/venv/lib/python3.10/site-packages │
│ /torch/cuda/_utils.py:32 in _get_device_index                                              │
│                                                                                            │
│   29 │   │   │   if device.type not in ['cuda', 'cpu']:                                    │
│   30 │   │   │   │   raise ValueError('Expected a cuda or cpu device, but got: {}'.format( │
│   31 │   │   elif device.type != 'cuda':                                                   │
│ ❱ 32 │   │   │   raise ValueError('Expected a cuda device, but got: {}'.format(device))    │
│   33 │   if not torch.jit.is_scripting():                                                  │
│   34 │   │   if isinstance(device, torch.cuda.device):                                     │
│   35 │   │   │   return device.idx                                                         │
╰────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Expected a cuda device, but got: cpu

If you follow the error further down the line, it turns into an issue with fp16 not being possible on cpu. It might be extremely slow, but proper cpu support may be worth adding.

I finally got it to work on my phone but after it crashed I can no longer run the app it keeps saying animatediff command not found

Hey when u enable Cpu Mode it still fails at cuda check at sertant parts I fixed it but 2 files one is src/animatediff/generate.py and the other is src/animatediff/pipeline/animation.py all u gotta do is change the (cuda) in brackets to Cpu in both those files u don't need to mess with the one by igpu and the torch.cuda.empty lines just the ones in brackets

Hi, can you show me how to use rife in the Readme in detail? I don't know how to use it .

Using scheduler "k_dpmpp_2m_sde" (DPMSolverMultistepScheduler) generate.py:68 INFO Loading weights from E:\animatediff-cli\data\models\sd\majicmixRealistic_v7.safetensors generate.py:73 08:13:58 INFO Merging weights into UNet... generate.py:90 08:13:59 INFO Creating AnimationPipeline... generate.py:110 INFO No TI embeddings found generate.py:131 INFO Sending pipeline to device "cuda" pipeline.py:22 INFO Selected data types: unet_dtype=torch.float16, tenc_dtype=torch.float16, device.py:90 vae_dtype=torch.float32 INFO Using channels_last memory format for UNet and VAE device.py:109 08:14:02 INFO Saving prompt config to output directory cli.py:290 INFO Initialization complete! cli.py:299 INFO Generating 1 animations from 1 prompts cli.py:300 INFO Running generation 1 of 1 (prompt 1) cli.py:309 INFO Generation seed: 10925512164 cli.py:319 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20/20 [ 0:11:50 < 0:00:00 , ? it/s ] 08:26:00 INFO Generation complete, saving... generate.py:175 INFO Saved sample to generate.py:188 output\2024-03-18T08-13-43-girl-majicmixrealistic_v7\00_10925561214_1girl_solo_best-quali ty_masterpiece_looking-at-viewer_purple-hair.gif Saving frames to 00-10925564 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 [ 0:00:00 < 0:00:00 , ? it/s ] INFO Generation complete! cli.py:345 INFO Done, exiting... The generated images and GIFs are pure black

(.venv) PS D:\python\animatediff-cli> animatediff generate
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.1.2+cu121 with CUDA 1201 (you have 2.1.2+cu118)
Python 3.10.11 (you have 3.10.7)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
Traceback (most recent call last):
File "C:\Users\ro612\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\ro612\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "D:\python\animatediff-cli.venv\Scripts\animatediff.exe_main.py", line 4, in
File "D:\python\animatediff-cli\src\animatediff\cli.py", line 12, in
from animatediff.generate import create_pipeline, run_inference
File "D:\python\animatediff-cli\src\animatediff\generate.py", line 12, in
from animatediff.models.clip import CLIPSkipTextModel
File "D:\python\animatediff-cli\src\animatediff\models\clip.py", line 7, in
from transformers.models.clip.modeling_clip import (
ImportError: cannot import name '_expand_mask' from 'transformers.models.clip.modeling_clip' (D:\python\animatediff-cli.venv\lib\site-packages\transformers\models\clip\modeling_clip.py)
(.venv) PS D:\python\animatediff-cli> pip install triton
ERROR: Could not find a version that satisfies the requirement triton (from versions: none)
ERROR: No matching distribution found for triton

Hi, thanks for your excellent work.
I have some trouble understanding the uniform function in pipeline/context.py. Why does this code introduce the bitwise operation?

i'm using same settings (such as number of steps, frames, resolution, scheduler) and both running on cuda.

i've tried 2 different gpus, gtx 1080 and tesla p40

and always i'm getting about 2-3 times slower result with your code (although animation also looks better)

did you noticed the same difference in performance? if not, mb some ideas what i could be overlooking in the settings

also in your code amount of vram used is lower - so i suspect that some computation which in original AnimateDiff done on gpu here are done on cpu (i quickly looked into the code of the pipeline and i saw that for long video sequential_mode is used which works on cpu - but i've been testing on 8-16 frames animations, so that shouldn't be the case)

My terminal crashes after loading 50% of the model weights can u please make smaller resolution I think it's the 512x512 limit but I need 320x320 but I'm trying to load absolutereality 1.6 model and plus transformers 4.33.0 is required to run. Animatediff cli I. On linux cpu BTW it works in comfyui but only 320x320 resolution,I even put -C to 1 and it still crashed! Maybe the model is corrupt idk I was hoping to use this cause it suppose to use less ram.

This uses up all my ram I thought this was supposed to use less please optimize this or add command line args like split attention and force-fp16 --cpu etc

im having an issue for installation atm, it worked the first time i installed it. but after deleting it.
im trying to install again, and im getting these error

I have configured the environment as required and downloaded the relevant models. When I run the above command, an error message is reported. The error message is as follows. How should I solve it:

animatediff-cli/src/animatediff/cli.py:252 in generate                                                                 │
│                                                                                                  │
│   249 │                                                                                          │
│   250 │   config_path = config_path.absolute()                                                   │
│   251 │   logger.info(f"Using generation config: {relative_path(config_path)}")                  │
│ ❱ 252 │   model_config: ModelConfig = get_model_config(config_path)                              │
│   253 │   infer_config: InferenceConfig = get_infer_config()                                     │
│   254 │                                                                                          │
│   255 │   # set sane defaults for context, overlap, and stride if not supplied                   │
│                                                                                                  │
│ animatediff-cli/src/an │
│ imatediff/settings.py:126 in get_model_config                                                    │
│                                                                                                  │
│   123                                                                                            │
│   124 @lru_cache(maxsize=2)                                                                      │
│   125 def get_model_config(config_path: Path) -> ModelConfig:                                    │
│ ❱ 126 │   settings = ModelConfig(json_config_path=config_path)                                   │
│   127 │   return settings                                                                        │
│   128                                                                                            │
│                                                                                                  │
│ in pydantic.env_settings.BaseSettings.__init__:39                                                │
│                                                                                                  │
│ in pydantic.main.BaseModel.__init__:342                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: 1 validation error for ModelConfig
scheduler
  value is not a valid enumeration member; permitted: 'ddim', 'pndm', 'heun', 'unipc', 'euler', 'euler_a', 'lms', 'k_lms', 'dpm_2', 'k_dpm_2', 'dpm_2_a', 'k_dpm_2_a', 'dpmpp_2m', 'k_dpmpp_2m', 'dpmpp_sde', 'k_dpmpp_sde', 'dpmpp_2m_sde', 'k_dpmpp_2m_sde' (type=type_error.enum; enum_values=[<DiffusionScheduler.ddim: 'ddim'>, 
<DiffusionScheduler.pndm: 'pndm'>, <DiffusionScheduler.heun: 'heun'>, <DiffusionScheduler.unipc: 'unipc'>, <DiffusionScheduler.euler: 'euler'>, <DiffusionScheduler.euler_a: 'euler_a'>, <DiffusionScheduler.lms: 'lms'>, <DiffusionScheduler.k_lms: 'k_lms'>, <DiffusionScheduler.dpm_2: 'dpm_2'>, <DiffusionScheduler.k_dpm_2: 'k_dpm_2'>, 
<DiffusionScheduler.dpm_2_a: 'dpm_2_a'>, <DiffusionScheduler.k_dpm_2_a: 'k_dpm_2_a'>, <DiffusionScheduler.dpmpp_2m: 'dpmpp_2m'>, <DiffusionScheduler.k_dpmpp_2m: 'k_dpmpp_2m'>, <DiffusionScheduler.dpmpp_sde: 'dpmpp_sde'>, <DiffusionScheduler.k_dpmpp_sde: 'k_dpmpp_sde'>, <DiffusionScheduler.dpmpp_2m_sde: 'dpmpp_2m_sde'>, 
<DiffusionScheduler.k_dpmpp_2m_sde: 'k_dpmpp_2m_sde'>])

Is there an extra command line option or something to kick rife off? I've installed as per the instructions but it doesn't seem to be doing anything. Also how to you specify movie output instead of gif?

I really want to be able to use a specific image as a starting point.
There is an example and places how this can be implemented for the original (AnimateDiff) repository.
https://github.com/search?q=repo%3Atalesofai%2FAnimateDiff%20init_image&type=code

Maybe you can adapt code for this repository?

And a little extra question: How to update animatediff-cli correctly? "git pull" and then "python -m pip install -e . " ?

When I install the dependency by python -m pip install -e '.[dev]', this error occured, how to solve it?

Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done ERROR: Exception: Traceback (most recent call last): File "/opt/anaconda3/envs/ldm/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper status = run_func(*args) File "/opt/anaconda3/envs/ldm/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper return func(self, options, args) File "/opt/anaconda3/envs/ldm/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 377, in run requirement_set = resolver.resolve( File "/opt/anaconda3/envs/ldm/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve result = self._result = resolver.resolve( File "/opt/anaconda3/envs/ldm/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve state = resolution.resolve(requirements, max_rounds=max_rounds) File "/opt/anaconda3/envs/ldm/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve raise ResolutionTooDeep(max_rounds) pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

(.venv) root@autodl-container-a1c3118008-79be1975:~/autodl-tmp/cli/animatediff-cli# animatediff generate -c 'config/prompts/A1.json' -W 576 -H 576 -L 128 -C 16
02:41:48 INFO Using generation config: config/prompts/A1.json cli.py:247
INFO Using base model: runwayml/stable-diffusion-v1-5 cli.py:258
INFO Will save outputs to ./output/2023-08-11T02-41-48-a1-realisticvisionv40_v40vae cli.py:266
INFO Checking motion module... generate.py:39
INFO Loading tokenizer... generate.py:51
INFO Loading text encoder... generate.py:53
02:41:50 INFO Loading VAE... generate.py:55
INFO Loading UNet... generate.py:57
02:42:03 INFO Loaded 417.1376M-parameter motion module unet.py:559
INFO Using scheduler "euler" (EulerDiscreteScheduler) generate.py:69
INFO Loading weights from /root/autodl-tmp/models/ckpt/realisticVisionV40_v40VAE.safetensors generate.py:74
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /root/autodl-tmp/cli/animatediff-cli/src/animatediff/cli.py:273 in generate │
│ │
│ 270 │ global last_model_path │
│ 271 │ if pipeline is None or last_model_path != base_model_path.resolve(): │
│ 272 │ │ │
│ ❱ 273 │ │ pipeline = create_pipeline( │
│ 274 │ │ │ base_model=base_model_path, │
│ 275 │ │ │ model_config=model_config, │
│ 276 │ │ │ infer_config=infer_config, │
│ │
│ /root/autodl-tmp/cli/animatediff-cli/src/animatediff/generate.py:77 in create_pipeline │
│ │
│ 74 │ │ logger.info(f"Loading weights from {model_path}") │
│ 75 │ │ if model_path.is_file(): │
│ 76 │ │ │ logger.debug("Loading from single checkpoint file") │
│ ❱ 77 │ │ │ unet_state_dict, tenc_state_dict, vae_state_dict = get_checkpoint_weights(mo │
│ 78 │ │ elif model_path.is_dir(): │
│ 79 │ │ │ logger.debug("Loading from Diffusers model directory") │
│ 80 │ │ │ temp_pipeline = StableDiffusionPipeline.from_pretrained(model_path) │
│ │
│ /root/autodl-tmp/cli/animatediff-cli/src/animatediff/utils/model.py:73 in get_checkpoint_weights │
│ │
│ 70 │
│ 71 def get_checkpoint_weights(checkpoint: Path): │
│ 72 │ temp_pipeline: StableDiffusionPipeline │
│ ❱ 73 │ temp_pipeline, _ = checkpoint_to_pipeline(checkpoint, save=False) │
│ 74 │ unet_state_dict = temp_pipeline.unet.state_dict() │
│ 75 │ tenc_state_dict = temp_pipeline.text_encoder.state_dict() │
│ 76 │ vae_state_dict = temp_pipeline.vae.state_dict() │
│ │
│ /root/autodl-tmp/cli/animatediff-cli/src/animatediff/utils/model.py:54 in checkpoint_to_pipeline │
│ │
│ 51 │ target_dir: Optional[Path] = None, │
│ 52 │ save: bool = True, │
│ 53 ) -> StableDiffusionPipeline: │
│ ❱ 54 │ logger.debug(f"Converting checkpoint {path_from_cwd(checkpoint)}") │
│ 55 │ if target_dir is None: │
│ 56 │ │ target_dir = pipeline_dir.joinpath(checkpoint.stem) │
│ 57 │
│ │
│ /root/autodl-tmp/cli/animatediff-cli/src/animatediff/utils/util.py:44 in path_from_cwd │
│ │
│ 41 │
│ 42 def path_from_cwd(path: PathLike) -> str: │
│ 43 │ path = Path(path) │
│ ❱ 44 │ return str(path.absolute().relative_to(Path.cwd())) │
│ 45 │
│ │
│ /root/miniconda3/lib/python3.10/pathlib.py:818 in relative_to │
│ │
│ 815 │ │ cf = self._flavour.casefold_parts │
│ 816 │ │ if (root or drv) if n == 0 else cf(abs_parts[:n]) != cf(to_abs_parts): │
│ 817 │ │ │ formatted = self._format_parsed_parts(to_drv, to_root, to_parts) │
│ ❱ 818 │ │ │ raise ValueError("{!r} is not in the subpath of {!r}" │
│ 819 │ │ │ │ │ " OR one path is relative and the other is absolute." │
│ 820 │ │ │ │ │ │ │ .format(str(self), str(formatted))) │
│ 821 │ │ return self._from_parsed_parts('', root if n == 1 else '', │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: '/root/autodl-tmp/models/ckpt/realisticVisionV40_v40VAE.safetensors' is not in the subpath of '/root/autodl-tmp/cli/animatediff-cli' OR one path is
relative and the other is absolute.

pls

Would be great to transform an already existing image and animate it instead of first generating using prompts and then animating.

Maybe also batch support so it can take multiple images from a folder and then transform all of them into gifs automatically without manual prompting.

I got this error when I tried to run it in Colab. Not sure how to resolve it.

Traceback (most recent call last): File "/usr/local/bin/animatediff", line 5, in <module> from animatediff.cli import cli File "/content/animatediff-cli/src/animatediff/cli.py", line 12, in <module> from animatediff.generate import create_pipeline, run_inference File "/content/animatediff-cli/src/animatediff/generate.py", line 13, in <module> from animatediff.models.unet import UNet3DConditionModel File "/content/animatediff-cli/src/animatediff/models/unet.py", line 18, in <module> from .unet_blocks import ( File "/content/animatediff-cli/src/animatediff/models/unet_blocks.py", line 9, in <module> from animatediff.models.attention import Transformer3DModel File "/content/animatediff-cli/src/animatediff/models/attention.py", line 10, in <module> from diffusers.utils import BaseOutput, maybe_allow_in_graph ImportError: cannot import name 'maybe_allow_in_graph' from 'diffusers.utils' (/usr/local/lib/python3.10/dist-packages/diffusers/utils/__init__.py)

I tested the newly added Clip_skip this time. For anime checkpoints, definitely 2 will generate cleaner and more natural images than 1.
Except for Clip_skip, Seed and Prompt are identical.

Thanks for the great work! Right now I have to run the generate every time I want to try a different seed from one prompt. It would be cool to be able to specify a number of videos that can be generated in sequence when a random seed[-1] is entered at one prompt! Like this Batch count = 10

Should this instead check if the file exists? Is model_index.json meant to do anything otherwise?

I don't think the motion weights have been trained to work with SDXL ... but ... do you see any efforts to update everything to SDXL?

Thank you for your project, it's very surprising,But I seem to have encountered a problem,It seems like I just loaded them, but I didn't use them. Is that the case? If so, how should I use them.

Hi @neggles,

Glad to see active development into AnimateDiff. You've got some cool ideas going forward. I'm also working on an AnimateDiff repo focusing on the UI side. I think it would be good if I can just call your CLI for the UI to consolidate some development efforts however I'm wondering if there's a way to cache the loading of stable diffusion as that would save about a min between each generation.

Everytime I get to the start of loading models it stops at 50% and crashes termux idk y this happens I. can't get pass this part. It works fine on my laptop which has 16gb ram but my phone has 12 but only 10602 gb is read.

I managed to get it setup and running on Colab. I did change the motion-module to mm_sd_v15_v2.ckpt.
But once I try to execute the ToonYou script, the process terminates at "Using generation config".
This does not happen when I am using mm_sd_v15.ckpt.

Hello, this animatediff implementation is great. Does it also support IP Adapter?

In the prompt Json I have:
"path": "c:/StableDiffusion/stable-diffusion-webui/models/Stable-diffusion/realisticVisionV50_v40VAE.safetensors",
After running I get:

ValueError:
'c:\\StableDiffusion\\stable-diffusion-webui\\models\\Stable-diffusion\\realisticVisionV50_v40VAE.safetensors' is not in
the subpath of 'C:\\sd\\animatediff-cli' OR one path is relative and the other is absolute.

I can't for the life of me figure out how to correctly specify an absolute path to a model. It used to work in the previous versions.

Another note: It would be great if in this case forward slash could be treated like a part of the path, and not as an escape symbol. Copying paths in Windows gives you forward slashes and you have to change them to backslashes all the time.

This works in the main branch:

Is the error I get on this branch:

As per title.
Thanks!