current code is not compatible with new diffusers about custom-diffusion HOT 9 CLOSED

Hi,
I have updated the diffusers code with the updated version i.e. 0.14.0. I still need to try it on all the datasets, but qualitatively it looks to have somewhat similar results.
Thanks!

from custom-diffusion.

Hi, thanks a lot for looking into it. It's quite interesting that even with the new_forward in the updated diffusers version, the quality is not the same and requires longer training.
I also plan to work on making our code compatible with the new diffusers version.
I will post any findings here.

from custom-diffusion.

Any updates on this?

from custom-diffusion.

thank you for updating the code!
current tests look promising, and xformers support makes it back on track memory- and speed-wise.

from custom-diffusion.

When I ran the custom diffusion code with diffusers 0.22.0. I am getting following error:

Steps: 100%|██████████| 250/250 [04:12<00:00,  1.05it/s, loss=1.03, lr=4e-5]
�Traceback (most recent call last):
  File "/workspace/customdiffusion/diffusers/examples/custom_diffusion/train_custom_diffusion.py", line 1340, in <module>
    main(args)
  File "/workspace/customdiffusion/diffusers/examples/custom_diffusion/train_custom_diffusion.py", line 1268, in main
    unet.save_attn_procs(args.output_dir, safe_serialization=not args.no_safe_serialization)
  File "/workspace/customdiffusion/diffusers/src/diffusers/loaders.py", line 674, in save_attn_procs
    save_function(state_dict, os.path.join(save_directory, weight_name))
  File "/workspace/customdiffusion/diffusers/src/diffusers/loaders.py", line 630, in save_function
    return safetensors.torch.save_file(weights, filename, metadata={"format": "pt"})
  File "/opt/conda/lib/python3.8/site-packages/safetensors/torch.py", line 281, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
  File "/opt/conda/lib/python3.8/site-packages/safetensors/torch.py", line 449, in _flatten
    raise ValueError(f"Key `{k}` is invalid, expected torch.Tensor but received {type(v)}")
ValueError: Key `down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor` is invalid, expected torch.Tensor but received <class 'dict'>

I ran the code for single concept.
I even tried with diffusers version 0.16.0 but the error persists.
Is there specific version of diffusers for which custom-diffusion code will run?
Any help in resolving this error is highly appreciated.

from custom-diffusion.

Hi, I have updated our code for more recent diffusers version 0.21.4. Please let me know if you still face any issues. You can find the requirements here. Thanks!

from custom-diffusion.

Hi, I have updated our code for more recent diffusers version 0.21.4. Please let me know if you still face any issues. You can find the requirements here. Thanks!

Hi @nupurkmr9, thanks for specifying the requirements. Even after using the same versions (listed below) I am getting same error.

accelerate==0.24.1
torch==2.0.1
torchvision==0.15.2
transformers==4.31.0
diffusers==0.21.4

Following is the error:

Traceback (most recent call last):
  File "/workspace/customdiffusion/diffusers/examples/custom_diffusion/train_custom_diffusion.py", line 1332, in <module>
    main(args)
  File "/workspace/customdiffusion/diffusers/examples/custom_diffusion/train_custom_diffusion.py", line 1260, in main
    unet.save_attn_procs(args.output_dir, safe_serialization=not args.no_safe_serialization)
  File "/workspace/customdiffusion/diffusers/src/diffusers/loaders.py", line 634, in save_attn_procs
    save_function(state_dict, os.path.join(save_directory, weight_name))
  File "/workspace/customdiffusion/diffusers/src/diffusers/loaders.py", line 600, in save_function
    return safetensors.torch.save_file(weights, filename, metadata={"format": "pt"})
  File "/opt/conda/lib/python3.8/site-packages/safetensors/torch.py", line 281, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
  File "/opt/conda/lib/python3.8/site-packages/safetensors/torch.py", line 449, in _flatten
    raise ValueError(f"Key `{k}` is invalid, expected torch.Tensor but received {type(v)}")
ValueError: Key `down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor` is invalid, expected torch.Tensor but received <class 'dict'>

from custom-diffusion.

Hi, I have updated our code for more recent diffusers version 0.21.4. Please let me know if you still face any issues. You can find the requirements here. Thanks!

Hi @nupurkmr9, thanks for specifying the requirements. Even after using the same versions (listed below) I am getting same error.

accelerate==0.24.1
torch==2.0.1
torchvision==0.15.2
transformers==4.31.0
diffusers==0.21.4

Following is the error:

Traceback (most recent call last):
  File "/workspace/customdiffusion/diffusers/examples/custom_diffusion/train_custom_diffusion.py", line 1332, in <module>
    main(args)
  File "/workspace/customdiffusion/diffusers/examples/custom_diffusion/train_custom_diffusion.py", line 1260, in main
    unet.save_attn_procs(args.output_dir, safe_serialization=not args.no_safe_serialization)
  File "/workspace/customdiffusion/diffusers/src/diffusers/loaders.py", line 634, in save_attn_procs
    save_function(state_dict, os.path.join(save_directory, weight_name))
  File "/workspace/customdiffusion/diffusers/src/diffusers/loaders.py", line 600, in save_function
    return safetensors.torch.save_file(weights, filename, metadata={"format": "pt"})
  File "/opt/conda/lib/python3.8/site-packages/safetensors/torch.py", line 281, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
  File "/opt/conda/lib/python3.8/site-packages/safetensors/torch.py", line 449, in _flatten
    raise ValueError(f"Key `{k}` is invalid, expected torch.Tensor but received {type(v)}")
ValueError: Key `down_blocks.0.attentions.0.transformer_blocks.0.attn1.processor` is invalid, expected torch.Tensor but received <class 'dict'>

So this is the issue of safe serialization, using --no_safe_serialization will resolve the issue.

from custom-diffusion.

Hi @nupurkmr9, great work.
Unfortunately, the code still fails merging concepts with src/diffusers_composenW.py.
Any update of that script would be much appreciated.

from custom-diffusion.