Clear Instructions for Finetuning Faces about custom-diffusion HOT 6 OPEN

Hi,
Thanks for your interest in our work. I have listed below the answers to your questions. Let me know if you have any other questions.

class_prompt: the category of the training images, e.g., person for human faces.
instance_prompt: the text prompt used during training on the target training images, i.e., photo of a <new1> person
training time: all our experiments in the paper were with stable-diffusion-1.4 and 2A100 GPUs, on which it takes ~6 minutes and 30GB VRAM. For human faces, a lower learning rate and more training steps are better, which can then take ~18 minutes of training. VRAM requirement can be reduced by using less batch size but more training time, or using deepspeed library with accelarate. Can you mention which GPU configuration you are training the model on?
regularization images: we collect real images from the LAION dataset using clip-retrieval with similar captions as class_prompt before starting the training. This can take some time, depending on the class prompt.
Number of images for human faces: I haven't benchmarked the performance on human faces extensively. But in my experiments, 15-20 high-resolution images should be good, more is better. An example result is shown here.

For training, using these configurations should be good.

accelerate launch src/diffuser_training.py \
          --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4"  \
          --instance_data_dir=./data/cat  \
          --class_data_dir=./real_reg/samples_cat/ \
          --output_dir=./logs/cat  \
          --with_prior_preservation --real_prior --prior_loss_weight=1.0 \
          --instance_prompt="photo of a <new1> person"  \
          --class_prompt="person" \
          --resolution=512  \
          --train_batch_size=2  \
          --learning_rate=5e-6  \
          --lr_warmup_steps=0 \
          --max_train_steps=750 \
          --num_class_images=200 \
          --scale_lr \
          --modifier_token "<new1>"

from custom-diffusion.

@nupurkmr9 If you check regularization images in real_reg/samples_person, you will find that those images are very bad. Most of them are not real faces.

from custom-diffusion.

Hi,
Thanks for pointing this out. Can you try increasing the aesthetic weight in this line? Increasing the weight leads to more time in retrieving the images.
I will update the readme/code to allow that with a flag easily.

Also, our method works with generated images as regularization as well which can be enabled by removing the --real_prior flag from the command.

from custom-diffusion.

Hi, Thanks for your interest in our work. I have listed below the answers to your questions. Let me know if you have any other questions.

class_prompt: the category of the training images, e.g., person for human faces. instance_prompt: the text prompt used during training on the target training images, i.e., photo of a <new1> person training time: all our experiments in the paper were with stable-diffusion-1.4 and 2A100 GPUs, on which it takes ~6 minutes and 30GB VRAM. For human faces, a lower learning rate and more training steps are better, which can then take ~18 minutes of training. VRAM requirement can be reduced by using less batch size but more training time, or using deepspeed library with accelarate. Can you mention which GPU configuration you are training the model on? regularization images: we collect real images from the LAION dataset using clip-retrieval with similar captions as class_prompt before starting the training. This can take some time, depending on the class prompt. Number of images for human faces: I haven't benchmarked the performance on human faces extensively. But in my experiments, 15-20 high-resolution images should be good, more is better. An example result is shown here.

For training, using these configurations should be good.

accelerate launch src/diffuser_training.py \
          --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4"  \
          --instance_data_dir=./data/cat  \
          --class_data_dir=./real_reg/samples_cat/ \
          --output_dir=./logs/cat  \
          --with_prior_preservation --real_prior --prior_loss_weight=1.0 \
          --instance_prompt="photo of a <new1> person"  \
          --class_prompt="person" \
          --resolution=512  \
          --train_batch_size=2  \
          --learning_rate=5e-6  \
          --lr_warmup_steps=0 \
          --max_train_steps=750 \
          --num_class_images=200 \
          --scale_lr \
          --modifier_token "<new1>"

Hi,

I appreciate your work. I used same command as you mentioned here.
Model takes at least 3000 steps to recover the identity loss.
Not produce good results on too long prompts.
Finetune with "runwayml/stable-diffusion-v1-5" producing good results than 2.1 version but not too good.

Some of the results comparison between 1.5 vs 2.1 versions.
1.5 prompts:
Ultra HD quality of styriodio woman as god of war character with beautiful open eyes
beautiful styriodio woman as viking.

Best Results:

2.1 prompts:
Ultra HD quality of styriodio woman as god of war character with beautiful open eyes
beautiful styriodio woman as viking.

Best Results:

On long prompts:
Portrait painting of styriodio as a pretty goth girl, gorgeous tattoos, beautiful hair, symmetrical! intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha

Total identity loss on Emma Watson Images. But dreambooth working fine on these long prompts.

Can you please guide me how I can improve the model on long prompts.
Also guide me about how I can improve the results. Because after 3000 steps still there are some identity loss.
Looking forward to your reply thanks.

from custom-diffusion.

@nupurkmr9 Thanks for amazing wonderful work, I used the these configurations, but the result is bad.

accelerate launch src/diffuser_training.py
--pretrained_model_name_or_path=$MODEL_NAME
--instance_data_dir=$INSTANCE_DIR
--class_data_dir=$CLASS_DIR
--output_dir=$OUTPUT_DIR
--with_prior_preservation --real_prior --prior_loss_weight=1.0
--instance_prompt="photo of a V* person"
--class_prompt="person"
--resolution=512
--train_batch_size=2
--learning_rate=5e-6
--lr_warmup_steps=0
--max_train_steps=3000
--num_class_images=200
--scale_lr --hflip
--mixed_precision "no"
--modifier_token " V*"

from custom-diffusion.

Hi,

I'm sorry for the delayed response in this thread. I trained a model on 16 face photos with the following configurations on 2 GPUs. Training on faces with the --freeze_model crossattn configuration is usually better than the default setting.

export MODEL_NAME="CompVis/stable-diffusion-v1-4"
CUDA_VISIBLE_DEVICES=1,2 accelerate launch src/diffuser_training.py \
          --pretrained_model_name_or_path=$MODEL_NAME  \
          --instance_data_dir=./face_photos  \
          --class_data_dir=./real_reg/samples_person  \
          --output_dir=./face_model  \
          --with_prior_preservation --real_prior --prior_loss_weight=1.0 \
          --instance_prompt="photo of a <new1> person"  \
          --class_prompt="person" \
          --resolution=512  \
          --train_batch_size=2  \
          --learning_rate=5e-6  \
          --lr_warmup_steps=0 \
          --max_train_steps=2000 \
          --num_class_images=200 \
          --scale_lr  --hflip \
          --freeze_model crossattn \
          --modifier_token "\<new1\>"

And these are the results with sampling done as python src/sample_diffuser.py --prompt <text-prompt> --delta_ckpt <path-to/delta.bin> --ckpt "CompVis/stable-diffusion-v1-4" --freeze_model crossattn --batch_size 10. I selected the best 3 out of 10 generations. I used two prompts from the above post.

from custom-diffusion.