Comments (6)
Hi,
Thanks for your interest in our work. I have listed below the answers to your questions. Let me know if you have any other questions.
class_prompt: the category of the training images, e.g., person
for human faces.
instance_prompt: the text prompt used during training on the target training images, i.e., photo of a <new1> person
training time: all our experiments in the paper were with stable-diffusion-1.4 and 2A100 GPUs, on which it takes ~6 minutes and 30GB VRAM. For human faces, a lower learning rate and more training steps are better, which can then take ~18 minutes of training. VRAM requirement can be reduced by using less batch size but more training time, or using deepspeed
library with accelarate
. Can you mention which GPU configuration you are training the model on?
regularization images: we collect real images from the LAION dataset using clip-retrieval with similar captions as class_prompt before starting the training. This can take some time, depending on the class prompt.
Number of images for human faces: I haven't benchmarked the performance on human faces extensively. But in my experiments, 15-20 high-resolution images should be good, more is better. An example result is shown here.
For training, using these configurations should be good.
accelerate launch src/diffuser_training.py \
--pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" \
--instance_data_dir=./data/cat \
--class_data_dir=./real_reg/samples_cat/ \
--output_dir=./logs/cat \
--with_prior_preservation --real_prior --prior_loss_weight=1.0 \
--instance_prompt="photo of a <new1> person" \
--class_prompt="person" \
--resolution=512 \
--train_batch_size=2 \
--learning_rate=5e-6 \
--lr_warmup_steps=0 \
--max_train_steps=750 \
--num_class_images=200 \
--scale_lr \
--modifier_token "<new1>"
from custom-diffusion.
@nupurkmr9 If you check regularization images in real_reg/samples_person, you will find that those images are very bad. Most of them are not real faces.
from custom-diffusion.
Hi,
Thanks for pointing this out. Can you try increasing the aesthetic weight in this line? Increasing the weight leads to more time in retrieving the images.
I will update the readme/code to allow that with a flag easily.
Also, our method works with generated images as regularization as well which can be enabled by removing the --real_prior
flag from the command.
from custom-diffusion.
Hi, Thanks for your interest in our work. I have listed below the answers to your questions. Let me know if you have any other questions.
class_prompt: the category of the training images, e.g.,
person
for human faces. instance_prompt: the text prompt used during training on the target training images, i.e.,photo of a <new1> person
training time: all our experiments in the paper were with stable-diffusion-1.4 and 2A100 GPUs, on which it takes ~6 minutes and 30GB VRAM. For human faces, a lower learning rate and more training steps are better, which can then take ~18 minutes of training. VRAM requirement can be reduced by using less batch size but more training time, or usingdeepspeed
library withaccelarate
. Can you mention which GPU configuration you are training the model on? regularization images: we collect real images from the LAION dataset using clip-retrieval with similar captions as class_prompt before starting the training. This can take some time, depending on the class prompt. Number of images for human faces: I haven't benchmarked the performance on human faces extensively. But in my experiments, 15-20 high-resolution images should be good, more is better. An example result is shown here.For training, using these configurations should be good.
accelerate launch src/diffuser_training.py \ --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" \ --instance_data_dir=./data/cat \ --class_data_dir=./real_reg/samples_cat/ \ --output_dir=./logs/cat \ --with_prior_preservation --real_prior --prior_loss_weight=1.0 \ --instance_prompt="photo of a <new1> person" \ --class_prompt="person" \ --resolution=512 \ --train_batch_size=2 \ --learning_rate=5e-6 \ --lr_warmup_steps=0 \ --max_train_steps=750 \ --num_class_images=200 \ --scale_lr \ --modifier_token "<new1>"
Hi,
I appreciate your work. I used same command as you mentioned here.
Model takes at least 3000 steps to recover the identity loss.
Not produce good results on too long prompts.
Finetune with "runwayml/stable-diffusion-v1-5" producing good results than 2.1 version but not too good.
Some of the results comparison between 1.5 vs 2.1 versions.
1.5 prompts:
Ultra HD quality of styriodio woman as god of war character with beautiful open eyes
beautiful styriodio woman as viking.
2.1 prompts:
Ultra HD quality of styriodio woman as god of war character with beautiful open eyes
beautiful styriodio woman as viking.
On long prompts:
Portrait painting of styriodio as a pretty goth girl, gorgeous tattoos, beautiful hair, symmetrical! intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha
Total identity loss on Emma Watson Images. But dreambooth working fine on these long prompts.
Can you please guide me how I can improve the model on long prompts.
Also guide me about how I can improve the results. Because after 3000 steps still there are some identity loss.
Looking forward to your reply thanks.
from custom-diffusion.
@nupurkmr9 Thanks for amazing wonderful work, I used the these configurations, but the result is bad.
accelerate launch src/diffuser_training.py
--pretrained_model_name_or_path=$MODEL_NAME
--instance_data_dir=$INSTANCE_DIR
--class_data_dir=$CLASS_DIR
--output_dir=$OUTPUT_DIR
--with_prior_preservation --real_prior --prior_loss_weight=1.0
--instance_prompt="photo of a V* person"
--class_prompt="person"
--resolution=512
--train_batch_size=2
--learning_rate=5e-6
--lr_warmup_steps=0
--max_train_steps=3000
--num_class_images=200
--scale_lr --hflip
--mixed_precision "no"
--modifier_token " V*"
from custom-diffusion.
Hi,
I'm sorry for the delayed response in this thread. I trained a model on 16 face photos with the following configurations on 2 GPUs. Training on faces with the --freeze_model crossattn
configuration is usually better than the default setting.
export MODEL_NAME="CompVis/stable-diffusion-v1-4"
CUDA_VISIBLE_DEVICES=1,2 accelerate launch src/diffuser_training.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--instance_data_dir=./face_photos \
--class_data_dir=./real_reg/samples_person \
--output_dir=./face_model \
--with_prior_preservation --real_prior --prior_loss_weight=1.0 \
--instance_prompt="photo of a <new1> person" \
--class_prompt="person" \
--resolution=512 \
--train_batch_size=2 \
--learning_rate=5e-6 \
--lr_warmup_steps=0 \
--max_train_steps=2000 \
--num_class_images=200 \
--scale_lr --hflip \
--freeze_model crossattn \
--modifier_token "\<new1\>"
And these are the results with sampling done as python src/sample_diffuser.py --prompt <text-prompt> --delta_ckpt <path-to/delta.bin> --ckpt "CompVis/stable-diffusion-v1-4" --freeze_model crossattn --batch_size 10
. I selected the best 3 out of 10 generations. I used two prompts from the above post.
from custom-diffusion.
Related Issues (20)
- Errors in using diffuser version HOT 3
- A question for the regularization captions HOT 1
- questions about crossattn in CustomDiffusionAttnProcessor
- Question about the cross attention HOT 1
- ImportError: cannot import name 'insecure_hashlib' from 'huggingface_hub.utils' 怎么解决? HOT 1
- Hi, How toMultiple-Concept Fine-tuning for DreamBooth?
- if it OK with no regularization? my results don't have any consistancy with reference images. HOT 3
- Ours (w/fine-tune all) ? HOT 2
- OOM while running sample.py HOT 4
- why my optimization based weights merging is not work? HOT 4
- Is it possible to train multi joint training with SDXL? diffusers_training_sdxl.py seems impossible for multi joint training. HOT 2
- Question about loss on valid image region during training HOT 1
- Missing text prompts? HOT 2
- Does the diffusers version now work properly? HOT 3
- why the results produced by the model training and verification are inconsistent with the results I test using the saved model weights? HOT 1
- Evaluation:where is the “prompts.json”file?
- Unable to retrieve the image used for regularization. HOT 1
- Is this version trained with lora? HOT 1
- Tensors not Updated after Training HOT 1
- retrieve.py question HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from custom-diffusion.