Comments (9)
Hi! As for the first point, there's a webui plugin for Auto1111 https://github.com/deforum-art/sd-webui-text2video with a GUI where you can specify anything for your generation. To convert your finetuned models to use in that GUI, use the script in this repo https://github.com/ExponentialML/Text-To-Video-Finetuning/blob/main/utils/convert_diffusers_to_original_ms_text_to_video.py
from text-to-video-finetuning.
Thank you kabachuha, for the convert_diffusers_to_original_ms_text_to_video.py, what Arguments do I need to put in? Should I put the root folder of the model, or link directly to the bin files for the Unet and text encoder, and do I need to specify an output folder? Thank you!
from text-to-video-finetuning.
python convert_diffusers_to_original_ms_text_to_video.py --model_path path-to-your-diffusers-model-folder --checkpoint_path text2video_pytorch_model.pth --clip_checkpoint_path clip.ckpt
. Don't use this clip.ckpt
, it's not converted well at the moment, and I need to remove it from requirements
from text-to-video-finetuning.
So should I put in the clip checkpoint path and just not used the clip file that is created, or should I leave the clip checkpoint path blank?
from text-to-video-finetuning.
@justinwking use this branch for now, before it's merged https://github.com/kabachuha/Text-To-Video-Finetuning/tree/patch-1
from text-to-video-finetuning.
Sorry to ask such basic questions..... but I couldn't find the files you suggested I include. So I am guessing they have a different name, at the bottom of this post, I created an interpretation of what I think you meant, please correct me if I am mistaken. If this is my folder structure....
Text to video Fine Tuning
- [ ] Models
- Model_scope_diffusers
- Scheduler - Text_encoder - Tokenizer - Unet - Vae
``- [ ] Outputs
- Train 2023….
- Cached Latents - CHECKPOINT 2500 - Checkpoint 5000
- Scheduler - Text-encoder - Tokenizer - Unet - Vae
- Lora - Samples
Does the following command look correct if I do everything from the text_to_finetuning folder....
python .Utils/convert_diffusers_to_original_ms_text_to_video.py --model_path models/model_scope_diffusers/ --checkpoint_path outputs/Train2003…/Lora/5000_unet.pt --clip_checkpoint_path outputs/Train2003…/Lora/5000_text_encoder.pt
from text-to-video-finetuning.
Use this folder as models_path "./Outputs/Train 2023…./Checkpoint 5000"
from text-to-video-finetuning.
Good morning, I believe I was able to get the script to work with your instructions, but I didn't see a new folder created. What do I need to do to get this into a format and location that t2v can use? All the file names are different, and the folder structures is different. Is this something that the script could do?
from text-to-video-finetuning.
I haven't been able to find a readme that explains the process, maybe there is one that I overlooked.
the following was generated when I did the training
Configuration saved in ./outputs\train_2023-04-24T00-05-34\vae\config.json
Model weights saved in ./outputs\train_2023-04-24T00-05-34\vae\diffusion_pytorch_model.bin
Configuration saved in ./outputs\train_2023-04-24T00-05-34\unet\config.json
Model weights saved in ./outputs\train_2023-04-24T00-05-34\unet\diffusion_pytorch_model.bin
Configuration saved in ./outputs\train_2023-04-24T00-05-34\scheduler\scheduler_config.json
Configuration saved in ./outputs\train_2023-04-24T00-05-34\model_index.json
04/24/2023 06:13:39 - INFO - main - Saved model at ./outputs\train_2023-04-24T00-05-34 on step 10000
then I put in the command,
(text2video-finetune) python ./Utils/convert_diffusers_to_original_ms_text_to_video.py --model_path "./Outputs/train_2023-04-24T00-05-34/Checkpoint-10000"--checkpoint_path "./Outputs/train_2023-04-24T00-05-34/Lora/10000_unet.pt" --clip_checkpoint_path "./Outputs/train_2023-04-24T00-05-34/Lora/10000_text_encoder.pt"
and the process worked, but I don't know where the new UNET is...
Saving UNET
Operation successfull
But now.... I don't see anything that looks like the modelscope folder that I am currently using in Automatic1111
configuration.json
open_clip_pytorch_model.bin
README.md
text2video_pytorch_model.pth
VQGAN_autoencoder.pth
from text-to-video-finetuning.
Related Issues (20)
- webui Lora Might be causing errors in checkpoint models. HOT 3
- How to train with folder video HOT 1
- Which paper? HOT 1
- RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1, 512] because the unspecified dimension size -1 can be any value and is ambiguous HOT 3
- Does this code support native finetune for damo text to video model? HOT 2
- AttributeError: 'Tensor' object has no attribute 'config' HOT 5
- How can I run the fine-tuning on a GPU with <= 16GB of VRAM? HOT 3
- I have some doubts about the framework HOT 4
- A typo
- TypeError: Linear.forward() got an unexpected keyword argument 'scale' HOT 6
- wrong norm method HOT 1
- issues on train.py HOT 1
- [inference] latents_window index error HOT 1
- Two forward passes in finetune_unet HOT 1
- Lora on ResnetBlock2D in modelscope model HOT 1
- Can the videocomposer model be adapted to this training framework?
- Normal finetuning instead of LoRA
- ControlNet
- init_video problem
- finetune train error of "UnboundLocalError: local variable 'use_offset_noise' referenced before assignment" HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from text-to-video-finetuning.