finegrain-ai / refiners Goto Github PK
View Code? Open in Web Editor NEWA microframework on top of PyTorch with first-class citizen APIs for foundation model adaptation
Home Page: https://refine.rs
License: MIT License
refiners's People
Contributors
Stargazers
Watchers
Forkers
techthiyanes mathieunlp devo-ody dezigns333 brycedrennan isamu-isozaki youssefadr chloedia charchit7 laurent2916 israfelsr julien-blanchon gabrielvidal1 leryud quocdat-le-insacvl adi-kmt steveefemsc zaferozerr piercus evdcush rodsiry baptlem louisrouss antonyashik nezlheimeur sanketsans ayoubmdl qq332067275 amo5 ily-rrefiners's Issues
DropoutAdapter injection is impacting state_dict().keys and the ability to load/save checkpoints smoothly
Hello refiners,
I'm experimenting with trainer and especially I'm facing a problem to load/save models weights
The sequence of the trainer is :
trainer.prepare_modelsis loading the checkpoint on a non-injected modelon_train_beginis injecting the dropout_adapteron_checkpoint_saveis saving the checkpoint (usingmodel.state_dict())
The named of the Dropout-impacted layers are changed in step 2.
As a result, the model saved in on_checkpoint_save are not compatible with the loading in trainer.prepare_models, and i cannot smootly save/load the model.
Toy example
The injection of the dropout adpater is changing the keys of weights in state_dict()
from refiners.fluxion.layers.chain import Chain
from refiners.fluxion.layers.linear import Linear
from refiners.training_utils.dropout import DropoutAdapter
network = Chain(
Linear(2, 3)
)
keys = network.state_dict().keys()
print(keys)
probability=0.5
for linear, parent in network.walk(Linear):
DropoutAdapter(target=linear, probability=probability).inject(parent)
keys2 = network.state_dict().keys()
print(keys2)is outputing
odict_keys(['Linear.weight', 'Linear.bias'])
odict_keys(['DropoutAdapter.Linear.weight', 'DropoutAdapter.Linear.bias'])
What i'm not clear is what is the target behavior
A. should .inject(parent) change the name of the weights and we should fix the save/load sequence in the trainer ?
B. should .inject(parent) not change the name of the weights in state_dict() when the adapter is not injecting new weights ?
I can help on this if needed
Injecting/ejecting multiple controlnet models results in an error
I was trying to inject -> eject -> inject two controlnet models but this doesn't seem to work properly.
Reproduction:
from refiners.foundationals.latent_diffusion import (
StableDiffusion_1
)
from refiners.foundationals.latent_diffusion.solvers import DDIM
from refiners.foundationals.latent_diffusion import SD1ControlnetAdapter
from refiners.fluxion.utils import load_from_safetensors
import torch
device, dtype = ("cuda", torch.float16)
solver = DDIM(num_inference_steps=10)
sd = StableDiffusion_1(device=device, dtype=dtype, solver=solver)
sd.clip_text_encoder.load_from_safetensors("../weights/sd-text-encoder.safetensors")
sd.lda.load_from_safetensors("../weights/sd-lda.safetensors")
sd.unet.load_from_safetensors("../weights/sd-unet.safetensors")
controlnet = {
"lineart": SD1ControlnetAdapter(
sd.unet, name="lineart", scale=1.0, weights=load_from_safetensors("../weights/cn-lineart.safetensors")
).to(device, dtype),
"canny": SD1ControlnetAdapter(
sd.unet, name="canny", scale=1.0, weights=load_from_safetensors("../weights/cn-canny.safetensors")
).to(device, dtype),
}
if controlnet:
for value in controlnet.values():
value.inject()
if controlnet:
for value in controlnet.values():
value.eject()
# Error
if controlnet:
for value in controlnet.values():
value.inject()Stack trace:
"name": "AssertionError",
"message": "SD1UNet(in_channels=4) not in SD1ControlnetAdapter(name=canny)",
"stack": "---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
File /home/azureuser/cloudfiles/code/Users/joachim/migrate to refiners/inject_eject.py:4
2 if controlnet:
3 for value in controlnet.values():
----> 4 value.inject()
File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/refiners/foundationals/latent_diffusion/stable_diffusion_1/controlnet.py:164, in SD1ControlnetAdapter.inject(self, parent)
162 assert cn.name != self.name, f\"Controlnet named {self.name} is already injected\"
163 self.target.insert(0, controlnet)
--> 164 return super().inject(parent)
File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/refiners/fluxion/adapters/adapter.py:78, in Adapter.inject(self, parent)
75 # In general, `true_parent` is `parent`. We do this to support multiple adaptation,
76 # i.e. initializing two adapters before injecting them.
77 true_parent = parent.ensure_find_parent(self.target)
---> 78 true_parent.replace(
79 old_module=self.target,
80 new_module=self,
81 old_module_parent=target_parent,
82 )
83 return self
File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/refiners/fluxion/layers/chain.py:607, in Chain.replace(self, old_module, new_module, old_module_parent)
605 new_module._set_parent(self)
606 if isinstance(old_module, ContextModule):
--> 607 old_module._set_parent(old_module_parent)
File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/refiners/fluxion/layers/module.py:187, in ContextModule._set_parent(self, parent)
185 return
186 # Always insert the module in the Chain first to avoid inconsistencies.
--> 187 assert self in iter(parent), f\"{self} not in {parent}\"
188 self._parent = [parent]
AssertionError: SD1UNet(in_channels=4) not in SD1ControlnetAdapter(name=canny)"inpaint pipelines
Hi!
Thank you for your great job!!
Do you provide a possibility to use inpainting pipelines (StableDiffusionInpaintPipeline)?
feature: support converting sdwebui Lora format to refiners
I got pokemon Lora example working but it seems most loras are in the larger sdwebui format that modifies more than just the unet.
This one for example:
https://civitai.com/models/7109/modern-disney-lora
Failed to run "Hello World" example
Hi,
I tried to run your "Hello World" example but it fails with the following settings:
- OS: Ubuntu 22.04.2 LTS
- Python: 3.10.12
- CUDA 11.5 and 11.8
- PyTorch CUDA 2.0.0 and 2.0.1
- Refiners from PYPI: 0.1.0
- GPU: Nvidia T4
- CPU: AWS 4-core instance
Here is the log: https://github.com/cknowledge/cm-reproduce/blob/main/script/run-refiners-hello-world/README-extra.md#log .
Can you please suggest me what happens? Maybe it's something obvious ...
Thank you for your project!
CC @arjunsuresh
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.