mikonvergence / controlnetinpaint Goto Github PK

The StableDiffusionInpaintPipeline introduces a strength parameter, as detailed in the documentation here. However, I couldn't locate this parameter in the StableDiffusionControlNetInpaintPipeline.

If I use the parameters num_inference_steps=40 and strength=0.93 in StableDiffusionInpaintPipeline, should I then use num_inference_steps=37 (calculated as 40 * 0.93) in StableDiffusionControlNetInpaintPipeline?

Instead on conditioning on a text prompt, can we somehow use controlnet to condition on another image and replace an masked area with this new image?

Thanks

Hello @mikonvergence, your work is awesome and i have a query regarding an issue which is rendering in my brain from days.

I have 10-15 different prompts and i want to infer on a single image, also with T4 GPU, the GPU goes into fragments for single image and single prompt.

Thanks and Regards,
Satwik Sunnam.

please tell me if i could fine tune your model with our dataset and give our objects a special prompt?

Hello,

I'm trying to use the provided Google Colab file to mask out a piece of cloth from the original image of a person wearing cloth and change the cloth with a textual prompt (like color for eg), but I'm encountering issues with the generated image. Specifically, the generated image appears to be of poor quality and has a mixed-up appearance.

Here are my inputs:
A person wearing a cloth.

A person wearing a grey cloth (representing no cloth).

Prompt

text_prompt="A woman wearing a green shirt"

It seems intuitive, however, the output image I'm receiving is not what I expected. I've followed the instructions provided in the repo, but I'm still unable to achieve satisfactory results.

OUTPUT

note:

1. I tried converting the grey color of the mask image to black to see if it yields any better results, but it did not, unfortunately.
2. I tried the canny with image and mask image to see any differences, but the generated image was still like this.

Could you please provide some guidance on how to improve the output image quality? If there are any known issues or limitations with the current implementation, please let me know as well.

Cheers
Seth

Every model except canny is giving this error:

/content/./ControlNetInpaint/src/pipeline_stable_diffusion_controlnet_inpaint.py in prepare_mask_and_masked_image(image, mask)
166 mask = torch.from_numpy(mask)
167
--> 168 masked_image = image * (mask < 0.5)
169
170 return mask, masked_image

RuntimeError: The size of tensor a (612) must match the size of tensor b (3) at non-singleton dimension 4

Hi! Thank you for this repo.

I did not understand if you retrained ControlNet using the SD-inpainting backbone, or if you copied over the weight that were trained for the regular SD-backbone by ControlNot authors, and those weights somehow work on the SD-inpainting backbone as well?

Thank you very much,
Thibault

I'd like to add support for controlnet inpainting in ComfyUI - so that we can use it in the AI horde - but none of our pipelines are using diffusers. Any idea how this can be done with the RunwayML version of stable diffusion?

pipe.to('cuda')

# generate image
generator = torch.manual_seed(0)
new_image = pipe(
    text_prompt,
    num_inference_steps=20,
    generator=generator,
    image=image,
    control_image=canny_image,
    controlnet_conditioning_scale = 0.5,
    mask_image=mask_image
).images[0]

new_image.save('output/canny_result.png')

Thanks for your great work, by running the above code in notebook, I get some issues:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:5                                                                                    │
│                                                                                                  │
│    2                                                                                             │
│    3 # generate image                                                                            │
│    4 generator = torch.manual_seed(0)                                                            │
│ ❱  5 new_image = pipe(                                                                           │
│    6 │   text_prompt,                                                                            │
│    7 │   num_inference_steps=20,                                                                 │
│    8 │   generator=generator,                                                                    │
│                                                                                                  │
│ d:\App\miniconda\envs\aigc\lib\site-packages\torch\autograd\grad_mode.py:27 in decorate_context  │
│                                                                                                  │
│    24 │   │   @functools.wraps(func)                                                             │
│    25 │   │   def decorate_context(*args, **kwargs):                                             │
│    26 │   │   │   with self.clone():                                                             │
│ ❱  27 │   │   │   │   return func(*args, **kwargs)                                               │
│    28 │   │   return cast(F, decorate_context)                                                   │
│    29 │                                                                                          │
│    30 │   def _wrap_generator(self, func):                                                       │
│                                                                                                  │
│ c:\Users\Arthur\Downloads\ControlNetInpaint-main\ControlNetInpaint-main\src\pipeline_stable_diff │
│ usion_controlnet_inpaint.py:394 in __call__                                                      │
│                                                                                                  │
│   391 │   │   )                                                                                  │
│   392 │   │                                                                                      │
│   393 │   │   # 4. Prepare image                                                                 │
│ ❱ 394 │   │   control_image = self.prepare_image(                                                │
│   395 │   │   │   control_image,                                                                 │
│   396 │   │   │   width,                                                                         │
│   397 │   │   │   height,                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: StableDiffusionControlNetPipeline.prepare_image() missing 1 required positional argument: 
'do_classifier_free_guidance'

RuntimeError Traceback (most recent call last)
Cell In[9], line 4
1 from controlnet_aux import OpenposeDetector
3 openpose = OpenposeDetector.from_pretrained('lllyasviel/ControlNet')
----> 4 pose_image = openpose(image)
5 pose_image

File /home/pai/lib/python3.9/site-packages/controlnet_aux/open_pose/init.py:83, in OpenposeDetector.call(self, input_image, detect_resolution, image_resolution, hand_and_face, return_pil)
81 H, W, C = input_image.shape
82 with torch.no_grad():
---> 83 candidate, subset = self.body_estimation(input_image)
84 hands = []
85 faces = []

File /home/pai/lib/python3.9/site-packages/controlnet_aux/open_pose/body.py:44, in Body.call(self, oriImg)
42 # data = data.permute([2, 0, 1]).unsqueeze(0).float()
43 with torch.no_grad():
---> 44 Mconv7_stage6_L1, Mconv7_stage6_L2 = self.model(data)
45 Mconv7_stage6_L1 = Mconv7_stage6_L1.cpu().numpy()
46 Mconv7_stage6_L2 = Mconv7_stage6_L2.cpu().numpy()

File /home/pai/lib/python3.9/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
1496 # If we don't have any hooks, we want to skip the rest of the logic in
1497 # this function, and just call forward.
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1499 or _global_backward_pre_hooks or _global_backward_hooks
1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
1502 # Do not call functions when jit is used
1503 full_backward_hooks, non_full_backward_hooks = [], []

File /home/pai/lib/python3.9/site-packages/controlnet_aux/open_pose/model.py:116, in bodypose_model.forward(self, x)
114 def forward(self, x):
--> 116 out1 = self.model0(x)
118 out1_1 = self.model1_1(out1)
119 out1_2 = self.model1_2(out1)

File /home/pai/lib/python3.9/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
1496 # If we don't have any hooks, we want to skip the rest of the logic in
1497 # this function, and just call forward.
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1499 or _global_backward_pre_hooks or _global_backward_hooks
1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
1502 # Do not call functions when jit is used
1503 full_backward_hooks, non_full_backward_hooks = [], []

File /home/pai/lib/python3.9/site-packages/torch/nn/modules/container.py:217, in Sequential.forward(self, input)
215 def forward(self, input):
216 for module in self:
--> 217 input = module(input)
218 return input

File /home/pai/lib/python3.9/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
1496 # If we don't have any hooks, we want to skip the rest of the logic in
1497 # this function, and just call forward.
1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1499 or _global_backward_pre_hooks or _global_backward_hooks
1500 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501 return forward_call(*args, **kwargs)
1502 # Do not call functions when jit is used
1503 full_backward_hooks, non_full_backward_hooks = [], []

File /home/pai/lib/python3.9/site-packages/torch/nn/modules/conv.py:463, in Conv2d.forward(self, input)
462 def forward(self, input: Tensor) -> Tensor:
--> 463 return self._conv_forward(input, self.weight, self.bias)

File /home/pai/lib/python3.9/site-packages/torch/nn/modules/conv.py:459, in Conv2d._conv_forward(self, input, weight, bias)
455 if self.padding_mode != 'zeros':
456 return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
457 weight, bias, self.stride,
458 _pair(0), self.dilation, self.groups)
--> 459 return F.conv2d(input, weight, bias, self.stride,
460 self.padding, self.dilation, self.groups)

RuntimeError: GET was unable to find an engine to execute this computation

Thanks a ton for this repo. I have 2 questions:

1. Is there a way to make it work with Sd 2 Inpainting and potentially upcoming inpainting models ( XL etc)
2. If I have a ckpt of an custom inpainting model, how can I convert that to diffusers format?

Controlnet 1.1 preprocessors like lineart would be great

Thanks for the great repo. I was trying to remove an object from an image. I try to use the canny method and also set the prompt to be nothing and decrease the controlnet_conditioning_scale to 0. This works on the default image in the colab but not with any other image. In fact, it produces sth else in the masked area. Could you please explain what else should be done to have removing effect ?

Great work @mikonvergence!
I have a question that is somewhat related to #1. Say I have a poster image and want to inpaint the face in the poster with a given avatar image like:

How can I achieve this given the fact that these avatars are a new "concept" for the LDM? I did try your method mentioned in the issue but it did not work out for me.

Hello! I'm trying to train a controlnet with the diffusers train script https://github.com/huggingface/diffusers/blob/main/examples/controlnet/train_controlnet.py
how can i use this pipeline for training with this script

Compatibility with SDXL would be awesome

In the original ControlNet pipeline, we can pass a list of controlnet models like this

        self.ptxt = StableDiffusionControlNetPipeline.from_pretrained(
                "runwayml/stable-diffusion-v1-5",
                safety_checker=None,
                requires_safety_checker=False,
                controlnet=[
                    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16),
                    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth", torch_dtype=torch.float16)
                ],
                torch_dtype=torch.float16).to("cuda")

Is this supported in this pipeline?

Cheers

Is there a way to do promptless inpainting with ControlNet and stable diffusion 1.5 inpainting model?. I want to recreate this https://civitai.com/articles/1907 in colab but don't know how, and I don't want gradio UIs or server because you can't run sd-webui on free Colab and my pc is weak..

Hi ! Thanks for your great work!
I'd like to ask you how to train the model? Is it training both unet for inpainting and controlnet? Or do you train these two separately?