SDXL with ControlNet does not respond to depth about dream-textures HOT 12 CLOSED

@Anarchid This should be resolved by #717, a new build is being created here: https://github.com/carson-katri/dream-textures/actions/runs/6678624526?pr=711

Let me know if you still have issues and I can look into further fixes.

from dream-textures.

@Anarchid Good catch. This seems to be a bug in diffusers when using auto pipelines. I've opened an issue there, but in the meantime I'll look for a workaround.

from dream-textures.

#720 works around this issue for now. Let me know if you run into any other issues with the latest on releases/0.3.0. A new build is being created here: https://github.com/carson-katri/dream-textures/actions/runs/6714616416?pr=711

from dream-textures.

The AutoPipeline bug was fixed in upstream diffusers, so I've updated the dependency in #721 and reverted the workaround. Everything should be back to normal.

from dream-textures.

From browsing around the source it looks like there are specific configs for using controlnets with other checkpoints (cldm_v15.yaml, cldm_v21.yaml), but no such for sdxl? Does something like this need to be configured specifically?

from dream-textures.

The primary issue seems to be that using ControlNet for depth projection isn't working in general, it's not a SDXL specific issue.

Those config files are only used for checkpoint conversion when using checkpoint importing or checkpoint linking. It doesn't require a SDXL ControlNet config file because these models so far are only distributed as diffusers weights.

from dream-textures.

Is there any likely place where it could be fruitful for me to look around rather than wait for more competent people to deal with it? :)

from dream-textures.

The fix works, thank you greatly!

from dream-textures.

Hm, looks like it only works for the first time per blender run. Restarting Blender helps, but after the first click on "project dream texture" any following attempts are met with this:

from dream-textures.

Switching to sd2 depth (without contorlnet) and back to sdxl with cn also resets the problem.

from dream-textures.

This seems to solve the previous problem, but now i get a Cuda OOM error on second execution - seems that the SDXL gets loaded into memory for a second time, where it doesn't quite fit, sadly.

from dream-textures.

After updating addon and diffusers, everything flies smooth now.

from dream-textures.