Timeline for SDXL Controlnet?

[J-Cott]

How much work will it be to get ControlNet working with the new SDXL model when v1 is released next week? Do the developers plan on doing this soon?

[Rkkss]

The support can be day 1 but they or someone need to train new models.

[dayunbao]

If this is the case, I wonder if there's anything we could do to help? I've been working with the SDXL version 0.9 that's available via Stability AI's API, and it's amazing. Going back to Stable Diffusion version 1.5 after SDXL, so that I can use the ControlNet pre-processor models, is really painful. Those models aren't compatible with Stable Diffusion version 2, either. I've tried using the Openpose pre-processor with the 512 and 768 versions of Stable Diffusion v2, but got errors like this:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (616x1024 and 768x320)

Or is this user error?

Someone has made some models that are supposed to be compatible here https://huggingface.co/thibaud/controlnet-sd21/tree/main, but I haven't tried them yet.

@lllyasviel I'm sure you're super busy with a full time job, a life, and trying to maintain ControlNet, so is there anyway we can help?

[Tollanador]

To use the SD 2.x ControlNet's in Automatic1111, use this attached file. It's saved as a txt so I could upload it directly to this post.

Rename the file to match the SD 2.x ControlNet model with a .yaml extension, do this for all the ControlNet models you want to use.

Now go enjoy SD 2.x with ControlNet, have fun!

control_v11p_sd21_fix.txt

[blacklig]

Yeah, seriously would love to know this as well :) Hope CN for SDXL will be out soon :) If needed I can provide one RTX 3090 for training for some days no problem

[omaralvarez]

I am interested in SDXL support also. I may have some available hardware to train them, although the training datasets are not public.

[DrakenZA]

From what was said at launch, they have trained some controlnets, but they are not happy with the size of them (1gb). Not sure if im misunderstanding them or whatever, but isnt that the size of the current controlnet models trained for sd1.5 ?

Not sure how that is an issue and they are not happy to release if that is the case.

Im talking about Stability AI here, not sure where the repo creator sits in all of this. Would be nice if he was the one to train up the controlnets, as the 1.5 version 1.1 controlnets are insanely well done. Really want the level for SDXL.

[Michaelvirga]

Yeah some of the current CN models are like 2 GB, so i dont see an issue if it's 1GB. Maybe they feel they haven't gotten it up to par performance wise that they want with SDXL, but just any kind of communication on this would be nice.

[melMass]

@DrakenZA Not sure if it's SOTA but the first I found is actually 5gb 😓
https://huggingface.co/SargeZT/controlnet-v1e-sdxl-depth/tree/main

CN 1.1 was around 1gb, or a bit less if pruned

[gumpngreen]

I tried putting the newer .safetensors file from SargeZT in the \extensions\sd-webui-controlnet\models directory. While the Preprocessor Preview worked normally and generated a usable depth map when I tried to generate I got this message: "ControlNet - WARNING - ControlNet does not support SDXL -- disabling" In other words, ControlNet itself will prevent us from using this in A1111, but perhaps ComfyUI can be made to work?

[fcabanski]

This is not good - there has been no info.

[dayunbao]

Recently he's been quite active in the AUTOMATIC1111 web UI dev Discord server. So hopefully that means he's working on SD XL compatibility.

[dayunbao]

Well, it seems that this https://github.com/lllyasviel/Fooocus is the most likely reason for lllyasviel being so active in the Automatic1111 Discord server. I guess we can just hope he is working on a new version of ControlNet.

[dgcnz]

HF's Diffusers have already trained canny and depth models, is there any other blockers for this to work? https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0
https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0

[dayunbao]

They also just released this https://huggingface.co/stabilityai/control-lora. They converted some trained ControlNet models into LoRAs. I think they converted the two models dgcnz listed, plus a few more.

[dayunbao]

This might also be of interest https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0. But do we know if the current version of ControlNet is capable of interfacing with SD XL?