Extension that patches Hypernetwork structures and training

For Hypernetwork structure, see AUTOMATIC1111/stable-diffusion-webui#4334

For Variable Dropout, see AUTOMATIC1111/stable-diffusion-webui#4288

No-Crop Training You can train without cropping images. THis feature is now implemented in original webui too! 🥳

Fix OSError while training

Unload Optimizer while generating Previews

Tensorboard integration, and Tuning

Residual-Block based Hypernetwork(in beta test)

Beta hypernetworks* can contain additional informations and specified dropout structures. It will be loaded without extension too, but it won't load dropout structure, so training won't work as original. Generating images should work identically.

This extension also overrides how webui loads and finds hypernetworks, to use variable dropout rates, and etc. Thus, hypernetwork created with variable dropout rate might not work correctly in original webui.

Well, at least now it should work, without having any problem except you cannot use variable dropout rate in original webui. If you have problem with loading hypernetworks, please create an issue. I can submit pr to original branch to load these beta typed hypernetworks correctly.

If you're unsure about options, just enable every checkbox, and don't change default value.

This also fixes some CUDA memory issues. Currently both Beta and Gamma Training is working very well, as far as I could say.

Now you can save hypernetwork generation / training setting, and load it in train_tuning tab. This will allow combination of hypernetwork structures, and training setups, to find best way for stuff.

Now you can select CLIP model, its difference is significant but whether its better or not is unknown.

The concept of ResNet, returning x + f(x) instead in layers, are available with option. Original webui does not support this, so you cannot load it without extension. Unlike expanding type (1 -> 2 -> 1), shrinking type(1 -> 0.1 -> 1) network will lost information at initial phase. In this case, we need to additionally train transformation that compresses and decompresses it. This is currently only in code, its not offered in UI at default.

Allow using general models with .safetensor save /loading

We cannot apply convolution for attention, it does do something, but hypernetwork here, only affects attention, and its different from 'attention map' which is already a decoded form(image BW vectors) of attention(latent space). Same goes to SENet, unfortunately.