StableFused is a toy library to experiment with Stable Diffusion inspired by 🤗 diffusers and various other sources! One of the main reasons I'm working on this project is to learn more about Stable Diffusion, and generative models in general. It is my current area of research at university.

It is recommended to use a virtual environment. You can use venv or conda to create one.

Unix:

python -m venv venv
source venv/bin/activate

Windows:

python -m venv venv
venv\Scripts\activate

For usage, install the package from PyPI.

pip install stablefused

For development, fork the repository, clone it and install the package in editable mode.

git clone https://github.com/<YOUR_USERNAME>/stablefused.git
cd stablefused
pip install -e ".[dev]"

Checkout the examples folder for notebooks 🥰

Contributions are welcome! Note that this project is not a serious implementation for training/inference/fine-tuning diffusion models. It is a toy library. I am working on it for fun and experimentation purposes (and because I'm too stupid to modify large codebases and understand what's going on).

As I'm not an expert in this field, I will have probably made a lot of mistakes. If you find any, please open an issue or a PR. I'll be happy to learn from you!

The following sources have been very helpful to me in understanding Stable Diffusion. I highly recommend you to check them out!

• 🤗 diffusers
• Karpathy's gist on latent walking
• Nateraw's stable-diffusion-videos
• AnimateDiff
• RunwayML StableDiffusion
• 🤗 Annotated Diffusion Blog
• Keras CV
• Lillian Weng's Blogs
• Emilio Dorigatti's Blogs
• The AI Summer Diffusion Models Blog

Refer to the notebooks for more details and enjoy the denoising process!

Image to Image

These results are generated using the Image to Image notebook.

Source Image	Denoising Diffusion Process
The Renaissance Astronaut	High quality and colorful photo of Robert J Oppenheimer, father of the atomic bomb, in a spacesuit, galaxy in the background, universe, octane render, realistic, 8k, bright colors	Stylistic photorealisic photo of Margot Robbie, playing the role of astronaut, pretty, beautiful, high contrast, high quality, galaxies, intricate detail, colorful, 8k
	image_to_image_diffusion.mp4

PS

The results from Image to Image Diffusion don't seem very great from my experimentation. It might be some kind of bug in my implementation, which I'll have to look into later...

There is a lot of ongoing research on the generation of videos from text prompts. It is also my current area of research at university. The implementation here is adapted from AnimateDiff.

There is immense potential in developing this kind of technology and its possible usecases are unlimited - personalized educational content, marketing and advertising, creativity and art, etc. to name a few. Imagine a world where you have your own personal ChatGPT/Bard like assistants for visual learning - a model that can generate 3Blue1Brown style videos explaining science topics, or depict a story! Current models are not that capable yet, but this is where we are headed, I think, and is what me and my team are researching on. The future of this technology will be fascinating to witness!

Image inpainting is a technique that aims to fill in missing or damaged parts of an image. It is used to restore or repair images by extrapolating the surrounding information to recreate the missing regions seamlessly.

These results are generated using the Inpainting notebook.

Inpainting using a fixed mask and different prompts

Inpainting
Prompt 1: Digital illustration of a mythical creature, high quality, realistic, 8k Prompt 2: Digital illustration of a mythical creature, high quality, realistic, 8k Prompt 3: Digital illustration of a dragon, high quality, realistic, octane render, 8k Prompt 4: Digital illustration of a ferocious lion, high quality, realistic, octane render, 8k Prompt 5: Digital illustration of an evil white rabbit, high quality, realistic, 8k Prompt 6: Digital illustration of samurai with a moon-like object in the background, high quality, realistic, octane render, 8k
Image	Mask

Guidance scale is a value inspired by the paper Classifier-Free Diffusion Guidance. The explanation of how CFG works is out-of-scope here, but there are many online sources where you can read about it (linked below).

• Guidance: a cheat for diffusion models
• Diffusion Models, DDPMs, DDIMs and CFG
• Classifier-Free Guidance Scale

In short, guidance scale is a value that controls the amount of "guidance" used in the diffusion process. That is, the higher the value, the more closely the diffusion process follows the prompt. A lower guidance scale allows the model to be more creative, and work slightly different from the exact prompt. After a certain threshold maximum value, the results start to get worse, blurry and noisy.

Guidance scale values, in practice, are usually in the range 6-15, and the default value of 7.5 is used in many inference implementations. However, manipulating it can lead to some very interesting results. It also only makes sense when it is set to 1.0 or higher, which is why many implementations use a minimum value of 1.0.

But... what happens when we set guidance scale to 0? Or negative? Let's find out!

When you use a negative value for the guidance scale, the model will try to generate images that are the opposite of what you specify in the prompt. For example, if you prompt the model to generate an image of an astronaut, and you use a negative guidance scale, the model will try to generate an image of everything but an astronaut. This can be a fun way to generate creative and unexpected images (sometimes NSFW or absolute horrendous stuff, if you are not using a safety-checker model - which is the case with StableFused).

The original images produced are too large to display in high quality here. You can find them in my Drive. These images are compressed from ~30 MB to ~6 MB in order for GitHub to accept uploads.

Generative models, like the ones used in Stable Diffusion, learn a latent representation of the world. A latent representation is a low-dimensional vector space embedding of the world. In the case of SD, this latent representation is learnt by training on text-image pairs. This representation is used to generate samples given a prompt and a random noise vector. The model tries to predict and remove noise from the random noise vector, while also aligning the vector to the prompt. This results in some interesting properties of the latent space.

Stable Diffusion models (atleast, the models used here) learn two latent representations - one of the NLP space for prompts, and one of the image space. These latent representations are continuous. If we choose two vectors in the latent space to sample from, we get two different/similar images depending on how different the chosen vectors are. This is the basis of latent walking. We can choose two vectors in the latent space, and sample from the latent path between them. This results in a smooth transition between the two images.

Similar Image Generation by sampling latent space

The results below show just how information rich the latent space of these stable diffusion models are.

Source Image	Latent Walks
Large futuristic mechanical robot in the foreground of a baroque-style battle scene, photorealistic, high quality, 8k

At the moment, I'm not sure if I'll continue to expand on this project, but if I do, here are some things I have in mind (in no particular order, and for documentation purposes):

• Add support for more techniques of inference - explore new sampling techniques and optimize diffusion paths
• Implement and stay up-to-date with the latest papers in the field
• Removing 🧨 diffusers as a dependency by implementing all required components myself
• Create user-friendly web demos or GUI tools to make experimentation easier.
• Add LoRA, training and fine-tuning support
• Improve codebase, documentation and tests
• Improve support for not only Stable Diffusion, but other diffusion techniques, involving but not limited to audio, video, etc.

MIT