axodox / axodox-machinelearning Goto Github PK

This repository contains a pure C++ ONNX implementation of multiple offline AI models, such as StableDiffusion (1.5 and XL), ControlNet, Midas, HED and OpenPose.

License: MIT License

C++ 96.62% C 1.80% PowerShell 1.59%

controlnet cpp directml holistically-nested-edge-detection image-generation midas mit-license native nuget onnx openpose stable-diffusion stable-diffusion-xl

axodox-machinelearning's Introduction

Introduction

This repository contains a fully C++ implementation of Stable Diffusion-based image synthesis, including the original txt2img, img2img and inpainting capabilities and the safety checker. This solution does not depend on Python and runs the entire image generation process in a single process with competitive performance, making deployments significantly simpler and smaller, essentially consisting a few executable and library files, and the model weights. Using the library it is possible to integrate Stable Diffusion into almost any application - as long as it can import C++ or C functions, but it is most useful for the developers of realtime graphics applications and games, which are often realized with C++.

a samurai drawing his sword to defend his land

a sailship crossing the high sea, 18st century, impressionist painting, closeup

close up portrait photo of woman in wastelander clothes, long haircut, pale skin, slim body, background is city ruins, (high detailed skin:1.2)

ControlNet support

The library also supports ControlNet, this allows using input images to guide the image generation process, for example:

In this first, example we use an OpenPose estimator and OpenPose conditioned ControlNet, we can guide the img2img generation by specifying the pose, so it produces better results.

Using HED edge detection and edge conditioned ControlNet, we change the style of the image to resemble a comic book illustration, but keep the layout intact.

Using a depth estimator and depth map conditioned ControlNet, we generate a different character, but keep the original setup.

Feature extractors

The library also provides GPU accelerated implementations of the following feature extractors (showcased above):

Pose estimation: extracts the skeleton of a human from an image using OpenPose
Depth estimation: estimates the depth of each pixel from a single image using MiDAS
Edge Detection: extracts edges from an image, using Holistically-Nested Edge Detection

Code examples

Here are some simple code examples:

Reference models

The AI models required for the library are stored in the ONNX format. All of the models have been run through Microsoft Olive and are optimized for DirectML. I have tested the library with the following models:

You may bring your own models, by converting them using this guide.

Please make sure to check the original license of the models if you plan to integrate them in your products.

Technical background

The implementation uses the ONNX to store the mathematical models involved in the image generation. These ONNX models are then executed using the ONNX runtime, which support a variety of platforms (Windows, Linux, MacOS, Android, iOS, WebAssembly etc.), and execution providers (such as NVIDIA CUDA / TensorRT; AMD ROCm, Apple CoreML, Qualcomm QNN, Microsoft DirectML and many more).

We provide an example integration called Unpaint which showcases how the libraries can be integrated in a simple WinUI based user interface. You may download the free app from the Microsoft Store to evaluate the performance characteristics of the solution.

The current codebase and the resulting Nuget packages target Windows and use DirectML, however only small sections of the code utilize Windows specific APIs, and thus could be ported to other platforms with minimal effort.

Licensing

The source code of this library is provided under the MIT license.

Integrating the component

Prebuilt versions of the project can be retrieved from Nuget under the name Axodox.MachineLearning and added to Visual Studio C++ projects (both desktop and UWP projects are supported) with the x64 platform.

Basic integration:

Add the Axodox.Common and Axodox.MachineLearning packages to your project
Make sure to only have x64 platform in your project as this lib is x64 only for now
Ensure that your compiler is set to C++20, we also recommend enabling all warnings and conformance mode
Add the following include statement to your code file or precompiled header: #include "Include/Axodox.MachineLearning.h"
Follow this example code to integrate the pipeline: https://github.com/axodox/unpaint/blob/main/Unpaint/StableDiffusionModelExecutor.cpp

We recommend adding appropriate safety mechanisms to your app to suppress inappropriate outputs of StableDiffusion, the performance overhead is insignificant.

The Stable Diffusion models we use have been generated using Microsoft Olive, please follow the linked example to convert models from HuggingFace. By changing the script you can also convert models stored on your disk from various formats (e.g. *.safetensors). You can find some preconverted models here for testing.

Building the project

Building the library is required to make and test changes. You will need to have the following installed to build the library:

Visual Studio 2022
- Select the following workloads:
  - Desktop development with C++
  - Game development with C++
- To build Unpaint as well also select these individual packages:
  - Universal Windows Platform development
  - C++ (v143) Universal Windows Platform tools

You can either run build_nuget.ps1 or open Axodox.MachineLearning.sln and build from Visual Studio.

Once you have built the library, you override your existing nuget package install by setting the AxodoxMachineLearning-Location environment variable to point to your local build.

For example C:\dev\axodox-machinelearning\Axodox.MachineLearning.Universal for an UWP app and C:\dev\axodox-machinelearning\Axodox.MachineLearning.Desktop for a desktop app.

Then add the project pointed by the path to the solution of your own project. This allows to add all projects into the same solution and make changes on the library and your app seamlessly without copying files repeatedly.

axodox-machinelearning's People

Contributors

Stargazers

Watchers

axodox-machinelearning's Issues

Could you compile and release a Windows CLI.exe for your project, please?

Is it possible to build the project in mac arm model or linux platform?

Hi! Amazing project, and I wonder if it is possible to build the project in the Mac m model or the x84_64 platform? I am not familiar with the Windows platform. Could you please provide a CmakeList.txt to build the project? Thanks for your answer in advance.

How to set the enviroment variable to use the library

Hello,

I'd like to ask for clarification on how to set up the following instructions specifically.

Once you have built the library, you override your existing nuget package install by setting the AxodoxMachineLearning-Location environment variable to point to your local build.

For example C:\dev\axodox-machinelearning\Axodox.MachineLearning.Universal for an UWP app and C:\dev\axodox-machinelearning\Axodox.MachineLearning.Desktop for a desktop app.

Then add the project pointed by the path to the solution of your own project. This allows to add all projects into the same solution and make changes on the library and your app seamlessly without copying files repeatedly.

I'm not very familiar with these, so I apologize for any inconvenience. Thank you for your assistance.

Can't build project, looks like there's at least one unlisted dependency

Hi,
I made sure I have all the dependencies listed in the readme, but I still can't build the project. It looks like there's at least one more dependency that I need to install, but I have no idea what it might be. Build output (see attached file for entire output):

Build started...
1>------ Build started: Project: Axodox.MachineLearning.Desktop, Configuration: Debug x64 ------
1>pch.cpp
1>C:\github2\axodox-machinelearning\packages\Axodox.Common.1.0.12\include\Graphics\Devices\GraphicsTypes.h(7,38): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\github2\axodox-machinelearning\packages\Axodox.Common.1.0.12\include\Graphics\Devices\GraphicsTypes.h(7,25): error C2146: syntax error: missing ';' before identifier 'IDXGIFactoryT'
1>C:\github2\axodox-machinelearning\packages\Axodox.Common.1.0.12\include\Graphics\Devices\GraphicsTypes.h(8,38): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\github2\axodox-machinelearning\packages\Axodox.Common.1.0.12\include\Graphics\Devices\GraphicsTypes.h(8,25): error C2146: syntax error: missing ';' before identifier 'IDXGIAdapterT'
// more errors follow ...

Looks like for whatever reason, the symbols in GraphicsTypes.h (IDXGIFactory4, IDXGIAdapter3, ID3D11Device5, etc) are undefined.
BuildErrors.txt

Is it possible to use library without WinRt?

Recently I tried to use library in my console application. It is important to note that application is not UWP based. Unfortunately I encountered error - Exception thrown at 0x00007FFD7857CF19 (KernelBase.dll) in Console.exe: WinRT originate error - 0x80073D54 : 'The process has no package identity.'. It seems like that WinRt restricts model of application. Is there any way to exclude usage of Winrt? or another way to combine WinRt library with non uwp application?

LoRA support

Is there any way to use LoRA models with the library? Should be they converted like main models?

Should StableDiffusionTest.cpp generate an image?

It compiles fine, but does not generate anything (not sure if it is meant to).
I also wonder where I have to put the model files exactly for that (libfolder)...
What I would like to do: Run stable diffusion with Open Frameworks (maybe as an addon - there is already one for .onnx files).
For that I do not need Axodox::Graphics, because I would copy the pixels of the resulting tensor to an openGL texture...

custom_op_cliptok.onnx

where do I find this model? Seems like it's hardcoded into TextTokenizer::TextTokenizer

Has iOS support and what model is supported?

Hi,

First, nice project.

It has iOS/Android support?
What i need to run in iOS?
And what models are supported?
Can i use safetensors or ckpt models?

Thanks.

NuGet fails with fresh Unreal Engine 5.2 Project

Hello!
Very excited for this project!

Attempting to add Axodox.Common via NuGet to an Unreal Engine 5 project fails with the following error:

Severity Code Description Project File Line Suppression State
Error The expression "[System.IO.Directory]::GetFiles(P:\Unreal\TestProject\packages\Axodox.Common.1.0.14\build\native....\lib\desktop\x64\Development_Editor, *.lib)" cannot be evaluated. Could not find a part of the path 'P:\Unreal\TestProject\packages\Axodox.Common.1.0.14\lib\desktop\x64\Development_Editor'. P:\Unreal\TestProject\packages\Axodox.Common.1.0.14\build\native\Axodox.Common.targets

This is after setting the C++ standard to C++20 in visual studio as well as:

public class TestProject: ModuleRules
{
	public TestProject(ReadOnlyTargetRules Target) : base(Target)
	{
		CppStandard = CppStandardVersion.Cpp20
                ...

Projects with Win32 target fail to build

Because this thing is x64 only now.

Workaround: remove the win32 target from your project.

hi,What kind of library is this, is onnx running on the onnxruntime?

This implementation is really awesome.Is onnx running on the onnxruntime?

[Feature Request] Support v-prediction

Some SD finetunes (and the official Stable Diffusion 2) use v-prediction because they observe that it's better
Currently, v-prediction models will export with Olive and run fine with the ONNX diffusers pipeline, but not here. This is "a castle" on one I exported

I tried modifying DpmPlusPlus2MScheduler::ApplyStep by modifying the predictedOriginalSample based off one of the diffusers implementations

        auto predictedOriginalSample = output.BinaryOperation<float>(input, [currentSigma](float a, float b) {
            float sigmaSquared = currentSigma * currentSigma;
            return (a * (-currentSigma / sqrt(sigmaSquared + 1))) + (b / sqrt(sigmaSquared + 1));
            });

~~I could only make the result semi-coherent~~

In DPMPlusPlus2MScheduler, the formula from HF's euler discrete works

        auto predictedOriginalSample = output.BinaryOperation<float>(input, [currentSigma](float model_output, float sample){
            float sigmaSquaredPlusOne = currentSigma * currentSigma + 1;
            return (model_output * (-currentSigma / std::sqrt(sigmaSquaredPlusOne))) + (sample / sigmaSquaredPlusOne);
            });

If you could finish this ~~(haven't figured out EulerAncestralScheduler)~~ (I have everything here), it would greatly improve compatibility. I uploaded the model in case it's useful.

Failed to set to the execution context

Hi axodox,

I want to run the test, but all of the test results show "Failed to set to the execution context."

I have run update_dependencies.ps1 and build_nuget.ps1.

Did I miss something?

Thanks!

LCM support

Does the API support LCM onnx models?