stochasticai / x-stable-diffusion Goto Github PK

Some people shared their result onnxruntime actually getting faster:

https://medium.com/microsoftazure/accelerating-stable-diffusion-inference-with-onnx-runtime-203bd7728540

I have try the process described in this code repo about tensorrt， but i can not reproduce the TensorRT latency on A100，
my fp16 result is about 3.2，greater than the statistic that you post on 。

As title.Can we use lora safetensors? If we can , how to use?

Thank you for your great work. I am trying to follow your instruction to convert UNET to tensorrt format, however, I got the following errors:

• when converting the model to onnx: I got stuck because the att:ScaledDotProductAttention is not supported. I found the solution on the internet by adding a customized function for scaleddotproductattention. However, I once again encounter this issue when trying to convert the model from onnx to trt. I just want to ask if you have ever encountered this error. And if yes, how did you manage to fix it? Thank you!

After got unet.engine, then run "python demo.py", and got the following errors, who know this issue, thanks!
(env_py39) [jch@localhost TensorRT]$ python demo.py
[12/27/2022-06:37:00] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
/home/jch/anaconda3/envs/env_py39/lib/python3.9/site-packages/tensorrt/init.py:166: FutureWarning: In the future np.bool will be defined as the corresponding NumPy scalar. (This may have returned Python scalars in past versions.
bool: np.bool,
Traceback (most recent call last):
File "/home/jch/work/x-stable-diffusion/TensorRT/demo.py", line 125, in
model = TrtDiffusionModel(args)
File "/home/jch/work/x-stable-diffusion/TensorRT/demo.py", line 48, in init
self.unet = TRTModel(args.trt_unet_save_path)
File "/home/jch/work/x-stable-diffusion/TensorRT/trt_model.py", line 41, in init
self.inputs, self.outputs, self.bindings, self.stream = self.allocate_buffers(
File "/home/jch/work/x-stable-diffusion/TensorRT/trt_model.py", line 79, in allocate_buffers
dtype = trt.nptype(engine.get_binding_dtype(binding))
File "/home/jch/anaconda3/envs/env_py39/lib/python3.9/site-packages/tensorrt/init.py", line 166, in nptype
bool: np.bool,
File "/home/jch/anaconda3/envs/env_py39/lib/python3.9/site-packages/numpy/init.py", line 284, in getattr
raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'bool'
Segmentation fault

Is it possible to build tensorrt engine with dynamic height and width?

When I enter "stochasticx stable-diffusion deploy --type aitemplate," the following message appears: "WARNING | stochasticx.client.stable_diffusion | Currently we only support a100 and t4 gpus. We may support other gpus in the future." My GPU is an RTX4090. Please add it to the list of supported devices.

As title，and WebUI refers to：WebUI
Thank you!!!

INFO: 172.17.0.1:53344 - "POST /predict/ HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 270, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 75, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 64, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 680, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 275, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 65, in app
response = await func(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 231, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
return await dependant.call(**values)
File "/code/./server.py", line 37, in predict
images = pipe(
File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/code/./pipeline_stable_diffusion_ait.py", line 310, in call
noise_pred = self.unet_inference(
File "/code/./pipeline_stable_diffusion_ait.py", line 105, in unet_inference
exe_module.run_with_tensors(inputs, ys, graph_mode=True)
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 483, in run_with_tensors
outputs_ait = self.run(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 438, in run
return self._run_impl(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 377, in _run_impl
self.DLL.AITemplateModelContainerRun(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 192, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerRun
ERROR:uvicorn.error:Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 270, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 75, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 64, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 680, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 275, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 65, in app
response = await func(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 231, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
return await dependant.call(**values)
File "/code/./server.py", line 37, in predict
images = pipe(
File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/code/./pipeline_stable_diffusion_ait.py", line 310, in call
noise_pred = self.unet_inference(
File "/code/./pipeline_stable_diffusion_ait.py", line 105, in unet_inference
exe_module.run_with_tensors(inputs, ys, graph_mode=True)
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 483, in run_with_tensors
outputs_ait = self.run(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 438, in run
return self._run_impl(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 377, in _run_impl
self.DLL.AITemplateModelContainerRun(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 192, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerRun
INFO: 172.17.0.1:53346 - "POST /predict HTTP/1.1" 307 Temporary Redirect
0it [00:00, ?it/s][22:02:37] ./tmp/UNet2DConditionModel/model-generated.h:3235: Got error: no error enum: 1 at ./tmp/UNet2DConditionModel/model-generated.h: 4757
[22:02:37] ./tmp/UNet2DConditionModel/model_interface.cu:102: Error: Got error: no error enum: 1 at ./tmp/UNet2DConditionModel/model-generated.h: 4757
0it [00:00, ?it/s]
INFO: 172.17.0.1:53346 - "POST /predict/ HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 270, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 75, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 64, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 680, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 275, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 65, in app
response = await func(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 231, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
return await dependant.call(**values)
File "/code/./server.py", line 37, in predict
images = pipe(
File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/code/./pipeline_stable_diffusion_ait.py", line 310, in call
noise_pred = self.unet_inference(
File "/code/./pipeline_stable_diffusion_ait.py", line 105, in unet_inference
exe_module.run_with_tensors(inputs, ys, graph_mode=True)
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 483, in run_with_tensors
outputs_ait = self.run(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 438, in run
return self._run_impl(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 377, in _run_impl
self.DLL.AITemplateModelContainerRun(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 192, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerRun
ERROR:uvicorn.error:Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 270, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 75, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 64, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 680, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 275, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 65, in app
response = await func(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 231, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
return await dependant.call(**values)
File "/code/./server.py", line 37, in predict
images = pipe(
File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/code/./pipeline_stable_diffusion_ait.py", line 310, in call
noise_pred = self.unet_inference(
File "/code/./pipeline_stable_diffusion_ait.py", line 105, in unet_inference
exe_module.run_with_tensors(inputs, ys, graph_mode=True)
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 483, in run_with_tensors
outputs_ait = self.run(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 438, in run
return self._run_impl(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 377, in _run_impl
self.DLL.AITemplateModelContainerRun(
File "/usr/local/lib/python3.9/site-packages/aitemplate/compiler/model.py", line 192, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerRun

I've been bouncing around various StableDiffusion optimisations the last couple of weeks, and figured I would link out to some of the ones I remember in hopes that they can be explored/added into the benchmarks/comparisons here:

• https://github.com/VoltaML/voltaML-fast-stable-diffusion

  • Lightweight library to accelerate Stable-Diffusion, Dreambooth into fastest inference models with single line of code

• https://github.com/kamalkraj/stable-diffusion-tritonserver

  • Deploy stable diffusion model with onnx + tritonserver

• https://github.com/luohao123/gaintmodels

  • Int8 StableFusion model

• AUTOMATIC1111/stable-diffusion-webui#1625

  • AITemplate

• AUTOMATIC1111/stable-diffusion-webui#4721
  • https://github.com/NVIDIA/TransformerEngine
• AUTOMATIC1111/stable-diffusion-webui#4096
  • https://github.com/ELS-RD/kernl

    • Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable.

  • https://www.reddit.com/r/MachineLearning/comments/ydqmjp/p_up_to_12x_faster_gpu_inference_on_bert_t5_and/

    • Up to 12X faster GPU inference on Bert, T5 and other transformers with OpenAI Triton kernels

• AUTOMATIC1111/stable-diffusion-webui#4606
  • https://github.com/hpcaitech/ColossalAI/tree/main/examples/images/diffusion?rgh-link-date=2022-11-11T18%3A20%3A17Z
  • Other related issues: AUTOMATIC1111/stable-diffusion-webui#4606 (comment)
• etc

latents = self.scheduler.step(noise_pred.cuda(), t, latents)[
"prev_sample"
]
i think there is a bug in this code, it should be:
latents = self.scheduler.step(noise_pred.cuda(), i, latents)[
"prev_sample"
]
because if i use t as the input , it will occur IndexError: index 999 is out of bounds for dimension 0 with size 51
and i change t to i ,the code can run right

Was trying to run this colab notebook - https://colab.research.google.com/drive/1m3n2n5bfNpRgWJ8K-xwTvhzrTQETWduq?usp=sharing#scrollTo=ONitKUKN4CCW

and seem to be stuck on this error despite pip installing the libraries in the cell above

"ImportError: Please install the accelerate library to use Diffusers with PyTorch. You can do so by running pip install diffusers[torch]. Or if torch is already installed, you can run pip install accelerate."

As title

I would like to know which frameworks support mac, ideally using Metal Performance Shaders (MPS) or Apple Neural Engine (ANE).
Seeing how these benchmark against windows and linux would be nice to see, too.

Hey friends,

Interested in getting the notebook running, would love to solve this! Thanks in advance!

Traceback (most recent call last):
File "/content/drive/MyDrive/x-stable-diffusion/TensorRT/demo.py", line 114, in
image = model.predict(
File "/content/drive/MyDrive/x-stable-diffusion/TensorRT/demo.py", line 88, in predict
latents = self.scheduler.step(noise_pred.cuda(), i, latents)["prev_sample"]
File "/usr/local/lib/python3.8/dist-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 218, in step
step_index = (self.timesteps == timestep).nonzero().item()
ValueError: only one element tensors can be converted to Python scalars

When I changed batchsize=4 while converting the model, it reported an error.
The error message is that the broadcast sizes do not match

Your colab example is working perfectly well. I am able to convert all onnx models to tensorrt engines with slight modification. But when i generate onnx file for unet model using the default diffusers onnx export script, that onnx file is not convertible with the colab notebook you provided. I have also tested on my own pc with RTX-3090.

So my question is how did you generate the onnx file for unet (that you used in colab tutorial)?

does this have support for making batch inferences?
does this have support for handling prompts that creates more than 75 tokens?

Hi. I am trying to convert my onnx file to Tensorrt using the conver_unet_to_tensorrt.py. But i get the below error

Traceback (most recent call last):
  File "convert_unet_to_tensorrt.py", line 52, in <module>
    convert(args)
  File "convert_unet_to_tensorrt.py", line 47, in convert
    f.write(serialized_engine)
TypeError: a bytes-like object is required, not 'NoneType' 

Except for the onnx file mentioned in the library, none of the other onnx files seem to convert. Any fix? Thanks.

I am wondering if the PyTorch baseline is actually optimized enough? Specifically, could you

• Remove autocast since the model is already in FP16? AutoCast would actually convert some other non-GEMM fp16 kernels in FP32 (or TF32 in the case of Ampere GPUs)
• Run some warm up iteration before measuring the inference latency (averaged across a few)? Like how you did it with TensorRT
• Use flags such as torch.backends.cudnn.benchmark = True before running GPU kernels.

On my local machine, just these optimizations (for lack of a better word as they are not really optimizations) would make PyTorch baseline at least 2X faster.