[Bug]: Optimization of Unet fails 6950 XT about olive HOT 6 OPEN

This is DXGI_ERROR_DEVICE_HUNG during inference/evaluation, which typically happens when some GPU work is taking excessively long. The recent AMD driver optimizations for stable diffusion / multi-head attention target the RDNA 3 architecture (e.g., the 7000 series, like the Radeon RX 7900 XTX) but not the RDNA 2 (6000 series). Still, we can try to repro this on an RDNA card to see if anything jumps out.

from olive.

6800xt has same err

from olive.

Error on my 6900XT as well, on 0.4.0

from olive.

The following error message seems be related to DirectML EP.

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\ExecutionProvider.cpp(896)\onnxruntime_pybind11_state.pyd!00007FFE31C80201: (caller: 00007FFE31C80C2F) Exception(2) tid(3c14) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

from olive.

@jstoecker, do you have any insight?

from olive.

seems similar with #510

from olive.