Better integration with gfx-hal about xrbevy HOT 7 OPEN

That sounds a really great approach, will send a message on Discord

from xrbevy.

Hi @zarik5 - thank you for the message. ALVR seems like a great project! Regarding this PoC of xr on bevy, all help is welcome :)

I saw your message in gfx-issue, and had a bit of thought already. Some initial pointers:

• taking the openxr-part from gfx-rs could be doable. Its mostly self-contained anyways
  • if possible, communication could be done through raw pointers. This would also encapsulate the openxr implementation, I like the idea
• there's a few parts that'd require tinkering and/or support from gfx, though:
  • multiview (link)
  • supplying xr-required api version (I don't remember if this is really needed, though, have to double check): link)
  • getting a list of xr-enabled devices (link). This is something that's done at wgpu-side
  • on xr, setting only a subset of queue families link

There might also be some openxr-reconfiguration cases which I haven't hit yet. Also, my pull request for gfx support only vulkan currently, but openxr supports opengl too.

wgpu-rs and wgpu crates are ~34k lines of code in total, I guess bevy benefits quite a bit of the graphics handling done here. Moving away from wgpu might not be feasible...

My PoC uses 100% the rendering done by bevy (using wgpu), only change is the render target - whereas in non-xr environment the target is a swapchain supplied by wgpu, in XR environment I'm creating a xr-specific swapchain.

If wgpu is used, then the challenge of moving raw handles between bevy <-> wgpu-rs <-> wgpu <-> gfx-rs will persist. So far, I managed to get away with a solution where the handles need to be moved only in initialization, not during rendering. Not sure if that covers all the cases, though. Current solution is explained here

from xrbevy.

Thanks for the reply. I agree that cutting out wgpu is not feasible. I don't have much time (because of school) but I'll try to work on gfx-hal, starting from exposing the raw handles. I'll be working here. If your PR on wgpu gets dismissed I'll propose to expose the handles also there.

supplying xr-required api version (I don't remember if this is really needed, though, have to double check): link)

The example on openxrs uses Vulkan version 1.1.0. I don't think there is any restriction on this.

from xrbevy.

Looking more into wgpu I think there is a clean way of implementing this. Since wpgu-core is advertised as not usable by third-party crates, it should allow exposing low level details from gfx-hal. In contrary wpgu-rs API should remain safe and cross platform. wgpu-rs has two backends: wgpu-core and web native. So I think it would make sense to create at this level a new XR abstraction layer that uses OpenXR or WebXR depending on the target. I plan to port ALVR client app to web so I would be glad to contribute the WebXR backend (when I have the time). While WebXR must be implemented at wgpu-rs level, OpenXR could be implemented in wgpu-core, but I don't think it is a good idea. wpgu-core is used also by wgpu-native (for WebGPU in Firefox) which is only a graphics API. Implementing OpenXR in wgpu-core means also that the XR API needs to be wrapped one extra time. Depending on what feedback we get we might consider creating a wgpu-xr crate (that sits between wgpu-core and wgpu-rs), so an extra abstraction layer might be needed. And at that point I could work on a wgpu-codec crate to wrap WebCodecs and native decoding APIs

from xrbevy.

Few thoughts:

• here's a good explanation of gfx-rs architecture: https://gfx-rs.github.io/2020/11/16/big-picture.html
• currently, there's quite a bit of interplay between bevy and gfx in my poc, abstraction is no easy feat (but maybe doable)
• OpenXR contains rendering, but also a input/output and even external hardware device layer, which might go outside of gfx scope. Some explanation at https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#input
• one easiest and maybe most beneficial part for abstraction could be the swapchain handling (see https://github.com/blaind/bevy/blob/6226677b9a60623d9b0c00f4dfc4eca80f62ddb8/crates/bevy_openxr_core/src/swapchain.rs#L39) - since that made bevy part a bit more complex - if wgpu handled that, that'd be awesome
  • however, there must be hooks in between (preparation of swapchain/rendering, final rendering, copying to textures) for some layer (app? bevy?) to handle i/o
• (edit: I have no experience in webxr, do not really know how it fits into the picture)

(@ gfx-rs/gfx#3761)

from xrbevy.

here's a good explanation of gfx-rs architecture: https://gfx-rs.github.io/2020/11/16/big-picture.html

Thank you for the link!

(edit: I have no experience in webxr, do not really know how it fits into the picture)

If we can break the wgpu-rs abstraction with backend specific details (as proposed here), I think the better way of handling this would be outside wgpu-rs. This means that WebXR can be ignored for now, and OpenXR can be fully implemented inside bevy_openxr_core.

I'm working on gfx-hal, I hope to have something to show soon. My goal is to expose what you need in gfx-hal without implementing anything xr specific in it.

from xrbevy.

My development plan is to first finish gfx-hal low level interop with Vulkan and test it modifying the vulkan example in openxrs by replacing vulkan with gfx-hal. After the gfx-hal API is complete I will work on wpgu-core and then on wpgu-rs, modifing the same example for testing. Probably this will take a lot of time. @blaind If you want to work one the same parts we can coordinate on Discord (I'm zarik5#6062).

from xrbevy.