Giter VIP home page Giter VIP logo

openxr-overlaylayer's Introduction

OpenXR XR_EXTX_overlay Test Implementation

Introduction

Existing XR APIs support the notion of 2D composition layers “on top” of an existing application’s content. Valve’s IVROverlay is an example.

Overlay applications can add a rich variety of content into other XR Applications:

  • Chat (e.g. Pluto VR)
  • Desktop OS windows in-world
  • In-game HUD
  • Virtual keyboard

The XR_EXTX_overlay experimental extension to OpenXR is intended to prove the concept and uncover issues. Some issues to address include:

  • Establishing Security and access control requirements
  • How to address input focus requirements

We hope OpenXR runtimes will incorporate a future version of the extension after any concerns are addressed and perhaps the extension is promoted to EXT or KHR.

Implementation Specifics

The XR_EXTX_overlay test implementation for Windows and Direct3D 11 consists of an OpenXR API layer and a separate remote overlay test application. The layer and application are implemented against the OpenXR 1.0.9 specification. The layer is named xr_extx_overlay.dll and the overlay test is named OverlaySample.exe. The API layer runs both in the main OpenXR app and the overlay app, and the instantiation of the API layer in the main app receives OpenXR commands over RPC (remote procedure calls using shared memory) from the instantiation of the API layer in the overlay app.

The test implementation demonstrates the remote overlay client application opening an Overlay session within “unaware” host OpenXR applications. At commit time we have tested only the OpenXR-SDK-Source hello_xr as the host application.

The overlay test app can also be compiled as a standalone OpenXR program by changing the variable createOverlaySession in main() to false. (Note that this may be visually disorienting in VR as there will be only the one layer and no other environmental cues.)

Development used CMake 3.14.6, although other versions may be compatible.

Building this test implementation used Visual Studio project files generated using the commands:

mkdir build
cd build
cmake -D OPENXR_SDK_SOURCE_ROOT=openxr-sdk-source-dir -D OPENXR_SDK_BUILD_SUBDIR=sdk-source-relative-build-dir -D OPENXR_LIB_DIR=built_openxr_loader_dir  -D FREEIMAGE_ROOT=freeimage-header-lib-dir -D FREEIMAGEPLUS_ROOT=freeimageplus-header-lib-dir -G visual-studio-generator-name -A x64 ..

As an example, if building using bash in mintty (msysgit "Git Bash"), you have downloaded the FreeImage dist into $HOME/Downloads and unpacked it there, and you're using Visual Studio 2017, your cmake command may look like this:

cmake -D OPENXR_SDK_SOURCE_ROOT=$HOME/trees/OpenXR-SDK-Source -D OPENXR_LIB_DIR=$HOME/trees/OpenXR-SDK-Source/build/win64/src/loader/Debug -D OPENXR_SDK_BUILD_SUBDIR=build/win64 -D FREEIMAGE_ROOT=$HOME/Downloads/FreeImage/dist/x64 -D FREEIMAGEPLUS_ROOT=$HOME/Downloads/FreeImage/Wrapper/FreeImagePlus/dist/x64 -G "Visual Studio 15 2017" -A x64 ..

The layer DLLs and executables were compiled with Visual Studio 2017, version 15.9.12 in “Debug” configuration. Only the 64-bit target is supported at this time.

Operation

The implementation has been tested on Microsoft Windows Mixed Reality OpenXR Developer runtime version 100.1910.1004 and on Oculus OpenXR developer channel runtime 1.52.0 with hello_xr from https://github.com/KhronosGroup/OpenXR-SDK-Source/tree/release-1.0.12 .

Replace references to $OVERLAY_PROJECT below with the top-level directory name containing the clone of the overlay project repository. (E.g. if you performed git clone https://github.com/LunarG/OpenXR-OverlayLayer.git from your home directory, this would be $HOME/OpenXR-OverlayLayer.) Replace references to $WORKING_DIRECTORY to a directory name which will contain runtime files relevant to the test implementation. (Perhaps $HOME/Overlay_OpenXR.)

Copy to $WORKING_DIRECTORY the following files:

The openxr_loader.dll built from OpenXR-SDK-Source or other repository
$OVERLAY_PROJECT/build/overlay-sample/Debug/OverlaySample.exe
$OVERLAY_PROJECT/build/overlay-sample/*.png
$OVERLAY_PROJECT/api-layer/xr_extx_overlay.json
$OVERLAY_PROJECT/build/api-layer/Debug/xr_extx_overlay.dll

Build src/tests/hello_xr from the OpenXR repository.

Run hello_xr.exe with the following environment variables:

XR_ENABLE_API_LAYERS=xr_extx_overlay
XR_API_LAYER_PATH=$WORKING_DIRECTORY

You will also need to set the environment variables for finding the runtime Manifest (JSON) if not in the Windows Registry (e.g. for Oculus).

These can be set, for example, in the Visual Studio Preferences for debugging hello_xr in the “Environment” section, or, as another example, at the command-line before running hello_xr. Also provide the command-line option -g D3D11, which can be set in the “Command Arguments” section of Preferences or on the command line.

Run hello_xr. Run OverlaySample.exe. If successful, the console output from OverlaySample.exe will contain various messages and finally should output “First Overlay xrEndFrame was successful! Continuing...”

Nota Bene

  • The runtime’s xrReleaseSwapchainImage function may return XR_ERROR_VALIDATION_FAILURE, and OverlaySample.exe will break into the debugger if one is running. The reason is unknown.
  • OverlaySample.exe does not suggest bindings for the Microsoft or Vive interaction profiles but instead suggests bindings for the “khr/simple_controller” profile. Probably OverlaySample.exe will need to have additional bindings added before being run on Microsoft or Vive runtimes. A workaround is to comment out the calls in openxr_program.cpp that suggest bindings for any profile but “khr/simple_controller”. This would not be an appropriate suggestion for a shipping application.

Troubleshooting

If the test program or the loader encounter a problem, please open a new issue and attach the contents of the Output pane in Visual Studio for both OverlaySample.exe and hello_xr.exe and the console output from both programs for review.

Additional useful information may be found in the loader output after setting the environment variable XR_LOADER_DEBUG to the value “all” before running hello_xr.exe.

openxr-overlaylayer's People

Contributors

bradgrantham avatar bradgrantham-lunarg avatar coderhymeslife avatar daveh-lunarg avatar fuag15 avatar jzulauf-lunarg avatar karenghavam-lunarg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openxr-overlaylayer's Issues

Named objects (FileMapping, Mutex, Semaphore) should be opened with ALL_APPLICATION_PACKAGES SID

UWP (AppContainer ) apps cannot open named resources without explicit granted permission. The API Layer uses named file mappings, mutexes, and semaphores, and those will need to be opened with ALL_APPLICATION_PACKAGES permission. It's not clear that those can be created from within UWP with that permission, so we may need to rearchitect with a service outside UWP that creates or brokers those objects.

Can't build xr_extx_overlay project

I build XR_overlay.sln following README.md.
Cmake build was successful and the OverlaySample.exe file was also created.
But not the xr_extx_overlay project. Visual studio MSB8066 error pops up saying that the custom build has ended.
Also, xr_generated_overlay.hpp and xr_generated_overlay.cpp, which should be included in this project, are not generated normally.
When I build the project, an error like the picture below.
how can i fix it?

image

compiler limit : blocks nested too deeply

When compiling ALL_BUILD target, using Visual Studio 2019, I encountered a fatal error with regards to the following Link.
https://docs.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1061?view=msvc-160

Build started...
1>------ Build started: Project: xr_extx_overlay, Configuration: Debug x64 ------
1>xr_generated_overlays.cpp
1>E:\SteamVR_Projekt\OpenXR_SDK\OpenXR-OverlayLayer-master\OpenXR-OverlayLayer-master\build\api-layer\xr_generated_overlays.cpp(9925,65): fatal error C1061: compiler limit: blocks nested too deeply
1>Done building project "xr_extx_overlay.vcxproj" -- FAILED.
2>------ Skipped Build: Project: ALL_BUILD, Configuration: Debug x64 ------
2>Project not selected to build for this solution configuration 
========== Build: 0 succeeded, 1 failed, 2 up-to-date, 1 skipped ==========

I have devised a crude workaround to at least get the build to run.

Workaround:
Replace "else" in file xr_generated_overlays.cpp(9925) with new line.

Overlaysample.exe closes immediatly - no overlay visible

-> Build all files and made the register implicit referencing to a .json for the overlay
-> Hello.xr is running without any errors (see screenshot)
-> Explorer is indicating that the overlay is available (see screenshot)

-> OverlaySample is build (x64) and all files (.exe, .png etc) are set within the same working directory
-> Openxr_loader.dll of the xr_extx_overlay and OverlaySample are identical (1.0.9)
-> After OverlaySample is run, a black command interface window is opened, and closed immediately.
-> No overlay is visible in OpenXR, only the hello.xr blocks

I added pauses statements in the OverlaySample in order to see where things go wrong.
As can be seen in attached schreenshot, everything seems to go fine and the 'program.CreateInstance("Overlay Sample",0,"none",0); ' is executed. Thereafter some programcode is run, but window is closed and no overlay visible.

-> It should be notes that the "XR_DEFINE_ATOM(XrPermissionIdEXT) is showing a C++ potential error that Uint64 type should be declared, but compiles Okay

  1. Is there a required folder structure for the OverlaySample to run smoothly?
  2. What might be wrong?

screen_Hello_XR
screen2_Explorer
screen3_OverlaySample

Usage

I built the library and I have the following files. How to integrate the library with OpenXR? the usage documents in not complete. @bradgrantham-lunarg please guide me.
image

Be quieter about failing to find a controller

Instead of streaming "Couldn't find a suggested binding matching the interaction profile for an Action; XrSpace will not be locatable; are controllers turned on?" off the screen if the controllers are turned off, print once and mark that it has been printed. (Or is this a job better suited for a log file filterer?)

suggested bindings for actions must include subpaths (e.g. ".../input/trigger", which the runtime binds to ".../input/trigger/value" for a BOOLEAN)

For all fully-specified paths, e.g. /user/hand/left/input/trigger/value, the API layer must also add the parent path, e.g. /user/hand/left/input/trigger/, to WellKnownStrings because an app can specify, e.g. BOOLEAN action binding to /user/hand/left/input/trigger/ and runtimes must take that path for BOOLEAN to be /user/hand/left/input/trigger/value.

Cascade of errors in Main if quit gracefully while Overlay app still running

ENTER in hello_xr console (exit gracefully) on Samsung Odyssey / WinMR results in:

[Verbose [VALID_ERRORGENERAL |  | VUID-xrSyncActions-session-parameterxrDestroyInstance | OpenXR-Loader | ] : xrSyncActionsEntering loader trampoline]:
Invalid XrSession handle "session" 0x0000000000000000
  Objects:
XXX [ | xrDestroyInstance | OpenXR-Loader ] : Entering loader trampoline
   [0] - XrSession (0x0000000000000000)
Verbose [GENERAL | xrDestroyInstance | OpenXR-Loader] : Entering loader terminator
XXX [ | xrDestroyInstance | OpenXR-Loader ] : Entering loader terminator
Verbose [GENERAL | xrDestroyInstance | OpenXR-Loader] : Completed loader terminator
Verbose [[VALID_ERRORGENERAL | xrDestroyInstance |  | OpenXR-LoaderVUID-xrSyncActions-session-parameter] : Completed loader trampoline |
xrSyncActionsInfo []: GENERAL | Invalid XrSession handle "session" 0x0000000000000000xrDestroyInstance
 |   Objects:OpenXR-Loader
] : RuntimeInterface::UnloadRuntime - Unloading RuntimeInterface   [
0Info [GENERAL] -  |  | XrSessionOpenXR-Loader] : RuntimeInterface being destroyed.
 (0x0000000000000000)
[VALID_ERROR | VUID-xrAcquireSwapchainImage-swapchain-parameter | xrAcquireSwapchainImage]: Invalid XrSwapchain handle "swapchain" 0x00000000000000dc
  Objects:
d:\agent\_work\2\s\src\vctools\crt\crtw32\stdcpp\thr\mutex.c(51): mutex destroyed while busy

Appears to be a destroy of a child session 0x0 which errors from the Overlay Layer and then from core_validation, then a subsequent invalid swapchain destroy results in an MSVCRT assertion because... a Mutex is being held?
Possibly the same issue as #13

Release gives openxr.lib error

Hello,

I use VS2022 and latest SDK's.
I can build the app in debug mode without error, however when I switch to release I get an error that openxr.lib is not found.
I need the release version, or otherwise I cannot use the app to test on a separate system.
The XR_EXTX_overlay is build, but the problem seems to sit in the OverlaySample.

My goal is to create a simple form as an OpenXR overlay, and wanted to use this example

Your support is appreciated!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.