Using ultraleap tracking with clicks mapped through HTCC, clicks and pointing work very well.
I have assigned joystick buttons to mouse scroll or even to other buttons (with mappings for cockpit interaction - increase/decrease in MSFS) to allow the operation of the rotaries.
The left hand works perfectly, however as soon as I rotate a dial using the right hand to point, the tracking ceases until I gesture with the left hand, then the right again (could this possibly linked to "perma-squeeze" issue: #24 (comment) ?)

XR_FB_hand_tracking_aim's hand 'aim' pose is currently used; it's a bit tricky to maintain this 'aim' point while performing a pinch gesture.

• index proximal probably feel best just for left-or-right-click
• palm or wrist may be better overall, as index is going to feel quite far off for ring and little finger pinches

This would both possibly provide a better experience for quest users, and remove a blocker on supporting other headsets.

• add executable metadata
• add install targets
• verify runtime dependencies
• #1 add MSIX
• write instructions
• #2 sign binaries
• test compatibility with OpenXR Toolkit
• test compatibility with OpenKneeboard
• test with WMR

The primary matter is deciding on a good control mapping.

For pointctrl, started discussion here: https://discord.com/channels/608478274472902656/840632276630765578/1046493380416393216

I have HTCC working with my Leap Motion Controller 2 and a finger mouse. It works well but sometimes when I try to operate cockpit controls moving my head slightly causes my cursor to move in the opposite direction. Then my hand and head start fighting for control lol! When looking to the left cursor moves right. Hand tracking is perfect if I don't move my head, the cursor goes where I point. Is it possible to couple the two trackers together so the cursor stays where I point when moving my head? Maybe I have a configuration parameter setup incorrectly? Thanks.

In one of the latest updates for Virtual Desktop, support for OpenXR was added.
However, tracking does not yet work for the meta quest 3

Although the current implementation works very well, it may be beneficial to allow changes to the "source" position of the laser pointer. My personal preference would be to move it more towards the shoulder so the aim feels like where your hand/arm is pointing rather than in a direct line from your eyes.
Fred I think you suggested the dominant eye which would also be interesting :)

I get the point that you would like only a pointer (e.g. headset-to-hand are the only two points on the vector). But since this is now the simplest to use guest2 hands-to-dcs option. Can you give us optionally a depth+rotation data to the virtual VR controller ?

Yes, this would then go towards VR controller emulation with hand, but honestly it is such a low hanging fruit and would definitelly help in both reaching switches in weird places and avoid accidental hits of cockipit switches. I would love to simply have a laser pointer comming out of my index finger or wrist instead of fixated to my eyes/hand vector. Can be optional checkbox next to the Oculus Touch VR controller.

PS: I am with you on the ring mouse buttons, but made quick/easy way by using a cheap aliexpres finger mounted bluetooth mouse with it's sensor cut-off, so getting an index finger laser pointer with 6DOF sounds like a dream come true

'left click' -> trigger
'right click' -> forward-and-back
decrease value -> counter-clockwise rotation
increase value -> clockwise rotation

If PointCTRL loses track of the FCU's, it keeps reporting the previous position apparently forever. This should be reported and considered inactive.

This will require 'joystick' firmware. The HMS will be used just as a receiver.

In turn:

• it will open support for other headsets
• real buttons are more accurate and tactile than pinching

Add registry settings to bind separate 'scroll up/scroll down' buttons, instead of 'emulate 3-button point ctrl'

The current soft world lock implementation is based on absolute position, which doesn't work well for pointctrl, as it is an angular device - the amount of movement needed varies hugely depending on distance.

An alternative would be a jitter-based approach

Hello i have a quest 2 and want to take the Baro knobster with hand tracking scrolling.
decrease its working good one onch per one step in baro
but if i want take the little finger for increase not working
the make same decrease the baro not increse
its wronk on my side ? pls give me a reg command to change the incrwase on the little finger

many thx

• would allow DCS without OpenComposite
• might allow Falcon BMS

Blocked on #19

When using @mbucchia's https://github.com/mbucchia/Varjo-Foveated/wiki with HTCC, DCS hangs (white screen, Not Responding in Task Manager). Varjo-Foveated log below:
2023-04-15 22:48:14 -0400: XR_APILAYER_MBUCCHIA_varjo_foveated layer (v0.1.0) is active
2023-04-15 22:48:14 -0400: Using layer: XR_APILAYER_MBUCCHIA_varjo_foveated
2023-04-15 22:48:14 -0400: Application: FREDEMMOTT_HTCC ultraleap compat hack
2023-04-15 22:48:14 -0400: XR_APILAYER_MBUCCHIA_varjo_foveated layer will be bypassed
2023-04-15 22:48:14 -0400: Using layer: XR_APILAYER_MBUCCHIA_varjo_foveated
2023-04-15 22:48:14 -0400: Requested extension: XR_KHR_D3D11_enable
2023-04-15 22:48:14 -0400: Requested extension: XR_KHR_visibility_mask
2023-04-15 22:48:14 -0400: Requested extension: XR_VARJO_quad_views
2023-04-15 22:48:14 -0400: Requested extension: XR_EXT_hand_tracking
2023-04-15 22:48:14 -0400: Requested extension: XR_KHR_win32_convert_performance_counter_time
2023-04-15 22:48:14 -0400: Requesting extension: XR_VARJO_foveated_rendering
2023-04-15 22:48:15 -0400: Application: DCS World
2023-04-15 22:48:15 -0400: Using OpenXR runtime: Varjo OpenXR Runtime 3.9.0
2023-04-15 22:48:15 -0400: Using OpenXR system: AERO
2023-04-15 22:48:15 -0400: supportsFoveatedRendering = 1
2023-04-15 22:48:15 -0400: Trying to locate configuration file at 'C:\Users\mark\AppData\Local\Varjo-Foveated\settings.cfg'...
2023-04-15 22:48:15 -0400: Not found
2023-04-15 22:48:15 -0400: Trying to locate configuration file at 'C:\Program Files\OpenXR-Varjo-Foveated\settings.cfg'...
2023-04-15 22:48:15 -0400: Found option 'peripheral_multiplier=1'
2023-04-15 22:48:15 -0400: Found option 'focus_multiplier=1'
2023-04-15 22:48:15 -0400: Found option 'turbo_mode=1'
2023-04-15 22:48:15 -0400: Found option 'no_eye_tracking=0'
2023-04-15 22:48:15 -0400: Peripheral resolution: 1908x1632 (multiplier: 1.000)
2023-04-15 22:48:15 -0400: Focus resolution 1140x1140 (multiplier: 1.000)
2023-04-15 22:48:15 -0400: Peripheral resolution: 1908x1632 (multiplier: 1.000)
2023-04-15 22:48:15 -0400: Focus resolution 1140x1140 (multiplier: 1.000)
2023-04-15 22:48:17 -0400: Creating swapchain with resolution: 1908x1632
2023-04-15 22:48:17 -0400: Creating swapchain with resolution: 1908x1632
2023-04-15 22:48:17 -0400: Creating swapchain with resolution: 1140x1140
2023-04-15 22:48:17 -0400: Creating swapchain with resolution: 1140x1140
2023-04-15 22:48:20 -0400: xrLocateViews failed with XR_ERROR_SIZE_INSUFFICIENT

Especially for PointCTRL

Filing this for visibility: it can not be worked on without changes to BMS; either:

• supporting OpenXR and VR controllers (will work with pointctrl, oculus hand tracking, and ultraleap)
• supporting absolute mouse position input (will work with pointctrl only)

Perhaps launching the settings app should offer to repair the config.

Only needed when using pointctrl as pointersource

When using Leap hand tracking, the left controller squeeze function is permanently "pressed".
With default bindings, this operates the left brake of the aircraft.
Unmapping the binding in the MSFS control options menu resolves the negative impact, although I suspect the command is still being issued due to right hand tracking never starting unless left hand has been "seen" first.

Mixed reports of compatibility issues here, e.g. works in MSFS but not in DCS

Suspecting that the game (or opencomposite) is picking the first available controller from its' list - whereas HTCC ignores anything except Oculus Touch

• DCS + G2 + PointCTRL
  • no clicks
• MSFS + G2 + PointCTRL
• HelloXR + G2 + Ultraleap
  • no tracking
• DCS + G2 + Ultraleap
• MSFS + G2 + Ultraleap

With virtual touch screen, you can click and hold the signal lamp test button on the A10C then look around, e.g. at the signal lamps :p

With virtual controller, DCS releases the button as soon as the pointer stops looking exactly at it

This is very fiddly even with real VR controllers, and just doesn't work well. Instead, the controller should be world-locked while any button is pressed.

Following Setup:
Oculus Quest 2 only
Fs2020
Developer mode enabled in headset.

right and left Controller are visible in the game. One with the right controller the click event works. Left controller has no events.

how can I help to debug the root cause?

I was trying this yesterday and it works nicely in the cockpit, however as soon as I use an external view since my hands are still being tracked, the camera starts to move all over the place and it's impossible to manage it.

It'd be nice to have a quick way to enable or disable the hand tracking

Thanks

• #5 - an alternative to pinch gestures is required
• #6 - an alternative source for hand poses is required

pls change in Option one Hand deaktivate than i can take vr Controller and Hand tracking both together

From @Wyverex42 on OpenKneeboard/OpenKneeboard#359 (comment) :

You have to parse the EDM file to read connector positions and their transformation rules, so that you're always in the right spot

Unlikely to work on this any time soon, but want to keep the note for reference.

• ignore hands that stay still for a while until they move 10cm
• ignore all 'clicks' until held for at least 100ms

Hi,

First off just wanted to comment on how much better of an experience this is in DCS than using the built in hand tracking.

I'm using a Leap Motion Controller with a Reverb G2.

Is it possible at all to add some sort of stabilization for the pointer? Between your head slightly shaking and your hands slightly shaking it can sometimes be difficult to get the pointer on the buttons long enough to do the pinch gesture.

This will allow some integration with voice attack

While touch screen mode is recommended for DCS and controller emulation works fine in MSFS, both should work.

Controller emulation is currently only usable with DCS via OpenComposite.

Probably using the hand orientation

When trying to make this work for the Ultraleap, if the 3 "workaround" options are enabled PRIOR to launching a Game such as DCS or MSFS, it launched as if there is no Pimax Crystal/VR headset attached and says something along the lines of "OpenXR is not installed or can't launch".

After a lot of troubleshooting, I uninstalled HTCC and everything worked, so tried again, installed fresh copies of everything including 5.16 of UltraLeap, and enabled the 3 Workaround options and got the same thing. If I DISABLED the three options, everything works just fine and launches all of a sudden. It's like the 3 workarounds disable VR in Windows, Odd.

I have a Pimax Crystal. During all of this, the Crystal is recognized by the computer, shows as active, and I can actually hear the game through the headset, it just says there is no VR/OpenXR headset active in the system. I can see the Pimax AND SteamVR home inside the Crystal as well. It's just as soon as those 3 'workaround' settings are enabled, VR is suddenly not recognized at the game level in Windows, but still at the hardware level as Pimax Software still sees it and interacts with it

Make sure you have Gemini v5.16.0 (October 2023) or later installed
Set ‘Hand tracking device’ to ‘OpenXR hand tracking’
Optional: enable pinch gestures
For v5.16.0, under ‘Workarounds’:
Turn on ‘Ignore XR_FB_hand_tracking_aim_pose’
Turn on ‘Force-enable XR_EXT_hand_tracking’
Turn on ‘Force-enable XR_FB_hand_tracking_aim’