Unity coordinate system and OpenXR coordinate system drift after sleep/movement on app launch about hl2ss HOT 21 CLOSED

Hello everyone,

sorry for the late reply. I've got very sick over the weekend.
Anyways I've created a PP presentation for you where I've visualized every step of the pipeline.
If you have any questions please don't hesitate to reach out to me.

arch.pdf

@jdibenes, thank you very much for the information. I indeed did not included the SU dll. I will try it again with it.

Once again I want to thank you all for your hard and astonishing work!
If you need more information or any other thing please reach out to me.

BR,

Birkenpapier

from hl2ss.

It looks like the same shift we were having with OpenXR.
We don't know of any other fix, except if someone knows how the OpenXR origin can be linked with the Unity one.

I just pushed another commit of my colleague fixing the native server app to Enzo-Kerkhof/hololens-2-sensor-streaming@505661f

@jdibenes You could decide to merge this, there is no conflict with your new release 1.0.18.0 as far as I tried.
All demo files are working, including the new recording and playing scripts. For which thank you btw.

from hl2ss.

Hi all,
Thank you for your feedback. I managed to fix this issue on my end (Unity 2020.3) using the PerceptionInterop methods. The code is in the unity_coordinate_system branch. Could you please confirm if this works for you as well?

from hl2ss.

I almost cried because it worked!

I can understand this, had a similar moment this morning when OpenXR worked for me.
I'm also looking into a point cloud registration pipeline, and yours looks very advanced already! I would be very interested in the implementation details.

from hl2ss.

@Enzo-Kerkhof
Thanks for confirming. I'll update the main branch soon.

@Birkenpapier
Are you including the Scene Understanding dll? Newer versions of the plugin depend on it. The scene understanding dll is included in the latest release in the plugin zip file.
I am also interested in the details of your work, it looks really cool.

from hl2ss.

@Mt-Perazim, thank you very much for reaching out to me.
Please contact me via e-mail: [email protected] so I can provide you the file.
This is due to some legal protection stuff unfortunately.

BR,

Birkenpapier

from hl2ss.

The drift is 5 to 10 cm in Unity 2020.3 with OpenXR.
Without OpenXR, using Unity 2019.4 and Windows Mixed Reality the drift is only when moving during the launch of the app. This is probably due to the origins being created in a slightly different place. This is breaking for any registration of Unity models to world space.

This can be fixed specifically for Unity 2019.4 and Windows Mixed Reality by linking the Unity origin as reference coordinate system for the Windows Mixed Reality runtime in the C++ plugin source files.
I can post the code if someone is interested, note this does not work for OpenXR.

from hl2ss.

Please do, I am interested.

from hl2ss.

Enzo-Kerkhof/hololens-2-sensor-streaming@cbd04bc

This code fixes any drift in the creations of Unity and native origin. Probably the OpenXR origin is created in an even more different place since this is how we found the larger drift. Reverting to older 2019 Unity solved this, but a drift still occurred if there is a movement with the HoloLens while starting the app. You can easily test this!

Sadly if the HoloLens loses its place in space and recovers the hologram positions, the Unity coordinate space drifts again and the world transforms are, again, wrong. For now, this means having to restart the app if this happens.

from hl2ss.

Hello everyone,

it seems that I am experiencing similar issues with OpenXR's and Unity's coordinate system as you can see in the video:

As you can read in my previous mention in the repository of @petergu684 I could also reproduce the same issue regarding the offset. This problem did not occur when using Unity's legacy XR plugin for spatial handling.

Is there some possible workaround besides downgrading Unity's version or calculating the new position in relation to the offset which IMO would not be that accurate?

Thank you very, very much in advance for any input!

BR,

Birkenpapier

from hl2ss.

Thank you very much for your quick answer!
So currently I only have the fix for downgrading as you did with your colleague, correct?
I will have a look at your changes and evaluate how I can implement them.

Thank you once again for your competent answer!

BR,

Birkenpapier

from hl2ss.

Could you please confirm if this works for you as well?

I'm afraid it doesn't work for me. I reverted all changes we made on our side in an updated branch and remade the plugin with your fix. The sensor streaming server is working, but I cannot send messages.

UNITY: Local IP Address is: 10.171.61.224
[MROpenXR][Info   ][06:57:20.554803][tid:1934] XrSpaceProvider_CreatingSessionOriginAnchor
UNITY: Override Coordinate System 3193719129952
Exception thrown at 0x00007FFA521759FC in 2020_U_Depth.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000F4520FF240.
The thread 0x584 has exited with code 0 (0x0).
Exception thrown at 0x00007FFA521759FC in 2020_U_Depth.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000F4520FF240.
Exception thrown at 0x00007FFA521759FC in 2020_U_Depth.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000F4520FF240.

...
RM4: Client connected
Exception thrown at 0x00007FFA521759FC in 2020_U_Depth.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000F4520FF240.
'2020_U_Depth.exe' (Win32): Loaded 'C:\Windows\system32\CompPkgSup.dll'. 
'2020_U_Depth.exe' (Win32): Loaded 'C:\Windows\system32\Windows.StateRepositoryPS.dll'. 
'2020_U_Depth.exe' (Win32): Loaded 'C:\Windows\system32\Windows.Media.dll'. 
Exception thrown at 0x00007FFA521759FC in 2020_U_Depth.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000F4520FF240.
Exception thrown at 0x00007FFA521759FC in 2020_U_Depth.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000F4520FF240.

...

RM4: Client disconnected

Directly after stating my Unity 2020.3 app it keeps giving this memory exception.
It looks like the clients can connect for sensor streams but no other communication.

Enzo-Kerkhof/hololens-2-sensor-streaming@28aff59
@Birkenpapier Above commit will fix the issue, regardless of Unity version!

from hl2ss.

Okay, nvm I'm getting a communication issue with both fixes. I'll try to find out what it is later.

Update: I think I handled a merge conflict wrong regarding the new way of Spatial Input updates. Hope this resolves the memory issue, will deploy tomorrow morning to test.

from hl2ss.

Hello all,

thank you very much for your work and expertise!
I did not manage to test it today therefore I'll come back to you tomorrow morning(CET) with the test results from the implementations of @jdibenes and @Enzo-Kerkhof.

Thank you once again and until tomorrow.

BR,

Birkenpapier.

PS: the reason why the dll tests today made problems was because I added WinRT projections to the unity project to test remote rendering and this broke the dll injection. It got loaded/unloaded all the time.

from hl2ss.

Okay, nvm I'm getting a communication issue with both fixes. I'll try to find out what it is later.
Update: I think I handled a merge conflict wrong regarding the new way of Spatial Input updates. Hope this resolves the memory issue, will deploy tomorrow morning to test.

@jdibenes Your fix works! It even keeps the Unity coordinate system and OpenXR linked after a loss of surroundings and reset.
Thanks a lot for solving this. (Yesterday I handled a merge conflict of the C# wrong and kept the SI Update lines in. That's why I had the memory exceptions.)

Please merge this into main!

from hl2ss.

Unfortunately I cannot test the new dll because I cannot be initialized:

`

'HL_Streamer.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\HL-Streaming-SensorsVS.Debug_ARM64.kpeivareh\hl2ss.dll'. Symbols loaded.
'HL_Streamer.exe' (Win32): Unloaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\HL-Streaming-SensorsVS.Debug_ARM64.kpeivareh\hl2ss.dll'
Exception thrown at 0x00007FFB0ED059FC in HL_Streamer.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000C5137FC970.
DllNotFoundException: Unable to load DLL 'hl2ss'. Tried the load the following dynamic libraries: Unable to load dynamic library 'hl2ss' because of 'C:\buildslave\unity\build\External\il2cpp\builds\external\baselib\Source\WinApi\Baselib_DynamicLibrary_WinApi.inl.h(36):Baselib_DynamicLibrary_OpenUtf16: Failed to open the requested dynamic library (0x06000000) - The specified module could not be found. (WinError:0000007e)
at hl2ss.InitializeStreams (System.UInt32 enable) [0x00000] in <00000000000000000000000000000000>:0
at hl2ss.Start () [0x00000] in <00000000000000000000000000000000>:0

'HL_Streamer.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\HL-Streaming-SensorsVS.Debug_ARM64.kpeivareh\hl2ss.dll'. Symbols loaded.
'HL_Streamer.exe' (Win32): Unloaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\HL-Streaming-SensorsVS.Debug_ARM64.kpeivareh\hl2ss.dll'
Exception thrown at 0x00007FFB0ED059FC in HL_Streamer.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x000000C5137FD760.
DllNotFoundException: Unable to load DLL 'hl2ss'. Tried the load the following dynamic libraries: Unable to load dynamic library 'hl2ss' because of 'C:\buildslave\unity\build\External\il2cpp\builds\external\baselib\Source\WinApi\Baselib_DynamicLibrary_WinApi.inl.h(36):Baselib_DynamicLibrary_OpenUtf16: Failed to open the requested dynamic library (0x06000000) - The specified module could not be found. (WinError:0000007e)
at hl2ss.MQ_SI_Peek () [0x00000] in <00000000000000000000000000000000>:0
at hl2ss.Update () [0x00000] in <00000000000000000000000000000000>:0

`

Any idea why this is happening?
I created also a completely new project to see if this might had something to do with my changes regarding the remote server streaming but it did not seems to be the case. When I build the dll with my changes from the commit 9214fe3 everything work again as intended in terms of running generally, not the offset problem.

Thank you once again for all your hard and excellent work!

BR,

Birkenpapier

from hl2ss.

Are you sure the paths to the additional include and lib directories are correct? Also, I think Unity only loads the DLL plugin once, are you restarting the editor to make sure it loads the new DLL?

from hl2ss.

All includes and lib. directories are correct as far as I checked it correctly. Using the old commit is working as intended.
Unity reloads the dll once it gots changed in the disk. Nevertheless I also restarted the editor but the problem remains.
Both projects, the complete clean one and the other previous one adapted by me, had the same errors with the new build of the dll and the hl2ss.cs, unfortunately.

BUT!

I've implemented the changes into the commit 9214fe3 regarding the coordinate system overwrite from unity to the dll and I almost cried because it worked!
In the near future I need to checkout your newest work and adapt it to my changes. Maybe I may can create a PR for another example regarding point cloud registration?

The offset is now corrected and the model gets placed almost perfect on the real object (calculation offset comming from the pointcloud registration algorithm NgeNet/GCNet).
If anyone is interested in the implementation details I can share them.

I would like to express my deepest gratitude to @jdibenes for his outstanding work without this results would never, at least not in that time, be possible and @Enzo-Kerkhof for his quick help, knowledge and support!

If I can do anything for one of you like supporting your open source, work or something else I will be glad to hear it.

BR,

Birkenpapier

PS: nevertheless it would be great to fix this DLLNotFoundException of the new version. So as it works for both of you I assume it has something to do with my local setup...

from hl2ss.

I would be very interested in the implementation details.

I will create a chart with the architecture of the complete pipeline until tomorrow or at latest Monday if this is suitable for you.
Thank you once again for your help and everything else!

BR,

Birkenpapier

from hl2ss.

That would be wonderful thank you! Monday is totally fine.

from hl2ss.

@Birkenpapier
I happened to come across this issue here via your post in the Research Mode fork of Petergu684 and am also very interested in your work. I appreciate to read that you have posted a PDF with more detailed information. Unfortunately, however, the link does not work. Would it be possible to upload the file again?

from hl2ss.