Giter VIP home page Giter VIP logo

stereokit.holographicremoting's Introduction

StereoKit.HolographicRemoting

This NuGet package is an implementation of HoloLens' Holographic Remoting for StereoKit! This is an easy way to get desktop functionality onto your HoloLens, or cut down on iteration time while testing on-device. Among other things, Holographic Remoting does not require UWP, so you can use .NET Core projects with this!

Getting started

First, add StereoKit.HolographicRemoting as a NuGet package to your project!

Here's the easiest possible integration of StereoKit.HolographicRemoting! The key element here is that HolographicRemoting should be added before SK.Initialize is called, as HolographicRemoting needs to request an OpenXR extension before OpenXR starts.

using StereoKit;
using StereoKit.HolographicRemoting;

// Replace the IP here with the IP of your HoloLens!
SK.AddStepper(new HolographicRemoting("192.168.1.231"));

SK.Initialize();
SK.Run(() => { });

A more robust option would be to add the option and IP address to your command line arguments. This pairs really well with .NET Core's launchSettings.json, where you can set up multiple launch profiles with different command line arguments!

using StereoKit;
using StereoKit.HolographicRemoting;

// You can invoke this app as:
// `app.exe -remote`
// or
// `app.exe -remote 192.168.1.231`
const string defaultIp = "192.168.1.231";
int          remoteArg = Array.IndexOf(args, "-remote");
if (remoteArg != -1)
{
	bool ipArg = (remoteArg + 1 < args.Length && args[remoteArg + 1].StartsWith("-") == false);
	SK.AddStepper(new HolographicRemoting(ipArg ? args[remoteArg + 1] : defaultIp));
}

SK.Initialize();
SK.Run(() => { });

Limitations

Since Holographic Remoting means that sensor data is transmitted from the HoloLens to the PC, some HoloLens features or sensors may not be available! For example, camera aligned Mixed Reality Capture won't work through Holographic Remoting.

This package also does not implement 100% of the HoloGraphic Remoting feature set! It does all the important bits, but skips on some of the optional features such as remote speech, and custom data channels. If these are important to your project, feel free to open a PR or raise an issue!

stereokit.holographicremoting's People

Contributors

edralos avatar maluoi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

stereokit.holographicremoting's Issues

Speech commands needed

You spoke about speech as not implemented in this version. This is a feature that is very much needed. Customers ask for this all the time. What do we do to get this implemented?

Integration with SK for remoting in listen mode

Since we have overcome the issue of 10 second time-out and the issue now is integrating SK so that it can connect to a remote device. Currently the app is running on the console device using the SK.Run() method to manage the app's life cycle.
The console app "opens" the backend OpenXR in a listen mode and then waits for a connection. The problem is when the user terminates the connection from the HL2 device (shutdown effectively) the SK.Run() method completes and that is the end of the console's main() method and the app terminates.
SK being static does not have a method to "recycle" itself back to a startup condition for a new connection (at least I don't readily see one.) I have tried all sorts of things but the SK is still initialized after the SK.Run() method ends. I have not been able to return to the "first" run state. If I could somehow kill the entire session are return to the main() that would seem to be what I am trying to achieve.

Added App Listening Capability - contribution

I have modified the project to operate in "listen" mode from the command line. I would like to see this added to the current code branch so that I don't have to continue using a fork of the remoting code.
I would very much like to see voice commands added and willing to work with you/team to implement voice commands.
MR-SK-rmtng.zip

Unstable images, GPU max

I am using StereoKit.HolographicRemoting in a console app rendering to my HoloLens 2. My Microsoft Surface Pro 7 (i7 with Iris GPU) hits Max as my models get more complex (see image).

So here are alternatives: 1 Scan at different resolutions for the type of use case, 2 up-scale the GPU to brute force the solution. Naturally, the best solution is, both. I should use common sense and scan for the type of presentation. I should also have sufficient GPU for the worst-case scenario.

The rendering is excellent for stable platform such as a 3D presentation on a fixed headset such as Windows Mixed Reality Headset and most likely any of the VR types. I am satisfied I could go on to about 20-40 million polygons in fixed position. HoloLens 2 is completely in another class in just about every way imaginable.

UPDATE…
I wrote all the above based on running in the development tools (debugger, etc.)
Running the same app in a console window without tools stabilized the image and cleaned up the individual models’ visual appearances. However, here is where my ignorance of GPU is glaringly obvious, I don’t know what to expect the utilization to be in the world of GPU. The GPU numbers did not change, still about 95%. My expectation is that the GPU numbers would move up or down based on the complexity of the object. I had 14 LVSRs at 182K polys and texture size of 29mBytes, 12 7tons, 12 LAVs, 16 Humvees with all about the same poly count but texture size about 10mBytes each. That’s about 8.1 million polys.

However, again my ignorance of GPU. I loaded all the models in such a way that they are cached on the GPU (per recommended StereoKit docs). I used a different pose position to display the models in different places in the frame (StereoKit Multiple Objects demo). So, do I really have 8.1 million or do I have < 400K polys? Is that why the GPU numbers are constant?

2022-08-17

StereoKit Run method times out after 10 seconds

Whether I am in "normal" Player listens and remote connects or the other way around "Player" connects and remote listens, the SK remote app times out after 10 seconds of no connection.
Here is what happens using the StereoKit Remoting solution out of the box. Build and run (HoloLens 2 is not on). PC remote app runs for 10 seconds and stops. (Notice the Platform diagnostic)
This must be some type of fail-safe in StereoKit. The same behavior happens when the PC remote app is in listen mode. If there is no connection attempt within 10 seconds, the app terminates. Obviously, for a server listener app we want it to run continuously waiting on a connection.

[SK info] Session Performance Report:
[SK info] ______________________________________________________
[SK info] | System | Initialize | Update | Shutdown |
[SK info] ||||___________|
[SK info] | Defaults | 38.31ms | | 0.07ms |
[SK info] | FrameBegin | | 0.719ms | |
[SK info] | Input | 1.14ms | 0.030ms | 0.69ms |
[SK info] | Physics | 0.12ms | 0.883ms | 0.01ms |
[SK info] | Renderer | 0.70ms | 0.001ms | 0.03ms |
[SK info] | UI | 5.34ms | 0.029ms | 0.08ms |
[SK info] | Platform | 10880.26ms | | 33.25ms |
[SK info] | App | 14.21ms | 21808.617ms | |
[SK info] | Animation | | 0.000ms | 0.00ms |
[SK info] | Lines | 0.20ms | 0.000ms | 0.00ms |
[SK info] | Sprites | 0.29ms | 0.000ms | 0.01ms |
[SK info] | Text | | 0.002ms | 0.00ms |
[SK info] | UILate | | 0.001ms | |
[SK info] | World | 0.18ms | 0.000ms | 0.00ms |
[SK info] | FrameRender | | 0.000ms | |
[SK info] | Assets | 0.16ms | 0.000ms | 0.02ms |
[SK info] | Audio | 245.59ms | 0.000ms | 41.37ms |
[SK info] |
||________|___________|

Problem with the nuget package for Preview2

Everything is fine in the StereoKitRemoting app until you start adding other namespaces and assemblies. It seems that the debugger can't find the StereoKitC.dll in the runtime folder.

How to show camera view on remoting computer screen

The Microsoft Holographic Remoting sample app shows the HoloLens Camera video on the remoting computer's screen. I need to enable that capability in the StereoKit version.
Use case: Remote Assist via Teams. The Dynamics Remote Assist is a great app but does not allow a person wearing the HoloLens 2 device to also run a second application (Remote Assist is immersive). I need to use Teams with screen sharing from the remoting computer to another person on Teams. With the SK remoting app running AND Teams screen sharing the person remotely located could see what a person with the HoloLens is seeing. This is effectively the reverse of Dynamics Remote Assist.
How can I enable the feature?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.