Comments (5)
@doughtmw Thanks for your detailed reply. I have dived into HoloLens 2 for a long time. I did find a solution to stream camera parameters along with the video frame. It works as expected, but it is based on socket rather than WebRTC. Besides that, its client works on Python, not C#. It should not be hard to make some adaptations.
Following your hints, I will try to stream real-time camera parameters and work out the CameraToWorldUnity
matrix.
from hmd-ego-pose.
Thanks for your interest in this project, @luffy-yu! Unfortunately, with the current MixedReality-WebRTC implementation, there is no access provided to the synced per-frame CameraToWorldUnity
matrices which are required for pose estimation while a user moves around wearing the HoloLens 2. For my work, I rigidly mounted the headset and printed out the CameraToWorldUnity
using an approach from another repo of mine, after which I used the computed matrix with HMD-EgoPose for drill tracking on the rigidly mounted headset.
I am sure there is a way to modify the WebRTC implementation to access the per-frame camera parameters (intrinsic and extrinsic) of the HoloLens 2, but I didn't have the time to investigate.
from hmd-ego-pose.
@doughtmw I figured out streaming extrinsic parameters of the left front camera, i.e., IResearchModeCameraSensor.GetCameraExtrinsicsMatrix()
.
A sample is like
[[ 7.73402284e-13 7.00452394e+31 0.00000000e+00 0.00000000e+00]
[-3.99546136e+23 4.59079390e-41 0.00000000e+00 0.00000000e+00]
[ nan 0.00000000e+00 0.00000000e+00 3.38098567e-39]
[ 2.25000000e+00 6.42855480e-40 0.00000000e+00 7.91733632e-43]]
You see some numbers are very large, while some are very small.
Does it make sense?
from hmd-ego-pose.
@luffy-yu awesome! The results seem to be a bit strange though, I'm wondering if there's an error in the way you're debugging the matrix params (float vs double for example)? In this repo, the CameraToWorldUnity
matrix parameters were:
CameraToWorldUnity:
0.99998 0.00714 -0.00042 0.00184
-0.00712 0.99973 0.02223 0.12197
-0.00058 0.02223 -0.99975 0.13058
0.00000 0.00000 0.00000 1.00000
from hmd-ego-pose.
@doughtmw I gave up the solution of getting extrinsic parameters via IResearchModeCameraSensor.GetCameraExtrinsicsMatrix()
. Instead, I refer to locatable camera to get the camera's position and rotation in the world coordinate.
from hmd-ego-pose.
Related Issues (3)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hmd-ego-pose.