Comments (8)
I don't have a comprehensive answer for you but the short answer is, the sensor takes a while to open and close (say 1-2 seconds, but I never tested it), so if you call your uvToWorld in each frame, there is obviously a problem there. You may want to let the sensor keep running in the background and figure out someway to query the uv by another function, for example via shared variable to let the thread know which uv you want to back-project and then save the result somewhere you can access from main thread. Your uv is basically the i and j in LongDepthSensorLoop. Hope that helps.
from hololens2-researchmode-unity.
@petergu684 Thanks for your answer!
I have set a boolean to control OpenStream() just run once and get the buffer every call, and it works! Although the result of the coordinate transformation now has obvious errors, I will manage to solve it next!
( maybe it's caused by that the input point comes from the MixedRealityCapture video stream via WIndowsDevicePortal is 1280x720 , and the LongThrow mode's resolution is 320x288, and my resolution transformation for this is wrong)
Thanks again for this repo and your reply!
from hololens2-researchmode-unity.
@Tolerm how did you get this working? when i try it tells me i overloded "std::invoke'. Was there something special you did with your header file?
from hololens2-researchmode-unity.
@EliasMosco Are you talking the OpenStream()
? I just used a boolean , in HL2ResearchMode.h
,
struct HL2ResearchMode : HL2ResearchModeT<HL2ResearchMode>
{
......
private:
...
std::atomic_bool m_isDepthSensorStreamOpen = true; // this boolean
static void myFunc(HL2ResearchMode* pHL2ResearchMode);
...
}
and inHL2ResearchMode.cpp
, it's like below
void HL2ResearchMode::MyFunc(HL2ResearchMode* pHL2ResearchMode)
{
if(pHL2ResearchMode->m_isDepthSensorStreamOpen)
{
pHL2ResearchMode->m_longDepthSensor->OpenStream();
pHL2ResearchMode->m_isDepthSensorStreamOpen = false;
}
...
}
it's very simple, you can change it according to your demands. If you are looking for a way to control multi-sensors stream control, the Mirosoft official docs ResearchMode-ApiDoc.pdf
and the repo HoloLens2ForCV
may help you , they can both be found in https://github.com/microsoft/HoloLens2ForCV
from hololens2-researchmode-unity.
Hello @Tolerm, did you manage to get it to work after all? If so, do you happen to have a repository? I'm trying to achieve the same thing!
from hololens2-researchmode-unity.
Hello @Tolerm, did you manage to get it to work after all? If so, do you happen to have a repository? I'm trying to achieve the same thing!
Well, I'm working in the transformation of PV Camera and Depth Camera, but I'm not sure if the method I'm using now is correct, in principle, same cameras and same mode according the ResearchMode ApiDoc, so the transformation in HoloLens2 should be the same as HoloLen1, and the works in HoloLen1 should be valuable. Finally, I mainly referred to the following:
- petergu684 's this repo;
- HoloLens2ForCV's sample, StreamRecorder https://github.com/microsoft/HoloLens2ForCV/tree/main/Samples/StreamRecorder#stream-recorder-sample
- LisaVelten's code, https://github.com/microsoft/HoloLensForCV/issues/119#issuecomment-553098740
if your work is the same as mine, get a (u,v) in PV Camera and transform it to 3D point in Unity's coord system with the depth of this image point, then my method now like :
- get a (u,v) in PV Camera coordinate system;
- (u,v) multiply by the inversed intrinsics of PV camera, the intrinsics got from camera calibration , and get a (x,y,z) coordinate;
- set a vector as (x,y,-1,1)οΌand this vector multiply by the PVToWorld matrix to get a world coodinate system point (X,Y,Z,W), the PVToWorld matrix can be found in the StreamRecorder project and understand it there
- now transform the (X,Y,Z,W) to the (u',v') in the Depth Camera according pertergu684's method , rigid transformation and perspective projection in Camera Model;
- get the depth of (u',v') according petergu684's method, and then get the world point in Unity's coordinate system (or other coordinate system)
if your work is to transform a (u,v) point in Depth Camera image coord system -> (X,Y,Z,W) in a given world coord system, you may do it like this:
- use the method MapImagePointToCameraUnitPlane(uv,xy)
- give a world coord system Windows::Perception::Spatial::SpatialCoordinateSystem (please refer to petergu684's repo) , and get the extrinsic matrix
- (x,y,1) * the extrinsic matrix, it's the reverse process of petergu684's work
from hololens2-researchmode-unity.
@Tolerm Thanks a lot for your answer! Could you provide more details about how you computed the inrtrinsics of the PV camera?
from hololens2-researchmode-unity.
@Tolerm Thanks a lot for your answer! Could you provide more details about how you computed the inrtrinsics of the PV camera?
I first tried the UndistortedProjectionTransform here, but the matrix I got was like this :
[ 992.376 0 0 0;
0 -993.519 0 0;
0 0 1 0;
628.272 378.181 0 1] .
I don't understand the meaning of minus sign '-' before 993.519 , and my PV camera always runs under one setting (1280x720, 30fps) , so I think the intrinsics of my PV camera won't change, I finally used the APP 'Camera Calibrator' of Matlab with a calibration checkerboard and get the intrinsic parameters.
BTW, run the Stream Recorder app will get some files of PV camera and Depth camera, and there will be extrinsics of PV camera, extrinsics of Long Throw mode Depth Camera ( this extrinsic matrix may be relative to the Rig Coordinate system)
from hololens2-researchmode-unity.
Related Issues (20)
- researchMode object seems to be null HOT 3
- [UWPDataAccess] No CurrentSpatialCoordinatesystem or EyesAPI Not available
- How to make it works in unity and show in Hololens 2? HOT 9
- how to run it on holographic remoting for play mode? HOT 1
- How can SaveAHAT same as the view in Unity?
- Can't connect the HoloLens 2 to the python server HOT 1
- How to get local accelerometer frame (Transform)?
- Getting the WorldOriginPtr with unity > 2021.2 HOT 12
- "Catastrophic Failure" in HL2ResearchMode Constructor HOT 2
- RigToWorld matrix in Unity
- How to unproject a 2D point in an captured image back to its accurate location in the physical world? HOT 2
- Pannel's Button does not working.
- IMU framerate
- PointCloud does not update after opening menu panel
- How to implement functions from ResearchMode.h HOT 1
- How do I capture an RGB image and a depth image at the same time using this plugin?
- Accuracy of HoloLens2 Depth Camera HOT 1
- How can I get the depth of a specific point in Hololens Camera
- OpenXR supported? Plugin namespace not recognized
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hololens2-researchmode-unity.