jdibenes / hl2ss Goto Github PK

Hello,

The hl2ss.exe App on HoloLens 2 crashes after some time.

Here is example of my code that I run in Jupyter Notebook

from pynput import keyboard

import numpy as np
import cv2


import os, sys
sys.path.append('D:\\GitHub\\hl2ss\\viewer\\')

import hl2ss
import hl2ss_imshow

from matplotlib import pyplot as plt

# Settings --------------------------------------------------------------------

# HoloLens address
host = "192.168.137.240"

# Port
port = hl2ss.StreamPort.RM_DEPTH_LONGTHROW

# Operating mode
# 0: video
# 1: video + rig pose
# 2: query calibration (single transfer)
mode = hl2ss.StreamMode.MODE_1

# PNG filter
png_filter = hl2ss.PngFilterMode.Paeth

#------------------------------------------------------------------------------

client = hl2ss.rx_decoded_rm_depth_longthrow(host, port, hl2ss.ChunkSize.RM_DEPTH_LONGTHROW, mode, png_filter)
client.open()

# Code to create a live image in Jupyter notebook.
%matplotlib inline
from time import time, sleep
import pylab
from matplotlib import pyplot as plt
from IPython import display
from time import time, ctime, sleep
camera_live_image = True
def show_live_camera(N = -1):
    fig = plt.figure(figsize = (12,4))
    data = client.get_next_packet()
    depth = data.payload.depth
    i = 0
    while camera_live_image:
        plt.clf()
        plt.subplot(111)
        plt.imshow(depth)#, vmax = 5000, vmin = 0)
        plt.title(f'time = {ctime(time())}\n distance = {depth.mean()}')
        
       #plt.subplot(132)
       #plt.imshow()
        #plt.colorbar()
        
       # plt.subplot(133)
       # plt.imshow()

        
        display.display(pylab.gcf())

        display.clear_output(wait=True)
        sleep(1)
        # Acquire next image
        data = client.get_next_packet()
        depth = data.payload.depth
        
        print(i)
        i+=1
show_live_camera()

After sometime (sometimes 40 sometimes 600 seconds), I get errors:

ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

I check Windows Device Portal and see that hl2ss.exe App is not running anymore.

Hi, Really thanks for your great work, it helps a lot!

I just wonder is it possible to get the coordinate of a point in the world space from the coordinate in the pixel space of the PV camera frame with the help of sm_manager?

I noticed that for hand and eye tracking in sample_si_pv.py, the gaze point in the world space is acquired through these code below:

        eye_ray = si.get_eye_ray()
        eye_ray_vector = hl2ss_utilities.si_ray_to_vector(eye_ray.origin, eye_ray.direction)
        d = sm_manager.cast_rays(eye_ray_vector)
        if (np.isfinite(d)):
            gaze_point = hl2ss_utilities.si_ray_to_point(eye_ray_vector, d)

where we get eye ray vector and get the distance to the gaze point by casting the eye ray using sm_manager, then we can get the gaze point in the world space using "hl2ss_utilities.si_ray_to_point"

So I suppose If we can get a ray vector from the pixel coordinate in the PV camera frame and get the distance using sm_manager.cast_rays, then we can have the object coordinate in the world space.

I know we can get the coordinates of a point in the world space using pixel coordinates from Depth camera, but the PV camera frame is necessary for me and there is a known issue for getting PV and depth frame simultaneously and unfortunately I can't solve that.

thanks

The example for AHAT depth camera configuration has several modes for VideoProfile profile = hl2ss.VideoProfile.H265_MAIN

• H264_BASE
• H264_HIGH
• H264_MAIN
• H265_MAIN
• mro
• RAW

What do these settings do?

Hello everyone! Great work!
This repository is a must for anyone working or doing research with the HoloLens2.
Since I am a newbie, I was wondering, can we enhance the already given examples of sensor streaming with human skeleton detection libraries?
Thanks in advance!

Hi, I'm incredibly grateful for your outstanding work,it has been incredibly helpful!

What would be the process to get the 3D location in world coordinate system of a pixel obtained in the PV image?

Currently, I'm employing this process:

1. inverting the projection, using the PV intrinsics (inverted), to obtain the ray where the pixel lies.
2. Using the closest aligned depth frame, I get the depth value of that pixel
3. With the ray and the depth, I can get the x,y,z coordinates in the camera reference system
4. This is the part I have most doubts about. Currently, I'm directly applying the pose that comes coupled with the PV frame to transform the camera reference system coordinate to the world coordinate system, using the reference_to_world function and homogenous coordinates.

However, when I draw the resulting point in Unity's coordinate system it doesn't look correct. Am I right to assume that the PV pose is the transformation from PV_camera to world? Or do I need to apply another transformation before?

Hi,

I will be referencing this library in a scientific publication. Do you have DOI for this library or what that we will need to create a DOI and create a version 1.0 (or something) of this library.

Does it sounds OK?

Dear authors and fellow GitHub colleagues,

I keep getting a time out error 10060 whenever I use the TCP Port 3816 that allows the client to send messages to a Unity Application.

The repo part regarding the viewer works as a charm so far so I figure it's me, but I'd still like to know if other people encounter this and have insights for me?

Trying to use the unity plugin as we speak. Would love to use the sensor data in my own projects as well.

If it's me close this issue and sorry for bothering.

Currently, this is not possible. Is this possible, and what would be the best way to do this?

I want to show a Unity application in the hololens 2 (the Unity application is currently running on an external computer that is rendering this unity view) and streamed to a viewer (that should be in the foreground). In the background, I want to run hl2ss server.

When start python , will exit with error.

Run sample_integrator_pv.py not reaction.
So , what should i do next..
python 3.10
hl2ss 1.0.14

Thanks for the great project.

Hi! Is it possible to get the intrinsic properties of the depth camera and if so, how? Thanks!

Dear author,

I'm wondering can I use one button in the Unity to control to get a depth image under the long_throw mode, and also get the corresponding projectionMatrix and cameraToWorldMatrix at the moment this depth image is taken?

I'm trying to use the projectionMatrix and cameraToWorldMatrix to back-project the 2D coordinates on the depth image into its corresponding accurate 3D coordinates in the physical world.

Thanks for your insights!

Thank you for your sharing a nice code!

In order to align virtual models made with 3D slicers (right-handed coordinate), I have to do several coordinate system transformations (3D Slicer(right-handed)->Optical Tracking System(right-handed)->Holens2 Depth Camera(maybe right-handed)->Unity(left-hanede)).
At this time, I lack an understanding of the camera_to_world matrix that converts to Depth Camera->Unity.

First of all, In unity_sample_stickers.py script, I found 4x4 matrix called 'camera to world' .
I think it's transform matrix converting from depth camera coordinate to Unity world coordinate.
If I use a point(x,y,z) of depth camera coordinate , camera_to_world * [x,y,z,1].T and have to flip z-axis (z -> -z).

1. Is camera_to_world matrix just right handed coordinate(depth camera coordinate) -> right handed coordinate(z-fliped Unity coordinate) ??
2. Why do I flip z-axis after multiply camera_to_world matrix by point of depth coordinate ??
3. When I try to use the rotation of the camera_to_world matrix, do I not need to convert right-handed coordinate to left-handed coordinate??

Thank you for your reading!
I hope you have a nice day!

Hi, I know this repo has been out for a while but I was wondering if there was any way to reduce the amount of samples per frame in client_rm_imu.py? I am printing the x,y,z in the following way:

print(data.timestamp,',',sample.x,',', sample.y,',',sample.z,',',sample.temperature) # NOT truncated

but am only getting 700 frames per minute, which is probably bad as far as sampling goes at about 11Hz.

Dear author,

I'm testing your unity sample application on HoloLens 2 but i can not get data from the RGB and depth camera for point cloud reconstruction (when i launch the python script to get camera frames i get nothing displayed on my desktop). Is there any manipulation in the player setting or in the HoloLens to allow getting data from camera sensors ?

I'm using unity 2020.3.47f1.

Thank you in advance for your answer.

Thanks for the awesome library!

I've been using the library on Windows successfully, but when running it on macOS (12.6.6) there seem to be some problems with grabbing the video. If I run the client_stream_pv.py script, I get the following error message:

Pose at time 10668188876
[[-0.9422886   0.07307079 -0.3267378   0.        ]
 [ 0.00987262  0.9815364   0.19103575  0.        ]
 [ 0.3346641   0.17678398 -0.92560947  0.        ]
 [-0.04399353  0.08268791  0.14916018  1.        ]]
Focal length: [1459.3796 1460.1498]
Principal point: [941.4674 512.4374]
Traceback (most recent call last):
  File "/Users/aikkala/Workspace/hl2ss/viewer/client_stream_pv.py", line 89, in <module>
    cv2.imshow('Video', data.payload.image)
cv2.error: OpenCV(4.8.0) /Users/xperience/GHA-OpenCV-Python/_work/opencv-python/opencv-python/opencv/modules/highgui/src/window.cpp:971: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'imshow'

Has anyone encountered this sort of an issue before? Any suggestions how to approach this?

I think there's a chance this isn't related to your library, but is rather some other macOS issue. If I try to use the "Live preview" feature in Windows Device Portal, the video doesn't render either, and I get an error message saying

Live preview stalled due to low bandwidth or loss of connectivity. If this problem persists, consider using a lower quality stream.

This happens even with the lowest video resolution. I can, however, record a video and take a photo without problems.

Hi,

I'm trying to get the pv extrinsic matrix using pv_extrinsic_calibration.py. But I run into this error named ConnectionResetError: [Errno 104] Connection reset by peer in L42. Could you help me fix this issue?

Btw, I'm able to run client_pv.py and some other streaming scripts in the viewer folder without any problem. So I'm not quite sure what's wrong.

Thanks!

Hi,

I've created a script to interact with a Unity model and place it somewhere in the world as I'd like.
When using the Spatial Input pose and orientation of e.g. the left-hand everything works as expected. After putting the HoloLens down and running that script again there will be an exact shift of the last known position when taking the headset off and putting it down. Is this due to not having an explicit link between the native c++ coordinate system and Unity in the plugin?

Best,
Enzo

Hi,

I have a question regarding the reprojection with the captured data. I want to verify the accuracy of the pose and depth maps by projecting the object center pixel from one frame to the other (ideally, different sensors and different timestamps). However, I cannot even project the object center in depth maps to pv image at the same timestamp. Could you maybe help me check what's wrong with it?

Here are the visualizations of pv frame and ahat frame at the same timestamp

Here is one bounding box I draw for projection:

centerX, centerY (186, 204) are the center of the object in the depth frame above.

Here is the code I used for reprojection:

uv2xy = hl2ss_3dcv.compute_uv2xy(ahat_calibration.intrinsics, hl2ss.Parameters_RM_DEPTH_AHAT.WIDTH, hl2ss.Parameters_RM_DEPTH_AHAT.HEIGHT)
xy1 = hl2ss_3dcv.to_homogeneous(uv2xy)
scale = np.linalg.norm(xy1, axis=2) * (ahat_calibration.scale / hl2ss.Parameters_RM_DEPTH_AHAT.FACTOR)

# raw depth is just the depth directly from the payload
depth = hl2ss_3dcv.rm_depth_normalize(raw_depth, ahat_calibration.undistort_map, scale)
xyz = xy1 * depth[:, :, np.newaxis] # compute xyz in depth coordinate, 
xyz1 = hl2ss_3dcv.to_homogeneous(xyz)

ahat_to_pv_image = hl2ss_3dcv.camera_to_rignode(ahat_calibration.extrinsics) \
                   @ hl2ss_3dcv.rignode_to_camera(pv_calibration.extrinsics) @ pv_calibration.intrinsics

uv, _ = hl2ss_3dcv.project_to_image(xyz1, ahat_to_pv_image)
# centerX, centerY are the pixel coordinate of the center of the object in the depth frame 
proj_uv = uv[centerY, centerX]

Here is the projected center shown in pv frame (right). It seems like the center is quite far from the object. I wonder if you see anything wrong in this process? Thanks in advance!

@jdibenes
Hello, thank you for sharing your nice code!

I have 2 hololneses (A,B) .

I wanna find a reflective ball in AB frame. so it did work well in A Hololens but A hololens was broken so I tried do run same code in B hololens.

If I use the code I used when using HoloLens A on HoloLens B, the positions of objects created in Unity World are different.

Actually I did calibrate by 'pv_extrinsic_calibration.py' , 'rm_calibration_downloader.py' and 'pv_intrinsics_downloader.py' each hololens and used each calibration result.

But I don't know why it works differently.

Can you help and advise me?

Thank you for reading and I hope you have a nice day!

This error rises when I try to build the solution in Visual Studio.

Hello, thanks for the great work you have done so far on this repository!

I was wondering if it is possible to generate data frame timestamps that are synchronized with an external reference such as UTC. From what I have read, the QPC timestamps given in each data frame do not have this property. If this is not possible with the current application, any advice on how to implement UTC-synchronized timestamps would be greatly appreciated.

Thank you!

Hello and thanks again for your timely response on my previous inquiry.

I have a few more questions about the application and I would appreciate it if you could shed some light on these topics:

1. For the magnetometer values returned by the IMU, can you offer insight on what they represent/what format they are in? After checking with the Research Mode API, it reports that, "Magnetometer frames contain magnetometer values." and when listing IMU sensors, "Magnetometer – Contain absolute orientation estimation." I would be very grateful if you could clarify the ambiguity in this information.
2. For the nanosecond sensor timestamps returned by the IMU, what are these values referenced to? Is there any way to relate these sample-specific timestamps to the overarching data frame timestamp that is the same for each batch of samples?

Thank you!

I am just starting to familiarize myself with the repo, nice work you have done!

Is there a demo where the gaze is read from the HL2 and projected back onto the PV sensor?
E.g., like client_si.py projects the hands onto the PV feed, is there something to reproject the gaze to this PV feed as well?

Thanks in advance.

In the client stream series file I see a data.timestamp print. But I try to read and analyze the author's code and instructions. Due to my poor programming background, I couldn't figure out how to convert data.timestamp to UTC time. Can the author explain how to convert a timestamp to UTC via python? Thank you very much！

Hello,

thank you very much once again for creating and documenting such a nice project!
After tinkering with it around I want to visualize the results of the pointcloud directly inside the unity application.
Therefore I am streaming the pointcloud array over a custom made rest api back to the unity app.

Unfortunately I found out that the constructed pointcloud from open3d is not aligned with unitys coordinate system.
It seems to be inversed because the points are placed in my observations mirrored.

Do you have an idea or maybe any other developer how to align the reconstructed pointcloud from open3d inside unity to be placed at the same positions where they got recorded?

There is another developer which did exactly that but with unity version < 2021.2 because the methods for getting the unityWorldOrigin coordinatesystem got discontinued:
IntPtr WorldOriginPtr = UnityEngine.XR.WSA.WorldManager.GetNativeISpatialCoordinateSystemPtr();
unityWorldOrigin = Marshal.GetObjectForIUnknown(WorldOriginPtr) as Windows.Perception.Spatial.SpatialCoordinateSystem;

I would appreciate any help and thankt you very, very much.

BR,

Birkenpapier

Hi,

When capturing only one stream, like VLC left front, the timestamp is correct and the gap between timestamps is about 33ms. But when capturing more than one stream, even only two streams like VLC left front and right front, it will lose some frames and the gap between timestamp will become unstable like 33, 66, 33, 33, ...66, 99.
Now my HL2 OS version is 10.0.19041.1467. Also, in newer OS version 10.0.20348.1440 the captured VLC frames will in wrong order. For example, the VLC frames which should be 1, 2, 3, 4, 5, 6 became 1, 2, 5, 6, 4 in timestamp increasing order.
What's your OS version of HL2?

Thanks in advance!
Kelly

Hello and thank you for your excellent work.

I have developed an application that uses the ZED camera's point cloud to get the distance from a specific point in the image. E.g., I get the pixel coordinates of a specific object and then I pass these coordinates to zed's point cloud. Can I achieve something similar using what you have developed here? Is there an example which you can provide?

hi! thanks for the awesome project. everything streams super smoothly without crashing so far :) a question for depth ahat stream -
referencing demo_video.py,

if i use "ahat_profile = hl2ss.VideoProfile.H264_BASE",
I get great frame rates (40+) but the depth maps looks pretty bad. It is stored in uint8 format as well, and given the /250 conversion it translates to 4mm resolutions.
if I use "ahat_profile = hl2ss.VideoProfile.RAW",
I get a map that is stored in higher precision, giving mm level resolution, but the frame rate is slow (around 10), and it slows down the other three camera streams that i'm running in parallel (vlc/rgb drops from 30 to 20).

do you have suggestions on what setting/what can i modify to improve performance for high quality depth?

thanks!!

Hello jdibenes,

thank you very much for your excellent work. The more I dig deeper into your project the more I am amazed by your skills.
I've understood that the current architecture is designed that the socket server is running on the HL and waiting for a connection to establish the stream.
I want to change that behavior because I want to stream the data to a remote server and process it there.

Mainly I am using the demo_2d_to_3d_segmentation.py which is unfortunately only working until the commit 9214fe3. There were some changes inside the other util. files which are not compatible anymore. Maybe it is also worth to check it out and support it again with the newest version of your project?

However, the HL is obviously not accesable from a different network so I want to know if there would be a suitable solution to stream the data to a remote server and then to a python client without making deep changes in your c++ project.

Is this possible, or should there be a WinSock client at the c++ application level to stream the data to the remote python socket server and then from there back to a python viewer of yours?

BR,

Birkenpapier

I am trying to save the data obtained from a 3D point cloud into files (RGB, Depth, intrinsic, extrinsic) and then read those saved file data to reproduce it in Open3D. The reproduction program does not use the hl2ss programs, and the code seems fine, but the 3D reconstruction is not working properly. Do you have any advice for me or can you point out any potential errors in the program?

https://drive.google.com/file/d/1LnvR1yCzgk20erD2YdhW_GfyuyIBJvLQ/view?usp=drive_link

Hi! I have just read in the README file that mixed reality capture is supported as an additional feature but I see no references anywhere in the project. Could you please explain to me where can I find it or at least if there is something I could do about it?
Thanks

Hi, great repository and thanks for your work!

I'm encountering timeout troubles connecting to port 3809 via the plugin. All the other client scripts have worked flawlessly, but any scripts relying on establishing a connection for IPCPort.REMOTE_CONFIGURATION face the same issue.

For example, the client_rc.py script errors on a timeout connecting at the first instance of attempting to open the port in this line.

Any ideas on how to fix this? I can provide more information on my setup if required.

Thanks!

I followed the steps mentioned in the readme file however when trying to build the release ARM64 I keep getting this error. I have installed the appxbundle on hololens, enable Research mode as well as Developer Mode. I have given permission for the app to access the camera, microphone, etc. My hololens were already paired due to previous apps.

Has this ever happened to you?

• I am using Visual Studio 2022 17.4.1

I think old version has the calibration folder. Please, chech this.

Hi guys,
Thanks for the project jdibenes, it's been really useful to capture and align sensor streams.

Right now, I am working on trying to align the LT, PV and VLC (Right front) cameras.
I am mostly having issues with the VLC part of the equation at the moment.

Using sample_rgbd_ahat_vlc.py I am getting this (note that this is the AHAT and not the LongThrow camera)

We can already see a duplicated finger here.

I have also noticed that AHAT alignment seems to be much more difficult to pull off, probably because of its fisheye aspect, and the great differences in resolution (especially when trying to align PV and AHAT).

By the way, if anyone has found a way to properly align AHAT and PV, or get rid of the noise/projection issues, I'd be interested to hear about it!

Now to the LT/VLC issue:
I am writing a code that aligns LT with PV and LT with VLC.
It is inspired by the provided samples, and the LT frame is used as a reference, nearest PV and VLC frames (Right front) are then captured, the transformation matrices are applied, and the images are displayed:
Here is the content of the code:

from pynput import keyboard

import multiprocessing as mp
import numpy as np
import time
import os
import cv2
import hl2ss
import hl2ss_mp
import hl2ss_3dcv
import json

# HoloLens address
host = '10.44.161.20'

# Calibration folder
calibration_path = '../calibration'
storage = './recording/'

# Value with which we multiply depth pixels so the depth map is brighter (experimental & empiric)
depth_pixel_factor = 128

# Front RGB camera parameters
pv_focus = 600
pv_width = 640
pv_height = 360
pv_framerate = 30

# Video encoding profile
pv_profile = hl2ss.VideoProfile.H264_MAIN

#Bitrates
vlc_bitrate = 1*1024*1024
pv_bitrate = 1*1024*1024

# Max depth in meters
max_depth = 3.0

if __name__ == '__main__':
    enable = True

    def on_press(key):
        global enable
        enable = key != keyboard.Key.esc
        return enable

    listener = keyboard.Listener(on_press=on_press)
    listener.start()

    hl2ss.start_subsystem_pv(host, hl2ss.StreamPort.PERSONAL_VIDEO)

    # Calibrating cameras - forcing download to make sure we have recent calibration
    calibration_lt = hl2ss_3dcv._download_calibration_rm(host, hl2ss.StreamPort.RM_DEPTH_LONGTHROW)
    calibration_rf = hl2ss_3dcv._download_calibration_rm(host, hl2ss.StreamPort.RM_VLC_RIGHTFRONT)

    # Compute depth-to-rgb registration constants
    uv2xy = hl2ss_3dcv.compute_uv2xy(calibration_lt.intrinsics, hl2ss.Parameters_RM_DEPTH_LONGTHROW.WIDTH, hl2ss.Parameters_RM_DEPTH_LONGTHROW.HEIGHT)
    xy1, scale = hl2ss_3dcv.rm_depth_compute_rays(uv2xy, calibration_lt.scale)

    # Configuring, initializing and starting producer
    producer = hl2ss_mp.producer()
    producer.configure_rm_depth_longthrow(True, host, hl2ss.StreamPort.RM_DEPTH_LONGTHROW,hl2ss.ChunkSize.RM_DEPTH_LONGTHROW, hl2ss.StreamMode.MODE_1,hl2ss.PngFilterMode.Paeth)
    producer.configure_rm_vlc(True, host, hl2ss.StreamPort.RM_VLC_RIGHTFRONT, hl2ss.ChunkSize.RM_VLC,hl2ss.StreamMode.MODE_1, hl2ss.VideoProfile.H264_MAIN, vlc_bitrate)
    producer.configure_pv(True, host, hl2ss.StreamPort.PERSONAL_VIDEO, hl2ss.ChunkSize.PERSONAL_VIDEO,hl2ss.StreamMode.MODE_1, pv_width, pv_height, pv_framerate, hl2ss.VideoProfile.H264_MAIN, pv_bitrate, 'rgb24')

    producer.initialize(hl2ss.StreamPort.RM_DEPTH_LONGTHROW, 2*5)
    producer.initialize(hl2ss.StreamPort.PERSONAL_VIDEO, 2 * 30)
    producer.initialize(hl2ss.StreamPort.RM_VLC_RIGHTFRONT, 2*30)

    producer.start(hl2ss.StreamPort.RM_DEPTH_LONGTHROW)
    producer.start(hl2ss.StreamPort.PERSONAL_VIDEO)
    producer.start(hl2ss.StreamPort.RM_VLC_RIGHTFRONT)

    # Initializing manager, consumer, and creating sinks
    manager = mp.Manager()
    consumer = hl2ss_mp.consumer()

    sink_lt = consumer.create_sink(producer, hl2ss.StreamPort.RM_DEPTH_LONGTHROW, manager, ...)
    sink_rf = consumer.create_sink(producer, hl2ss.StreamPort.RM_VLC_RIGHTFRONT, manager, None)
    sink_pv = consumer.create_sink(producer, hl2ss.StreamPort.PERSONAL_VIDEO, manager, ...)

    sink_lt.get_attach_response()
    sink_rf.get_attach_response()
    sink_pv.get_attach_response()

    # Make directory to save observations
    #folder_time = int(time.time() * 1000)
    #storage_path = storage + str(folder_time) + "/"
    #if not os.path.exists(storage_path):
        #os.makedirs(storage_path)

    while (enable):
        sink_lt.acquire()

        # Using LongThrow as a reference, getting the most recent frame
        _, data_lt = sink_lt.get_most_recent_frame()
        if ((data_lt is None) or (not hl2ss.is_valid_pose(data_lt.pose))):
            continue

        # Getting closest RF frame
        _, data_rf = sink_rf.get_nearest(data_lt.timestamp)
        if ((data_rf is None) or (not hl2ss.is_valid_pose(data_rf.pose))):
            continue

        # Getting closest PV frame
        _, data_pv = sink_pv.get_nearest(data_lt.timestamp)
        if ((data_pv is None) or (not hl2ss.is_valid_pose(data_pv.pose))):
            continue

        # Getting depth and RGB images from payload
        depth = hl2ss_3dcv.rm_depth_undistort(data_lt.payload.depth, calibration_lt.undistort_map)
        depth = hl2ss_3dcv.rm_depth_normalize(depth, scale)
        rf_image = cv2.remap(data_rf.payload, calibration_rf.undistort_map[:, :, 0], calibration_rf.undistort_map[:, :, 1], cv2.INTER_LINEAR)
        pv_image = data_pv.payload.image

        # Creating pv intrinsics from data
        pv_intrinsics = hl2ss.create_pv_intrinsics(data_pv.payload.focal_length, data_pv.payload.principal_point)

        #Depth related matrices
        lt_points = hl2ss_3dcv.rm_depth_to_points(xy1, depth)
        lt_to_world = hl2ss_3dcv.camera_to_rignode(calibration_lt.extrinsics) @ hl2ss_3dcv.reference_to_world(data_lt.pose)
        world_to_lt = hl2ss_3dcv.world_to_reference(data_lt.pose) @ hl2ss_3dcv.rignode_to_camera(calibration_lt.extrinsics)

        #World to PV and world to VLC images
        world_to_pv_image = hl2ss_3dcv.world_to_reference(data_pv.pose) @ hl2ss_3dcv.camera_to_image(pv_intrinsics)
        world_to_rf_image = hl2ss_3dcv.world_to_reference(data_rf.pose) @ hl2ss_3dcv.rignode_to_camera(calibration_rf.extrinsics) @ hl2ss_3dcv.camera_to_image(calibration_rf.intrinsics)

        #Projecting depth points to world frame of reference
        world_points = hl2ss_3dcv.transform(lt_points, lt_to_world)

        #Mapping of PV to LT pixel and VLC to LT pixel
        pv_uv = hl2ss_3dcv.project(world_points, world_to_pv_image)
        rf_uv = hl2ss_3dcv.project(world_points, world_to_rf_image)

        #Mapping LT pixels to PV image and converting BGR to RGB
        pv_image = cv2.remap(pv_image, pv_uv[:, :, 0], pv_uv[:, :, 1], cv2.INTER_LINEAR)
        pv_image_rgb = cv2.cvtColor(pv_image, cv2.COLOR_BGR2RGB)

        #Mapping LT pixels to RF image
        rf_image = cv2.remap(rf_image, rf_uv[:, :, 0], rf_uv[:, :, 1], cv2.INTER_LINEAR)

        #Creating masks for depth images, duplicating depth image and applying masks to set depth to 0 on pixels outside of colorized/grayscale range
        mask_uv_pv = hl2ss_3dcv.slice_to_block((pv_uv[:, :, 0] < 0) | (pv_uv[:, :, 0] >= pv_width) | (pv_uv[:, :, 1] < 0) | (pv_uv[:, :, 1] >= pv_height))
        mask_uv_rf = hl2ss_3dcv.slice_to_block((rf_uv[:, :, 0] < 0) | (rf_uv[:, :, 0] >= hl2ss.Parameters_RM_VLC.WIDTH) | (rf_uv[:, :, 1] < 0) | (rf_uv[:, :, 1] >= hl2ss.Parameters_RM_VLC.HEIGHT))
        depth_pv = depth
        depth_rf = depth
        depth_pv[mask_uv_pv] = 0
        depth_rf[mask_uv_rf] = 0

        #This line was in the ahat lf sample code but doesn't seem to be used
        lt_to_rf_image = hl2ss_3dcv.camera_to_rignode(calibration_lt.extrinsics) @ hl2ss_3dcv.reference_to_world(data_lt.pose) @ hl2ss_3dcv.world_to_reference(data_rf.pose) @ hl2ss_3dcv.rignode_to_camera(calibration_rf.extrinsics) @ calibration_rf.intrinsics

        image_lt_rf = np.hstack((depth_rf[:, :, 0] / np.max(depth_rf), rf_image / 255))
        image_lt_pv = np.hstack((hl2ss_3dcv.rm_depth_to_rgb(depth_pv) / max_depth, pv_image_rgb / 255))
        cv2.imshow('LT_RF', image_lt_rf)
        cv2.imshow('LT_PV', image_lt_pv)
        cv2.waitKey(1)

    sink_lt.detach()
    sink_rf.detach()
    sink_pv.detach()
    producer.stop(hl2ss.StreamPort.RM_DEPTH_LONGTHROW)
    producer.stop(hl2ss.StreamPort.RM_VLC_RIGHTFRONT)
    producer.stop(hl2ss.StreamPort.PERSONAL_VIDEO)
    listener.join()
    hl2ss.stop_subsystem_pv(host, hl2ss.StreamPort.PERSONAL_VIDEO)

Here are the images I am getting:
On top LT and VLC
On the bottom, LT and PV
(There is also some brightness difference I have to investigate, but it is less relevant. I had artificially changed the brightness so the depth images could be used with segmentation algorithms)

As you can see, the effect of the duplicated/ghost hand is even more noticeable than in the AHAT/VLC sample, while the LT/PV seems fine. Note that my hand is not moving on these captures.
However, when no hands, arms or objects are close to the camera, and when it is just the environment, there seems to be no duplication issues:

I was thinking that I may have made a mistake in the masks, or some of the transformation matrices, or accidentally written some parts of an image on top of another one, but I can't see where.

Is this simply a hardware + matrices imprecision issue? Are these simply the Hololens 2 limitations?
I was curious to see if anyone had encountered this issue and found a fix/workaround.

Thanks!

Hi @jdibenes and thanks a lot for the contribution. I am working in a MR project and we are trying to test the data extracted using the HoloLens 2. The main idea is to obtain the pointcloud of a full scene to later on train an algorithm. Now we are in the testing phase but later on it will ideal even to create our own dataset using the HoloLens 2 headset and potentially this library. The problem that I cannot achieve the transfer of data between the Hololens and my pc. I am usinf the following specifications:

pc OS: Linux 20.04

Miniconda env with:
Python 3.9
I have install the libraries opencv-python, av, numpy, pynout and open3d.

I have achived the Preparation and Installation steps. (installed the version v1.0.25.0)
I have checked the connectivity of the headset and my pc accesing the data portal via HoloLens IP address.

Then I am trying to run the python script client_stream_pv.py and I got the following error:

Traceback (most recent call last):
  File "/home/ither1/hl2ss/viewer/client_stream_pv.py", line 83, in <module>
    client.open()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 1395, in open
    self.get_next_packet()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 1400, in get_next_packet
    data = super().get_next_packet()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 741, in get_next_packet
    return self._client.get_next_packet()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 388, in get_next_packet
    self._unpacker.extend(self._client.recv(self._chunk_size))
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 276, in recv
    chunk = self._socket.recv(chunk_size)
ConnectionResetError: [Errno 104] Connection reset by peer

Any idea about what could be happening? Thanks in advance!

Hello,

When I ran pv_extrinsic_calibration.py, it raised an error:

Traceback (most recent call last):
  File "pv_extrinsic_calibration.py", line 28, in <module>
    client_rc = hl2ss.tx_rc(    host, hl2ss.IPCPort.REMOTE_CONFIGURATION)
AttributeError: module 'hl2ss' has no attribute 'tx_rc'

BTW, thanks for your work. It is really helpful!

Dear Juan Carlos Dibene,

First of all, thank you a lot for all the well written code and documentation. I've been able to run everything smoothly with your readme documents.
For my project, I would like to interact with preloaded objects/holograms already in the Unity app.
Would there be a method to find the key/id of the objects that are present in the Unity scene and be able to use all communication functions you have made as is?

Best,
Enzo

Hi All,

Please see settings for AHAT depth mode below. The images I get are 512x512 in size. However, the specs on the device webpage https://learn.microsoft.com/en-us/hololens/hololens2-hardware show that the depth camera is 1024x1024 (1MPx).

Why do I get smaller depth images?

AHAT

# Settings --------------------------------------------------------------------

# Port
port = hl2ss.StreamPort.RM_DEPTH_AHAT

# Operating mode
# 0: video
# 1: video + rig pose
# 2: query calibration (single transfer)
mode = hl2ss.StreamMode.MODE_1

# PNG filter
png_filter = hl2ss.PngFilterMode.Paeth
bitrate = 8*1024*1024
profile = hl2ss.VideoProfile.H265_MAIN

#------------------------------------------------------------------------------
try:
    client.close()
except:
    print('client it not initialize')
client = hl2ss.rx_decoded_rm_depth_ahat(host, port, hl2ss.ChunkSize.RM_DEPTH_AHAT, mode, profile, bitrate)
client.open()

LongThrow

# Settings --------------------------------------------------------------------

# Port
port = hl2ss.StreamPort.RM_DEPTH_LONGTHROW


# Operating mode
# 0: video
# 1: video + rig pose
# 2: query calibration (single transfer)
mode = hl2ss.StreamMode.MODE_1

# PNG filter
png_filter = hl2ss.PngFilterMode.Paeth
bitrate = 8*1024*1024
#------------------------------------------------------------------------------
client = hl2ss.rx_decoded_rm_depth_longthrow(host, port, hl2ss.ChunkSize.RM_DEPTH_LONGTHROW, mode, png_filter)
client.open()

Thanks for your work! But I want to know what the difference between your work and "https://github.com/cgsaxner/HoloLens2-Unity-ResearchModeStreamer" is ?

I was looking to install the hl2ss app in the Hololens-2, however whenever I run the code on VS Studio 2019 this error appears and I can´t quite figure out what it stands for and why it happens.

Is it possible to return a stream of X,Y,Z coordinates of where HoloLens2 thinks it is?

Hello jdibenes,

Thank you very much for your excellent work. Your work help us a lot as we are not familiar with hololens2. We recently want to integrate webrtc into the project, but it does not support ARM64. Does hl2ss support ARM? looking forward to your answer

Dear author

The IP address of Hololens is correct but the python script still can not connect.

TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Hi,

I'm trying to stream data from multiple sensors simultaneously. I have followed the examples shown in the viewer folder to stream data from HoloLens2. However, I found most of them only able to retrieve 1 or at most 2 sensors. In my case, I want to extract frames from pv camera, and also the corresponding frames from 4 grayscale cameras and depth. Could you maybe provide an example script?

I have wrote a script based the hl2ss_mp and hl2ss_utility before update. Here is an updated version following the most recent commits. Could you help me check whether this is best practice since I feel like I shouldn't start a separate pv client? In my case, if I combine everything in the producer and consumer, the program will just freeze for some reason.

`

hl2ss.start_subsystem_pv(host, hl2ss.StreamPort.PERSONAL_VIDEO)

enable = True

def on_press(key):
    global enable
    enable = key != keyboard.Key.esc
    return enable

listener = keyboard.Listener(on_press=on_press)
listener.start()

producer = hl2ss_mp.producer()
producer.configure_rm_vlc(True, host, port1, hl2ss.ChunkSize.RM_VLC, hl2ss.StreamMode.MODE_1, profile, vlc_bitrate)
producer.configure_rm_vlc(True, host, port2, hl2ss.ChunkSize.RM_VLC, hl2ss.StreamMode.MODE_1, profile, vlc_bitrate)
producer.configure_rm_vlc(True, host, port3, hl2ss.ChunkSize.RM_VLC, hl2ss.StreamMode.MODE_1, profile, vlc_bitrate)
producer.configure_rm_vlc(True, host, port4, hl2ss.ChunkSize.RM_VLC, hl2ss.StreamMode.MODE_1, profile, vlc_bitrate)
producer.configure_rm_depth_longthrow(True, host, hl2ss.StreamPort.RM_DEPTH_LONGTHROW, hl2ss.ChunkSize.RM_DEPTH_LONGTHROW, hl2ss.StreamMode.MODE_1, hl2ss.PngFilterMode.Paeth)    

producer.initialize(port1, hl2ss.Parameters_RM_VLC.FPS * buffer_length)
producer.initialize(port2, hl2ss.Parameters_RM_VLC.FPS * buffer_length)
producer.initialize(port3, hl2ss.Parameters_RM_VLC.FPS * buffer_length)
producer.initialize(port4, hl2ss.Parameters_RM_VLC.FPS * buffer_length)
producer.initialize(hl2ss.StreamPort.RM_DEPTH_LONGTHROW, hl2ss.Parameters_RM_DEPTH_LONGTHROW.FPS * buffer_length)

producer.start(port1)
producer.start(port2)
producer.start(port3)
producer.start(port4)
producer.start(hl2ss.StreamPort.RM_DEPTH_LONGTHROW)

manager = mp.Manager()
# consumer = hl2ss_utilities.consumer()
consumer = hl2ss_mp.consumer()
sink_vlc1 = consumer.create_sink(producer, port1, manager, None)
sink_vlc2 = consumer.create_sink(producer, port2, manager, None)
sink_vlc3 = consumer.create_sink(producer, port3, manager, None)
sink_vlc4 = consumer.create_sink(producer, port4, manager, None)
sink_depth = consumer.create_sink(producer, hl2ss.StreamPort.RM_DEPTH_LONGTHROW, manager, ...)

sinks = [sink_vlc1, sink_vlc2, sink_vlc3, sink_vlc4, sink_depth]
[sink.get_attach_response() for sink in sinks]

# pv camera
client_pv = hl2ss.rx_decoded_pv(host, hl2ss.StreamPort.PERSONAL_VIDEO, hl2ss.ChunkSize.PERSONAL_VIDEO, hl2ss.StreamMode.MODE_1, width, height, framerate, profile, pv_bitrate, 'bgr24')
client_pv.open()

while (enable):
    data_pv = client_pv.get_next_packet()

    _, data_vlc1 = sink_vlc1.get_nearest(data_pv.timestamp)
    _, data_vlc2 = sink_vlc2.get_nearest(data_pv.timestamp)
    _, data_vlc3 = sink_vlc3.get_nearest(data_pv.timestamp)
    _, data_vlc4 = sink_vlc4.get_nearest(data_pv.timestamp)

    _, data_depth = sink_depth.get_nearest(data_pv.timestamp) # payload.depth --> ndarray, uint16

`

I saw cv2.remap is used to register RGB with Depth images.

There is a scenario that I need to get the depth information of one pixel in the origin RGB image (i.e., before remap).

I investigated the cv2.remap function and found that it is not easy.

I am using a hack approach to do this:

1. Create a same-size empty image;
2. Set the target pixel (x,y) a non-zero value, e.g. 255;
3. Remap;
4. Find the non-zero value, get its index (x', y');
5. Get depth info at the position of (x', y').

However, I think this approach is not much efficient.

Another similar scenario is that get the RGB pixel corresponding to one pixel in the depth image.

Hi, I'm trying to load a obj file, saved in a server, while the app unity is running on the hololens. I've created a new command to send the url of the file and download it with the same method of OBJLoader project. But It seems to not work. Someone has an idea?

My code:

public uint MSG_LoadOBJFile(byte[] data)
{
//string url = Encoding.UTF8.GetString(data);
// Create a UnityWebRequest to download the OBJ file
UnityWebRequest request = UnityWebRequest.Get(url);

    // Send the request and wait for it to complete
    request.SendWebRequest();
    while (!request.isDone)
    {
        System.Threading.Thread.Sleep(1);
    }

    // Check for errors during the download
    if (request.result != UnityWebRequest.Result.Success)
    {
        Debug.LogError("Failed to download OBJ file. Error: " + request.error);

    }

    // Get the downloaded data as bytes
    byte[] downloadedData = request.downloadHandler.data;

    // Create a stream and load the OBJ file
    var textStream = new MemoryStream(downloadedData);
    var loadedObj = new OBJLoader().Load(textStream);

    // Add the loaded object to your game
    uint id = AddGameObject(loadedObj);
    return id;
}