There is some code that is duplicate in the Depth and Firmware modules, e.g. "firmware_get_device_version". We should investigate ways of reduce the duplication.

C code is specifically needed because the header file is C based and we also need to test to confirm the C header stays compatible for C projects (right now all users of record are CPP). Over time we will make CPP, C# version of the simple sample for users of the SDK.

YUV support for resolutions higher than 720p. Azure Kinect DK ISP cannot handle higher resolution but this could be achieved through color conversion in media foundation similarly to BGRA mode support. Linux needs to be figured out as well.

OpenGL has limitation that only allow one context per thread, unlike DirectX allow handles simultaneously to access GPU device. Thus, if a client using OpenGL while invoking transformation API on Linux (which internally using OpenGL for transformation engine optimization), it does not work. This is not an issue for Windows because DirectX handles this. Thus, we need to move transformation engine optimization on its own thread to enable Linux to work when Client is also using OpenGL (e.g. Azure Kinect Viewer)

• .NET Standard wrapper
• Functional tests
• Unit tests
• CI Automation
• NuGet Packaging
• SDK Packaging

C# support

Target .NET Frameworks

• .NET Framework (desktop apps)
• .NET Core (Windows and Linux)

Target UI Framewoks

• WPF
• WinForms

This is already covered in the documentation for the BGRA format, but it is a useful structure for consumers to be able to decode the component of a pixel.

Provide simple API to provide gravity direction

Contributing.md is currently a minimal doc describing Microsoft contributing rules. It needs to be updated to contain the following info:

• Which branch to submit PRs against
• Our branching strategy
• An explanation of our private and public CI builds
• How to debug functional test failures.

Currently we only have the firmware blob in the MSI. This means if you want to try out a firmware you must install the new SDK. Or if you want to roll back after installing a new SDK and firmware, that you need to install the old SDK somewhere.

Ideally, we should have a way to directly get at an arbitrary firmware version.

We need to add some documentation for this.

Proposal

For logging in Azure Kinect we are planning to add an interface that will allow log messages from the SDK to be forward to the users executable. This will allow customers to merge K4a.dll logs into their own logging solution for better debuggability of system issues. It will also allow tools like K4A viewer can show error logs in real time in a dedicated window if it so chooses.

The proposal is to add k4a_register_debug_message_handler(). The function will take in a function pointer to deliver the messages to, a context for the callback function, and minimum log level setting the SDK should be sending to the user.

The registration API will only allow 1 callback function to be registered. If the user wants to change the logging level or the callback function, then the existing callback function needs to be unregistered first, and then a new callback function can be added. (I suppose we could allow the registration API to be called with the same callback function and treat that as an update to the logging level for a runtime change that doesn’t require unregistration. Anyone think that would be useful?)

The callback function will be invokable by any caller attempting to log. So if there are 3 threads running on 3 CPU, and all of then generate an error message at the same time, then all three of them can call into the users logging callback function at the same time. The user will need to protect against this.

The existing logging features (through the ENV variables) will continue to work as implemented when there is NO callback registered. When a callback is registered, then our current logging will be disabled.

The supported logging levels:

typedef enum
{
    K4A_LOG_LEVEL_CRITICAL = 0,
    K4A_LOG_LEVEL_ERROR,
    K4A_LOG_LEVEL_WARNING,
    K4A_LOG_LEVEL_INFO,
    K4A_LOG_LEVEL_TRACE,
} k4a_log_level_t;

Callback function signature:

typedef void(k4a_debug_logging_cb_t)(void *context, k4a_log_level_t level, char *message);

Register & unregister API’s

K4A_EXPORT k4a_result_t k4a_register_debug_message_handler(k4a_debug_logging_cb_t pfn, void *context, uint32_t min_log_level);
K4A_EXPORT k4a_result_t k4a_unregister_debug_message_handler(k4a_debug_logging_cb_t pfn);

Considerations

Using a callback function is a different programming paradigm when we currently use with K4A. In the case of debug messages, if we want to stay API philosophy of implementing a get_debug_message() API (similar to get_capture in that the user must own the thread), then we would be forced to implement a lot more code, that would likely be a duplication of functionality that the users logger would need to implement. Having an API like get_debug_message() would mean our SDK would have to queue up and store the debug messages in the event the users logger could not keep up. We would also then have to deal with dropped debug messages when the queue got full, and have to get the locking mechanics correct for having a single reader and multiple writers. Additionally we would also then want to consider how long a message has been in our queue and consider how the message will get the correct time stamp applied to it.

All these factors need to be taken into account by any logger that is accepting messages from multiple writers. So the logic that the SDK would need would be duplicated by the customers logger. So for this reason we are not considering the get_debug_message() style of API and instead going with a callback that can forwarded by any one of the SDK threads when an issue needs to be logged.

The k4a_color_resolution_t maps to a known set of width/height values which are useful to an application allocating memory for images.

Adding a function that maps those to values would be helpful for developers.

Similarly for depth modes from k4a_depth_mode_t, however the resolutions for each of those modes may be hardware dependent. Should we make the function dependent on a device handle perhaps?

When the depth mode doesn't support the fps mode the user sets, the error message is very confusing: “depth_fps_and_mode_supported == true returned failure in validate_configuration()”.

Changing the error message to something more human-readable will help our customers to figure out the issue easier. Here are the possible examples:
"Depth mode X doesn’t support Y fps" or
"The depth mode doesn’t support the FPS mode you set. "

early production devices had the wrong calibration provisioned on the device.

The depth engine plugin is being loaded at runtime. For debugging, it would be nice to know what version of the plugin was loaded

In 1.1.x we want to add event based API’s to notify the user when devices are connected or disconnected from the PC. The main events to cover are USB device attach and USB device detach. Other events that might be interesting in future iterations might be: the loss of just the color camera, loss of the depth camera, device reset, high water mark on capture queue, to name a few. It would be nice have a single event API that can grow in the future if we want it to.

There is also a desire for the API to work without requiring a handle be opened to any device. Meaning k4a_device_t should not be needed to call this API.

Option 1

This options follows the SDK’s existing style of not having any callback functions. Just like we have k4a_device_get_capture/imu_sample we will add:

K4A_EXPORT k4a_wait_result_t k4a_get_next_event([out][optional] uint32_t *device_index,
                                                [out][optional] char *serial_number,
                                                [out][optional] size_t serial_number_size,
                                                [out] k4a_event_t *event,
                                                [in] int32_t timeout_in_ms);

Device_index, serial_number, and serial_number_size are optional parameters, if the user doesn’t care about them, then they can pass in NULL for those values. The user will have to call into this blocking API to be notified when a device is attached or detached from the PC. I see two calling patterns with this:

1. The user may call this API from a dedicated thread and specify the timeout as K4A_WAIT_INFINITE.
2. The user may poll for updates with a timeout of 0 and do it on the same thread that is waiting for captures.
   The word ‘next’ is added to the name to imply that we are not keeping history here. So when you call it the first time you can expect it only return events from that moment forward. There needs to be bit of state caching due to the fact an event can happen after k4a_get_next_event has returned and user is currently processing an event. The storing of this state information should be minimal.
   In the case of using K4A_WAIT_INFINITE a 2nd API is needed to cancel the blocking call when the thread needs to tear down.

K4A_EXPORT void k4a_cancel_get_event(void);

This then drives the k4a_event_t enum to have 3 values:

typedef enum
{
    k4a_event_device_attached = 0, /**< Azure Kinect device has been attached to the PC */
    k4a_event_device_detached,     /**< Azure Kinect device has been detached from the PC */
    k4a_event_wait_canceled,       /**< k4a_get_event has been cancelled by the user */
} k4a_event_t;

When k4a_cancel_get_event() is called, it will unblock k4a_get_next_event() and return k4a_event_wait_canceled. If the call was blocking then its returned immediately. If the k4a_get_next_event was not actively blocking, then the next call to the API will return k4a_event_wait_canceled.
Drawbacks:

• Some kind of state is needed to handle calling into k4a_get_next_event / k4a_cancel_get_event
• Without a context we will have to say k4a_get_next_event only supports 1 caller per process.
• k4a_cancel_get_event() might result in some unexpected behavior if the user calls it, but k4a_get_next_event is not called. The cancel API has to work in two cases; 1) when the API is actively blocking, and 2) when the API is used in a polling loop and will be called in the near future. The user might call the cancel API, stop doing work for their session, and then start a new session 1 hour later. The first call to k4a_get_next_event might return the xxx_wait_canceled event if it were still pending from the first session. We will have to cover this with documentation, or we can add a flag to the k4a_get_next_event to indicate restart of listening or a flush flag to clear out the existing state.

Option 2

This option is to use a traditional callback based API. We will need 3 API’s:

1. The k4a API that registers the callback function.
2. The k4a API that unregisters the callback function.
3. The users callback function the events will be delivered to.
   With this implementation events are delivered asynchronously to the user. Multithreaded protection here is the most difficult aspect to get right. And this is usually where bug come from and are the hardest to fix. We also have to decide when the unregistration API can be called.

• The unregistration API can be called from a worker thread.
• Can it be called it from the callback thread?
• Does the unregister API block until the callback is complete?

Summary

The data the user needs is the same in both scenario’s; SN, index, and attach/detach state. The serial number is a string and a copy of it has to be shared with the user in the callback case, so owner ship of that can get tricky. In option 1 the serial number is just copied to users buffer, so ownership of that buffer is never in question. Option 1 is also consistent with the existing API’s. And the option 1 behavior quirk can be addressed with documentation, such as saying “k4a_get_next_event() will return success and an event of xxx_wait_canceled exactly 1 time during or after a call from k4a_cancel_get_next_event().”

Example output with power cable disconnected

C:\Program Files\Azure Kinect SDK\tools\amd64\release>AzureKinectFirmwareTool.exe -l
== Azure Kinect DK Firmware Tool ==
Found 1 connected devices:
[2019-03-02 08:09:58.645] [error] [t=23232] [K4A] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (306): find_libusb_device(). Unable to open LIBUSB at index 0
[2019-03-02 08:09:58.647] [error] [t=23232] [K4A] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth_mcu\depth_mcu.c (68): usb_cmd_create(USB_DEVICE_DEPTH_PROCESSOR, device_index, NULL, &depthmcu->usb_cmd) returned failure in depthmcu_create()
ERROR: 0: Failed to open device
[2019-03-02 08:09:58.647] [error] [t=23232] [K4A] Invalid depthmcu_t 0000000000000000
[2019-03-02 08:09:58.647] [error] [t=23232] [K4A] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth_mcu\depth_mcu.c (101): Invalid argument to depthmcu_get_serialnum(). depthmcu_handle (0000000000000000) is not a valid handle of type depthmcu_t
ERROR: 0: Failed to get serial number length
[2019-03-02 08:09:58.648] [error] [t=23232] [K4A] Invalid depthmcu_t 0000000000000000
[2019-03-02 08:09:58.648] [error] [t=23232] [K4A] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth_mcu\depth_mcu.c (87): Invalid argument to depthmcu_destroy(). depthmcu_handle (0000000000000000) is not a valid handle of type depthmcu_t

[2019-01-28 10:03:05.968] [error] [t=30132] [K4A] ....\src\dynlib\dynlib_windows.c (28): dynlib->handle != NULL returned failure in dynlib_create
[2019-01-28 10:03:05.969] [error] [t=30132] [K4A] ....\src\dynlib\dynlib_windows.c (32): dynlib_create Failed to load DLL depthengine with error code: 126
[2019-01-28 10:03:05.969] [error] [t=30132] [K4A] ....\src\deloader\deloader.c (90): destub_depth_engine_create_and_initialize Failed to load depth engine plugin
[2019-01-28 10:03:05.969] [error] [t=30132] [K4A] ....\src\dewrapper\dewrapper.c (146): depth_engine_start_helper Depth engine create and initialize failed with error code: 108.
[2019-01-28 10:03:05.969] [error] [t=30132] [K4A] ....\src\dewrapper\dewrapper.c (148): deresult == K4A_DEPTH_ENGINE_RESULT_SUCCEEDED returned failure in depth_engine_start_helper
[2019-01-28 10:03:05.969] [error] [t=30132] [K4A] ....\src\dewrapper\dewrapper.c (182): depth_engine_start_helper(dewrapper, dewrapper->fps, dewrapper->depth_mode, &depth_engine_max_compute_time_ms, &depth_engine_output_buffer_size) returned failure in depth_engine_thread

Currently the firmware module requires a depthmcu_t handle. Since the consumer of the module shouldn't be trying to do anything else while performing firmware operations, the firmware module should completely encapsulate the depthmcu module. This should help clean up the update tool and test code.

Getting started here:
https://docs.microsoft.com/en-us/azure/devops/artifacts/symbols/setting-up-github-sourcelinking?view=azure-devops

CPP modules like logger and color call 'new' to allocate handles, as well as call into other CPP/COM code that may throw exceptions. This code should attempt to catch these exceptions, cleanup, and return an error to the caller of the given C function

Due to HW segmentation, reset commands can be issued to either the Depth ISP or the Color ISP. The result however is that both are reset. In certain conditions the Depth device is not responsive and a reset to the color device can clear it. Currently the tool only supports a reset sent to the depth ISP

Microsoft open source files must have the appropriate copyright notice and license text added to the header of each file. Currently these copyright notices are missing or inconsistent.

https://docs.opensource.microsoft.com/releasing/copyright-headers.html

Depth Engine initialization and discovery of the fact the binary is missing happens on the worker thread and has no direct way to return an error code.

call
k4a_device_open
k4a_device_get_installed_count
k4a_device_close

you should not get the following results.
libusb: warning [libusb_exit] some libusb_devices were leaked

Enable applications like k4aviewer to enable logging to single file with SDK.

We notice our SensorApp.exe/CameraSample.exe are gradually using more memory when capturing video. For example, after 1 hour, 32MB memory was increased. After debugging, we suspect the increased memory is related to the Azure Kinect SDK.
We observed similar symptoms in K4aViewer.exe (a camera tool bundled with the SDK). So, we need the SDK team to confirm if this is a memory leak issue or by design.

This is the snapshot taken after CameraSample.exe has stopped capturing video (after 1-hour capturing). You can see there were ~90K allocations not freed.
• Before the capturing was stopped, this number kept increasing (never dropped).
• After the capturing was stopped and waiting for half hour, the number still had no change. I’m not sure if the SDK has the garbage collection design to recycle the memory or not.
image

And these memory blocks are all allocated by k4a.dll!depth_engine_thread.
image

This is the snapshot taken after K4aViewer.exe has stopped capturing video and closed the device.
image

And these are also all allocated by k4a.dll!depth_engine_thread.
image

Thanks,
Scott

We currently have a CPP based record tool that is in the examples folder. It has gotten fairly complex, so we should consider moving the current example into the tools folder. We should also write a C-Based sample. C code is specifically needed because the header file is C based and we also need to test to confirm the C header stays compatible for C projects (right now all users of record are CPP). Over time we will make CPP, C# version of the simple sample for users of the SDK.

We should skip interpolation for large depth discontinuity to clean the artifacts (line of dots) at those regions, but we should also be disruptive on the slanted surface

k4aviewer has been getting complicated over the time and we should have simplified viewer that just open Depth/RGB camera stream and can work as sample for devs to build their own viewer.

Capability to set manual exposure as parameter when starting recording.

In k4aviewer\platform\windows\filesystem17.cpp there is a call to PathStripPath(). The implementation is in Shlwapi.lib, which does not seem to be included in the link statement. Perhaps adding it here in the the CMakeLists file:
list(APPEND EXTERNAL_LIBRARIES
setupapi.lib
pathcch.lib
Shlwapi.lib +++add this
)

Support for ROS with Azure Kinect DK

$vsWhereOutput = (& $vswhere -format xml)

Might want to consider having the -prerelease flag to add searches for non-release versions of VS. Otherwise the script fails when a user only has prerelease versions installed.

Support for getting connected and disconnected event to avoid polling the state.

K4A_DECLARE_CONTEXT uses malloc and free for allocating/deallocating wrapper context. If there is any cpp objects inside the context, it doesn't call their constructor and destructor properly.

k4a_record_context_t and k4a_playback_context_t are two contexts that have this memory leak issue now.

tests/Utilities/ConnEx/ConnEx.cpp connection_exerciser::get_current_reading() needs to be updated to properly calculate the measured current.

From Matt:

The original author noticed that if the number was negative, it was really 1000 + the value. So if the value was -2.98A, then the number was 1298. 2,98A would be 298. If a number was 1000 or greater, they would set the current to 1000 - current. This is correct, As 1000 - 1298 would give up -298.

The only small bug here is if the connection exerciser returns "negative zero" (1000). In that case we'd report 10 Amps, when we should report 0 amps. We just need to change if (raw_value > 1000) to if (raw_value >= 1000). I made the suggestion in the code.

VS 2019, latest NASM, build in VS doesn't work. Command line to nasm is buggered with cl.exe parameters /MTd /ZH:SHA_256 /D_CRT_SECURE_NO_WARNINGS and with an incorrectly parsed output directory (extern\libjpeg-turbo\src\simd\CMakeFiles\simd.dir\x86_64\jsimdcpu.asm.obj) which confuse NASM. I can build libjpeg-turbo separately just fine. I am pretty stupid on cmake so trying to get myself up to speed to provide better info. But if this is a known issue please close this one and I'll move on to other things.

Support for playing back recorded IMU samples