This toolbox contains some useful code in calibrating Kinect, which also replicates my review work in Kinect calibration.

For anyone intersted in this work, you can find details from my review paper "A review and quantitative comparison of methods for kinect calibration" as well as the corresponding technical report.

You will get three of the following directories:

1. capture--including libfreenect library and its Matlab wrapper.
2. record--a tool that helps to record data stream from Kinect in Matlab.
3. demo--including the core of calibration and sample dataset.

Run main.m to start, program is quite easy to understand as long as you follow the command prompts. There are two ways for you to calibrate your kinect:

1. offline--calibrate using your pre-obtained data.
2. online--calibrate using data captured in real-time.

I am providing dataset '../smallset/' for reference (however, you must capture data with your own Kinect) which examfies file format of our offline calibration: (instructions cited from document attached with the original code)

1. Put all files in the same directory, whose path needs to be specified in the command prompt.
2. For a successful calibration 4 types of images must be present: frontal plane, plane rotated around the X axis, plane rotated around the Y axis, and full image planar surface for distortion correction.
3. If you want to calibrate the disparity distortion model, you should also take some images of a bigger plane (e.g. a wall) that fills the entire image.
4. For a proper calibration, the algorithm requires around 30 images from different angles.
5. All images taken by Kinect color camera is formatted as "0000-c1.jpg".
6. All images taken by Kinect depth camera is formatted as "0000-d.pgm".
7. Images corresponding to the same plane pose should have the same prefix number.
8. A given plane pose may have only one color image, only the depth image, or any combination.
9. The calibration uses the information from those images that are present.

It should be noted that during offline calibration, intermediate data will be saved in directory where all images exist, in case that any accident happens. If you want to redo any intermediate step, just delete corresponding .mat file and its previous .mat file, then algorithm should go through correctly.

Note on the sequence of intermediate data storage: dataset.mat->rgb_corners.mat->depth_plane_mask.mat->calib0.mat->final_calib.mat

There are several important default parameters that should be clarified in online calibration:

• sqr_def: square size for checkerboard (m).
• win_def: window size of automatic corner finder (pixel).
• cnt_x_def: inner corner count of checkerboard in X direction.
• cnt_y_def: inner corner count of checkerboard in Y direction.
• pic_num: number of images that must be taken for each of the three views including frontal, around x and around y, at various distances.
• wall_num: number of images that must be taken in front of a wall, at various distances.

Decrease pic_num and wall_num for your ease of use, however, as for a proper calibration, 30 images are needed while 50 images (approximately) will help achieve a satisfactory result. That is to say, 8 images for every pose plus 5 of wall images (default) meet the basic requrement, while 10 for every pose plus 8 wall images are definitely better (approximately).

Also, please see the following illustration on how to count inner corner count of the checkerboard:

Be aware that during online calibration, all intermediate data including images captured, rgb corners detected and depth corners selected, etc., will be deleted for concerns of conflicting data. This may not be a good design, therefore, I will add functions to save intermediate data in the future.

In the original code, the author only provided offline calibration, which makes corner detection a headache problem due to low-quality captured images and far placement of checkerboard. Offline calibration of this demo remains the same with original one, i.e., for each of the images in specified directory, the corner finder will try to detect (cnt_x_def+1) × (cnt_y_def+1) number of squares with side size sqr_def (by default).

If the detection failed, however, you will be asked to manually label the four corners of checkerboard. But remember, most likely the corner finder will still not detect all the points successfully, given the image quality is low and/or checkerboard is placed far away (more than 5m, empirically) from the camera. To avoid this, I suggest:

• Using online calibration when possible.
• If corner detection failed in offline calibration, delete that specific picture (and corresponding depth) and re-take one with the same position, adjust illumination condition and/or distance. Do not forget to name the images retaken (both rgb and depth) with same index in prescribed format.
• If corner detection failed in online calibration, simply adjust illumination condition and/or distance and restart following the prompt.
• For your ease of use, try to make the distances smaller than 5m.

To have a better understanding of how you should capture your data, please do it according to the following illustration:

If you have difficulty in viewing pgm file with Windows built-in Windows Photo Viewer, please download Xnview.

Run mex_test (make sure your kinect is well connected) to see whether you can use libfreenect smoothly, you will see a window pop-up like this

This program will exit once you click 'x' on the figure.

Run record (make sure your kinect is well connected) to start, the program will begin recording data stream in '../output/pic/'. Once you are turning on make_avi flag in record.m, the stream will be recorded as video with specified frame rate, in directory '../output/video/'. Besides, you can also adjust starting index of data set with variable set_ind, and one for file name with variable cnt. The format of file name is for eg. "set0_0000.jpg" and "set0_0000.pgm", which stores rgb and raw depth (also called disparity) respectively.

Note that the main.m is able to capture data by its own and therefore does not rely on this tool.

This program will exit once you right-click on the figure.

Be aware that in order to capture Kinect data stream instantly from Matlab, you must have the prescribed configuration on your system.

The code is dependent on 3rd-lib libfreenect to connect Kinect, I have compiled its Matlab wrapper (named kinect_mex.mex) for your convinient use in Matlab. Currently, it supports all three popular platforms of Matlab with 64-bit environment:

You may find that you are not able to run kinect.mex when capturing data, which means, you have to compile libfreenect yourself.

However, if you are using Matlab/Windows of 32-bit, you may encounter painful experiences in resolving dependencies to compile libfreenect on your own system. For your reference, I am attaching the environment and all dependencies used when I was compiling libfreenect:

• Platforms: Windows 7 (64-bit) + Matlab (64-bit) + Visual Studio 2010 (32-bit--only) + Windows SDK 7.1
• Libraries: libfreenect-0.1.2 + glut-3.7.6-bin + libusb-1.0.19 + libusb-win32-bin-1.2.4.0 + pthreads-w32-2-8-0-release + unistd

Unfortunately the newest version (v0.5.0--Saturn) of libfreenect that I compiled can not run successfully and after digging for quite a while, I found libfreenect-0.1.2 is most suitable for the wrapper we use. As you may notice, there are two versions of libusb linked, the reason is that libfreenect-0.1.2 call interfaces from both versions.

In addition, always make sure that you are using the same environment of the following:

1. Mex compiler embedded in Matlab
2. C compiler (usually located in Visual Stuio IDE), using mex -setup to find C compiler for Matlab
3. All other 3-rd part dependencies

to compile for mex in Matlab. For e.g., if you are running 32-bit Matlab, you can use default C compiler located in Visual Studio. However, for 64-bit Matlab, please download WinSDK 7.1 in order to let Matlab utilize 64-bit C compiler and meanwhile, re-compile all other 3-rd dependencies using 64-bit compiler and link all together for kinect.mex. You may encounter "undefined external symbols" when you are trying to compile for mex files, if any of your dependencies' bit environment is different from your compiler's, or the two compilers' differ from each other.

See compile_mex.m in capture directory for details.

In demo/toolbox directory, find file show_calib_result.m, with which you can utilize to visually evaluate your calibration result, as shown below:

Note that in show_calib_result.m, change calib_filepath to the path where final_calib.mat locates, you can either test on single image file (file that must be taken by this specific Kinect you have calibrated), or on the data stream, simply by swtiching the variable testing_realdata. Besides, turn off add_noncalib_comparison to skip comparison with non-calibrated result.

Also, be aware that the program utilized a multi-scale bilateral filter to fill the 'holes' from depth image, which resulted from surfaces of high/low reflectivity. Code of filter can be downloaded from NYU Kinect toolbox V2.

This program will exit once you click 'x' on the figure.

For anyone who wants to use calibration result to reproject depth points from depth space to rgb space, please make use of the function compute_rgb_depthmap. Also, refer to show_calib_result.m on how to use this function correctly.

If you find this toolbox useful, please cite our paper at:

@inproceedings{xiang2015review,
    title={A review and quantitative comparison of methods for kinect calibration}, 
    author={Xiang, Wei and Conly, Christopher and McMurrough, Christopher D and Athitsos, Vassilis}, 
    booktitle={Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction}, 
    pages={3}, 
    year={2015}, 
    organization={ACM} 
}

This work originates from the University of Oulu, by: Herrera C., D., Kannala J., Heikkila, J., "Joint depth and color camera calibration with distortion correction", TPAMI, 2012. Please see link of the original code: http://www.ee.oulu.fi/~dherrera/kinect/, as well as corresponding document .

The libfreenect Matlab wrapper is provided by Alexander Berg.

For details on how to use Kinect, please take a look at: Kramer, Jeff, Nicolas Burrus, and Florian Echtler. Hacking the Kinect. New York, NY, USA:: Apress, 2012.

Curret up-to-date version is v02_11_2015 (named by the last date I modified the files).

Questions, comments and bugs, please email to: [email protected]