PRiMEStereoMatch

Theoretical Background

A heterogeneous and fully parallel stereo matching algorithm for depth estimation. Stereo Matching is based on the disparity estimation algorithm, an algorithm designed to calculate 3D depth information about a scene from a pair of 2D images captured by a stereoscopic camera. The algorithm contains the following stages:

• Cost Volume Construction - weighted absolute difference of colours and gradients function.
• Cost Volume Filtering - Adaptive Support Weight (ADSW) Guided Image Filter (GIF) function.
• Disparity Selection - Winner-Takes-All (WTA) minimum cost selection.
• Post Processing - left-right occlusion check, invalid pixel replacement and weight-median filtering.

Implementation Details

• All stages of the algorithm have been developed in both C++ and OpenCL.
  • C++ parallelism is introduced via the POSIX threads (pthreads) library. Disparity level parallelism is supported, enabling up to 64 concurrent threads.
  • OpenCL parallelism is inherent through the concurrent execution of kernels on an OpenCL-compatible device. The optimum level of parallelism will be bounded by the platform & devices.
• Support for live video disparity estimation using the OpenCV VideoCapture interface as well as static image computation.
• Embedded support for experimentation with the OpenCV standard Semi-Global Block Matching (SGBM) algorithm.

Installation

Prerequisites

• Hardware:
  • Development Platform - optionally but ideally including OpenCL compatible devices
• Software:
  • OpenCV 2.4 or later - Installation in Linux instructions
  • pthread and realtime libraries (lpthread, lrt)
  • [optional] OpenCL Library (-lOpenCL)

Compilation

• Download project folder to your platform.
• Enter the base directory using cd DE_APP.
• Natively compile the project using make -jN.
  • Set N to the number of simultaneous threads supported on your compilation platform, e.g. 8.

Deployment

• Run the application using ./bin/Release/DE_APP <program arguments>

• The following program arguments must be specified:

  • Matching Algorithm type:
    • STEREO_GIF - Guided Image Filter
    • STEREO_SGBM - Semi Global Block Matching
  • Media type:
    • VIDEO
    • IMAGE

• When specifying the VIDEO media type, the following optional arguments can be included:

  • RECALIBRATE - recalculate the intrinsic and extrinsic parameters of the stereo camera. Previously captured chessboard images must be supplied if the RECAPTURE flag is not also set.
  • RECAPTURE - record chessboard image pairs in preparation for calibration. A chessboard image must be presented in front of the stereo camera and in full view of both cameras. Press the R key to capture a frame. The last frame captured is shown beneath the video stream.

• For example, to run with the guided image filter algorithm using a stereo camera, specify:

  • ./bin/Release/DE_APP STEREO_GIF VIDEO

• To run with calibration and capture beforehand, specify:

  • ./bin/Release/DE_APP STEREO_GIF VIDEO RECALIBRATE RECAPTURE

• Image disparity estimation is achieved using for example:

  • ./bin/Release/DE_APP STEREO_GIF IMAGE left_img.png right_img.png

• The first time the application is deployed using a stereo camera, the RECALIBRATE and RECAPTURE flags must be set in order to capture chessboard image to calculate the intrinsic and extrinsic parameters.

• This process only needs to be repeated if the relative orientations of the left and right cameras are changed or a different resolution is specified.

• Once the intrinsic and extrinsic parameters have been calucalted and saved to .yml files, the application can be re-run with the same camera without needing to recalibrate as teh parameters will be loaded from these files. The files can be found in the data directory.

Additional Resources

• Configuring OpenCL on the ODROID-XU3

Directory Structure

DE_APP		- Project top level directory

folders:
	assets			- OpenCL kernel files
	bin				- binary executable files
	common			- OpenCL common utility/helper functions (C) ARM Ltd
	data			- program data including input images, stereo camera parameters, calibration images, etc
	include			- Project header files (h/hpp)
	src				- Project source files (c/cpp)
	
files:
	cbp2make.linux-x86_64 	- codeblocks to makefile synthesis tool (for x86_64 PC)
	cbp2make_usage.txt		- cbp2make tool manual
	DE_APP.cbp				- Code::Blocks project file
	DE_APP.depend			- Code::Blocks settings file
	DE_APP.layout			- Code::Blocks settings file
	main.cpp				- main C++ file
	Makefile				- project Makefile

References

Code

CrossScaleStereo - The basis for some C++ DE functions (GNU Public License)

Literature

[Hosni2011CVPR]: C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR, 2011

[Hosni2011ICME]: A. Hosni, M. Bleyer, C. Rhemann, M. Gelautz and C. Rother, Real-time local stereo matching using guided image filtering, in Multimedia and Expo (ICME), 2011 IEEE International Conference on, Barcelona, 2011.

[Ttofis2014]: C. Ttofis and T. Theocharides, High-quality real-time hardware stereo matching based on guided image filtering, in Design, Automation and Test in Europe Conference and Exhibition (DATE), Dresden, 2014.

[He2012]: K. He, J. Sun and X. Tang, Guided Image Filtering, Pattern Analysis and Machine Intelligence, IEEE Transactions on, pp. 1397-1409, 02 October 2012.

License

This software is released under the BSD 3 Clause License. See LICENSE.txt for details.