OpenCV  3.4.20
Open Source Computer Vision
Using Kinect and other OpenNI compatible depth sensors

Prev Tutorial: Creating a video with OpenCV

Next Tutorial: Using Creative Senz3D and other Intel Perceptual Computing SDK compatible depth sensors

Depth sensors compatible with OpenNI (Kinect, XtionPRO, ...) are supported through VideoCapture class. Depth map, BGR image and some other formats of output can be retrieved by using familiar interface of VideoCapture.

In order to use depth sensor with OpenCV you should do the following preliminary steps:

  1. Install OpenNI library (from here http://www.openni.org/downloadfiles) and PrimeSensor Module for OpenNI (from here https://github.com/avin2/SensorKinect). The installation should be done to default folders listed in the instructions of these products, e.g.:
    OpenNI:
    Linux & MacOSX:
    Libs into: /usr/lib
    Includes into: /usr/include/ni
    Windows:
    Libs into: c:/Program Files/OpenNI/Lib
    Includes into: c:/Program Files/OpenNI/Include
    PrimeSensor Module:
    Linux & MacOSX:
    Bins into: /usr/bin
    Windows:
    Bins into: c:/Program Files/Prime Sense/Sensor/Bin
    If one or both products were installed to the other folders, the user should change corresponding CMake variables OPENNI_LIB_DIR, OPENNI_INCLUDE_DIR or/and OPENNI_PRIME_SENSOR_MODULE_BIN_DIR.
  2. Configure OpenCV with OpenNI support by setting WITH_OPENNI flag in CMake. If OpenNI is found in install folders OpenCV will be built with OpenNI library (see a status OpenNI in CMake log) whereas PrimeSensor Modules can not be found (see a status OpenNI PrimeSensor Modules in CMake log). Without PrimeSensor module OpenCV will be successfully compiled with OpenNI library, but VideoCapture object will not grab data from Kinect sensor.
  3. Build OpenCV.

VideoCapture can retrieve the following data:

  1. data given from depth generator:
    • CAP_OPENNI_DEPTH_MAP - depth values in mm (CV_16UC1)
    • CAP_OPENNI_POINT_CLOUD_MAP - XYZ in meters (CV_32FC3)
    • CAP_OPENNI_DISPARITY_MAP - disparity in pixels (CV_8UC1)
    • CAP_OPENNI_DISPARITY_MAP_32F - disparity in pixels (CV_32FC1)
    • CAP_OPENNI_VALID_DEPTH_MASK - mask of valid pixels (not occluded, not shaded etc.) (CV_8UC1)
  2. data given from BGR image generator:
    • CAP_OPENNI_BGR_IMAGE - color image (CV_8UC3)
    • CAP_OPENNI_GRAY_IMAGE - gray image (CV_8UC1)

In order to get depth map from depth sensor use VideoCapture::operator >>, e. g. :

VideoCapture capture( CAP_OPENNI );
for(;;)
{
Mat depthMap;
capture >> depthMap;
if( waitKey( 30 ) >= 0 )
break;
}

For getting several data maps use VideoCapture::grab and VideoCapture::retrieve, e.g. :

VideoCapture capture(0); // or CAP_OPENNI
for(;;)
{
Mat depthMap;
Mat bgrImage;
capture.grab();
capture.retrieve( depthMap, CAP_OPENNI_DEPTH_MAP );
capture.retrieve( bgrImage, CAP_OPENNI_BGR_IMAGE );
if( waitKey( 30 ) >= 0 )
break;
}

For setting and getting some property of sensor` data generators use VideoCapture::set and VideoCapture::get methods respectively, e.g. :

VideoCapture capture( CAP_OPENNI );
cout << "FPS " << capture.get( CAP_OPENNI_IMAGE_GENERATOR+CAP_PROP_FPS ) << endl;

Since two types of sensor's data generators are supported (image generator and depth generator), there are two flags that should be used to set/get property of the needed generator:

Some depth sensors (for example XtionPRO) do not have image generator. In order to check it you can get CAP_OPENNI_IMAGE_GENERATOR_PRESENT property.

bool isImageGeneratorPresent = capture.get( CAP_PROP_OPENNI_IMAGE_GENERATOR_PRESENT ) != 0; // or == 1

Flags specifying the needed generator type must be used in combination with particular generator property. The following properties of cameras available through OpenNI interfaces are supported:

For more information please refer to the example of usage videocapture_openni.cpp in opencv/samples/cpp folder.