RGB-D and Volumetric Reconstruction#

Detailed Description#

Classes#

Name	Description
`class` `cv::Odometry`	View details
`class` `cv::OdometryFrame`	An object that keeps per-frame data for Odometry algorithms from user-provided images to algorithm-specific precalculated data. When not empty, it contains a depth image, a mask of valid pixels and a set of pyramids generated from that data. A BGR/Gray image and normals are optional. OdometryFrame is made to be used together with Odometry class to reuse precalculated data between Rt data calculations. A correct way to do that is to call Odometry::prepareFrames() on prev and next frames and then pass them to Odometry::compute() method. View details
`class` `cv::OdometrySettings`	View details
`class` `cv::RgbdNormals`	View details
`class` `cv::Volume`	View details
`class` `cv::VolumeSettings`	View details

Enumerations#

enum struct cv::OdometryAlgoType {
    cv::OdometryAlgoType::COMMON = 0,
    cv::OdometryAlgoType::FAST = 1
}

View details

enum cv::OdometryFramePyramidType {
    cv::PYR_IMAGE = 0,
    cv::PYR_DEPTH = 1,
    cv::PYR_MASK = 2,
    cv::PYR_CLOUD = 3,
    cv::PYR_DIX = 4,
    cv::PYR_DIY = 5,
    cv::PYR_TEXMASK = 6,
    cv::PYR_NORM = 7,
    cv::PYR_NORMMASK = 8,
    cv::N_PYRAMIDS
}

View details

enum struct cv::OdometryType {
    cv::OdometryType::DEPTH = 0,
    cv::OdometryType::RGB = 1,
    cv::OdometryType::RGB_DEPTH = 2
}

View details

enum cv::RgbdPlaneMethod {
    cv::RGBD_PLANE_METHOD_DEFAULT
}

View details

enum struct cv::VolumeType {
    cv::VolumeType::TSDF = 0,
    cv::VolumeType::HashTSDF = 1,
    cv::VolumeType::ColorTSDF = 2
}

View details

Enumeration Type Documentation#

OdometryAlgoType#

enum class cv::OdometryAlgoType

#include <opencv2/ptcloud/odometry.hpp>

These constants are used to set the speed and accuracy of odometry

Enumerator:

COMMON
Python: cv.OdometryAlgoType_COMMON

FAST
Python: cv.OdometryAlgoType_FAST

OdometryFramePyramidType#

enum cv::OdometryFramePyramidType

#include <opencv2/ptcloud/odometry_frame.hpp>

Indicates what pyramid is to access using getPyramidAt() method:

Enumerator:

`PYR_IMAGE` Python: `cv.PYR_IMAGE`	The pyramid of grayscale images.
`PYR_DEPTH` Python: `cv.PYR_DEPTH`	The pyramid of depth images.
`PYR_MASK` Python: `cv.PYR_MASK`	The pyramid of masks.
`PYR_CLOUD` Python: `cv.PYR_CLOUD`	The pyramid of point clouds, produced from the pyramid of depths.
`PYR_DIX` Python: `cv.PYR_DIX`	The pyramid of dI/dx derivative images.
`PYR_DIY` Python: `cv.PYR_DIY`	The pyramid of dI/dy derivative images.
`PYR_TEXMASK` Python: `cv.PYR_TEXMASK`	The pyramid of “textured” masks (i.e. additional masks for normals or grayscale images)
`PYR_NORM` Python: `cv.PYR_NORM`	The pyramid of normals.
`PYR_NORMMASK` Python: `cv.PYR_NORMMASK`	The pyramid of normals masks.
`N_PYRAMIDS` Python: `cv.N_PYRAMIDS`

OdometryType#

enum class cv::OdometryType

#include <opencv2/ptcloud/odometry.hpp>

These constants are used to set a type of data which odometry will use

Enumerator:

DEPTH
Python: cv.OdometryType_DEPTH

RGB
Python: cv.OdometryType_RGB

RGB_DEPTH
Python: cv.OdometryType_RGB_DEPTH

RgbdPlaneMethod#

enum cv::RgbdPlaneMethod

#include <opencv2/ptcloud/depth.hpp>

Enumerator:

RGBD_PLANE_METHOD_DEFAULT
Python: cv.RGBD_PLANE_METHOD_DEFAULT

VolumeType#

enum class cv::VolumeType

#include <opencv2/ptcloud/volume_settings.hpp>

Enumerator:

TSDF
Python: cv.VolumeType_TSDF

HashTSDF
Python: cv.VolumeType_HashTSDF

ColorTSDF
Python: cv.VolumeType_ColorTSDF

Function Documentation#

depthTo3d()#

void cv::depthTo3d(
InputArray depth,
InputArray K,
OutputArray points3d,
InputArray mask = noArray() )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.depthTo3d(depth, K[, points3d[, mask]]) -> points3d

Converts a depth image to 3d points. If the mask is empty then the resulting array has the same dimensions as depth, otherwise it is 1d vector containing mask-enabled values only. The coordinate system is x pointing left, y down and z away from the camera

Parameters

depth — the depth image (if given as short int CV_U, it is assumed to be the depth in millimeters (as done with the Microsoft Kinect), otherwise, if given as CV_32F or CV_64F, it is assumed in meters)
K — The calibration matrix
points3d — the resulting 3d points (point is represented by 4 channels value [x, y, z, 0]). They are of the same depth as depth if it is CV_32F or CV_64F, and the depth of K if depth is of depth CV_16U or CV_16S
mask — the mask of the points to consider (can be empty)

depthTo3dSparse()#

void cv::depthTo3dSparse(
InputArray depth,
InputArray in_K,
InputArray in_points,
OutputArray points3d )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.depthTo3dSparse(depth, in_K, in_points[, points3d]) -> points3d

Parameters

depth — the depth image
in_K
in_points — the list of xy coordinates
points3d — the resulting 3d points (point is represented by 4 chanels value [x, y, z, 0])

findPlanes()#

void cv::findPlanes(
InputArray points3d,
InputArray normals,
OutputArray mask,
OutputArray plane_coefficients,
int block_size = 40,
int min_size = 40 *40,
double threshold = 0.01,
double sensor_error_a = 0,
double sensor_error_b = 0,
double sensor_error_c = 0,
RgbdPlaneMethod method = RGBD_PLANE_METHOD_DEFAULT )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.findPlanes(points3d, normals[, mask[, plane_coefficients[, block_size[, min_size[, threshold[, sensor_error_a[, sensor_error_b[, sensor_error_c[, method]]]]]]]]]) -> mask, plane_coefficients

Find the planes in a depth image

Parameters

points3d — the 3d points organized like the depth image: rows x cols with 3 channels
normals — the normals for every point in the depth image; optional, can be empty
mask — An image where each pixel is labeled with the plane it belongs to and 255 if it does not belong to any plane
plane_coefficients — the coefficients of the corresponding planes (a,b,c,d) such that ax+by+cz+d=0, norm(a,b,c)=1 and c < 0 (so that the normal points towards the camera)
block_size — The size of the blocks to look at for a stable MSE
min_size — The minimum size of a cluster to be considered a plane
threshold — The maximum distance of a point from a plane to belong to it (in meters)
sensor_error_a — coefficient of the sensor error. 0 by default, use 0.0075 for a Kinect
sensor_error_b — coefficient of the sensor error. 0 by default
sensor_error_c — coefficient of the sensor error. 0 by default
method — The method to use to compute the planes.

registerDepth()#

void cv::registerDepth(
InputArray unregisteredCameraMatrix,
InputArray registeredCameraMatrix,
InputArray registeredDistCoeffs,
InputArray Rt,
InputArray unregisteredDepth,
const Size & outputImagePlaneSize,
OutputArray registeredDepth,
bool depthDilation = false )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.registerDepth(unregisteredCameraMatrix, registeredCameraMatrix, registeredDistCoeffs, Rt, unregisteredDepth, outputImagePlaneSize[, registeredDepth[, depthDilation]]) -> registeredDepth

Registers depth data to an external camera Registration is performed by creating a depth cloud, transforming the cloud by the rigid body transformation between the cameras, and then projecting the transformed points into the RGB camera.

uv_rgb = K_rgb * [R | t] * z * inv(K_ir) * uv_ir

Currently does not check for negative depth values.

Parameters

unregisteredCameraMatrix — the camera matrix of the depth camera
registeredCameraMatrix — the camera matrix of the external camera
registeredDistCoeffs — the distortion coefficients of the external camera
Rt — the rigid body transform between the cameras. Transforms points from depth camera frame to external camera frame.
unregisteredDepth — the input depth data
outputImagePlaneSize — the image plane dimensions of the external camera (width, height)
registeredDepth — the result of transforming the depth into the external camera
depthDilation — whether or not the depth is dilated to avoid holes and occlusion errors (optional)

rescaleDepth()#

void cv::rescaleDepth(
InputArray in,
int type,
OutputArray out,
double depth_factor = 1000.0 )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.rescaleDepth(in_, type[, out[, depth_factor]]) -> out

If the input image is of type CV_16UC1 (like the Kinect one), the image is converted to floats, divided by depth_factor to get a depth in meters, and the values 0 are converted to std::numeric_limits::quiet_NaN() Otherwise, the image is simply converted to floats

Parameters

in — the depth image (if given as short int CV_U, it is assumed to be the depth in millimeters (as done with the Microsoft Kinect), it is assumed in meters)
type — the desired output depth (CV_32F or CV_64F)
out — The rescaled float depth image
depth_factor — (optional) factor by which depth is converted to distance (by default = 1000.0 for Kinect sensor)

warpFrame()#

void cv::warpFrame(
InputArray depth,
InputArray image,
InputArray mask,
InputArray Rt,
InputArray cameraMatrix,
OutputArray warpedDepth = noArray(),
OutputArray warpedImage = noArray(),
OutputArray warpedMask = noArray() )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.warpFrame(depth, image, mask, Rt, cameraMatrix[, warpedDepth[, warpedImage[, warpedMask]]]) -> warpedDepth, warpedImage, warpedMask

Warps depth or RGB-D image by reprojecting it in 3d, applying Rt transformation and then projecting it back onto the image plane. This function can be used to visualize the results of the Odometry algorithm.

Parameters

depth — Depth data, should be 1-channel CV_16U, CV_16S, CV_32F or CV_64F
image — RGB image (optional), should be 1-, 3- or 4-channel CV_8U
mask — Mask of used pixels (optional), should be CV_8UC1, CV_8SC1 or CV_BoolC1
Rt — Rotation+translation matrix (3x4 or 4x4) to be applied to depth points
cameraMatrix — Camera intrinsics matrix (3x3)
warpedDepth — The warped depth data (optional)
warpedImage — The warped RGB image (optional)
warpedMask — The mask of valid pixels in warped image (optional)