Point Cloud Processing#

Topics#

Detailed Description#

Point-cloud and mesh processing: reading/writing point clouds and meshes, triangle rasterization, octree partitioning, and RGB-D / volumetric 3D reconstruction (odometry, TSDF volumes).

Namespaces#

Classes#

Name

Description

class cv::Octree

Octree for 3D vision. View details

class cv::Odometry

View details

class cv::OdometryFrame

An object that keeps per-frame data for Odometry algorithms from user-provided images to algorithm-specific precalculated data. When not empty, it contains a depth image, a mask of valid pixels and a set of pyramids generated from that data. A BGR/Gray image and normals are optional. OdometryFrame is made to be used together with Odometry class to reuse precalculated data between Rt data calculations. A correct way to do that is to call Odometry::prepareFrames() on prev and next frames and then pass them to Odometry::compute() method. View details

class cv::OdometrySettings

View details

class cv::RgbdNormals

View details

struct cv::TriangleRasterizeSettings

Structure to keep settings for rasterization. View details

class cv::Volume

View details

class cv::VolumeSettings

View details

Enumerations#

enum cv::TriangleCullingMode {
    cv::RASTERIZE_CULLING_NONE = 0,
    cv::RASTERIZE_CULLING_CW = 1,
    cv::RASTERIZE_CULLING_CCW = 2
}

Face culling settings: what faces are drawn after face culling. View details

enum cv::TriangleGlCompatibleMode {
    cv::RASTERIZE_COMPAT_DISABLED = 0,
    cv::RASTERIZE_COMPAT_INVDEPTH = 1
}

GL compatibility settings. View details

enum cv::TriangleShadingType {
    cv::RASTERIZE_SHADING_WHITE = 0,
    cv::RASTERIZE_SHADING_FLAT = 1,
    cv::RASTERIZE_SHADING_SHADED = 2
}

Triangle fill settings. View details

enum struct cv::OdometryAlgoType {
    cv::OdometryAlgoType::COMMON = 0,
    cv::OdometryAlgoType::FAST = 1
}

View details

enum cv::OdometryFramePyramidType {
    cv::PYR_IMAGE = 0,
    cv::PYR_DEPTH = 1,
    cv::PYR_MASK = 2,
    cv::PYR_CLOUD = 3,
    cv::PYR_DIX = 4,
    cv::PYR_DIY = 5,
    cv::PYR_TEXMASK = 6,
    cv::PYR_NORM = 7,
    cv::PYR_NORMMASK = 8,
    cv::N_PYRAMIDS
}

View details

enum struct cv::OdometryType {
    cv::OdometryType::DEPTH = 0,
    cv::OdometryType::RGB = 1,
    cv::OdometryType::RGB_DEPTH = 2
}

View details

enum cv::RgbdPlaneMethod {
    cv::RGBD_PLANE_METHOD_DEFAULT
}

View details

enum struct cv::VolumeType {
    cv::VolumeType::TSDF = 0,
    cv::VolumeType::HashTSDF = 1,
    cv::VolumeType::ColorTSDF = 2
}

View details

Enumeration Type Documentation#

TriangleCullingMode#

enum cv::TriangleCullingMode

#include <opencv2/ptcloud.hpp>

Face culling settings: what faces are drawn after face culling.

Enumerator:

RASTERIZE_CULLING_NONE
Python: cv.RASTERIZE_CULLING_NONE

all faces are drawn, no culling is actually performed

RASTERIZE_CULLING_CW
Python: cv.RASTERIZE_CULLING_CW

triangles which vertices are given in clockwork order are drawn

RASTERIZE_CULLING_CCW
Python: cv.RASTERIZE_CULLING_CCW

triangles which vertices are given in counterclockwork order are drawn

TriangleGlCompatibleMode#

enum cv::TriangleGlCompatibleMode

#include <opencv2/ptcloud.hpp>

GL compatibility settings.

Enumerator:

RASTERIZE_COMPAT_DISABLED
Python: cv.RASTERIZE_COMPAT_DISABLED

Color and depth have their natural values and converted to internal formats if needed.

RASTERIZE_COMPAT_INVDEPTH
Python: cv.RASTERIZE_COMPAT_INVDEPTH

Color is natural, Depth is transformed from [-zNear; -zFar] to [0; 1] by the following formula: \( \frac{z_{far} \left(z + z_{near}\right)}{z \left(z_{far} - z_{near}\right)} \)

In this mode the input/output depthBuf is considered to be in this format, therefore it’s faster since there’re no conversions performed

TriangleShadingType#

enum cv::TriangleShadingType

#include <opencv2/ptcloud.hpp>

Triangle fill settings.

Enumerator:

RASTERIZE_SHADING_WHITE
Python: cv.RASTERIZE_SHADING_WHITE

a white color is used for the whole triangle

RASTERIZE_SHADING_FLAT
Python: cv.RASTERIZE_SHADING_FLAT

a color of 1st vertex of each triangle is used

RASTERIZE_SHADING_SHADED
Python: cv.RASTERIZE_SHADING_SHADED

a color is interpolated between 3 vertices with perspective correction

OdometryAlgoType#

enum class cv::OdometryAlgoType

#include <opencv2/ptcloud/odometry.hpp>

These constants are used to set the speed and accuracy of odometry

Enumerator:

COMMON
Python: cv.OdometryAlgoType_COMMON

FAST
Python: cv.OdometryAlgoType_FAST

OdometryFramePyramidType#

enum cv::OdometryFramePyramidType

#include <opencv2/ptcloud/odometry_frame.hpp>

Indicates what pyramid is to access using getPyramidAt() method:

Enumerator:

PYR_IMAGE
Python: cv.PYR_IMAGE

The pyramid of grayscale images.

PYR_DEPTH
Python: cv.PYR_DEPTH

The pyramid of depth images.

PYR_MASK
Python: cv.PYR_MASK

The pyramid of masks.

PYR_CLOUD
Python: cv.PYR_CLOUD

The pyramid of point clouds, produced from the pyramid of depths.

PYR_DIX
Python: cv.PYR_DIX

The pyramid of dI/dx derivative images.

PYR_DIY
Python: cv.PYR_DIY

The pyramid of dI/dy derivative images.

PYR_TEXMASK
Python: cv.PYR_TEXMASK

The pyramid of “textured” masks (i.e. additional masks for normals or grayscale images)

PYR_NORM
Python: cv.PYR_NORM

The pyramid of normals.

PYR_NORMMASK
Python: cv.PYR_NORMMASK

The pyramid of normals masks.

N_PYRAMIDS
Python: cv.N_PYRAMIDS

OdometryType#

enum class cv::OdometryType

#include <opencv2/ptcloud/odometry.hpp>

These constants are used to set a type of data which odometry will use

Enumerator:

DEPTH
Python: cv.OdometryType_DEPTH

RGB
Python: cv.OdometryType_RGB

RGB_DEPTH
Python: cv.OdometryType_RGB_DEPTH

RgbdPlaneMethod#

enum cv::RgbdPlaneMethod

#include <opencv2/ptcloud/depth.hpp>

Enumerator:

RGBD_PLANE_METHOD_DEFAULT
Python: cv.RGBD_PLANE_METHOD_DEFAULT

VolumeType#

enum class cv::VolumeType

#include <opencv2/ptcloud/volume_settings.hpp>

Enumerator:

TSDF
Python: cv.VolumeType_TSDF

HashTSDF
Python: cv.VolumeType_HashTSDF

ColorTSDF
Python: cv.VolumeType_ColorTSDF

Function Documentation#

loadMesh()#

void cv::loadMesh(
const String & filename,
OutputArray vertices,
OutputArrayOfArrays indices,
OutputArray normals = noArray(),
OutputArray colors = noArray(),
OutputArray texCoords = noArray() )

#include <opencv2/ptcloud.hpp>

Python:

cv.loadMesh(filename[, vertices[, indices[, normals[, colors[, texCoords]]]]]) -> vertices, indices, normals, colors, texCoords

Loads a mesh from a file.

The function loads mesh from the specified file and returns it. If the mesh cannot be read, throws an error Vertex attributes (i.e. space and texture coodinates, normals and colors) are returned in same-sized arrays with corresponding elements having the same indices. This means that if a face uses a vertex with a normal or a texture coordinate with different indices (which is typical for OBJ files for example), this vertex will be duplicated for each face it uses.

Currently, the following file formats are supported:

filename

Name of the file

vertices

vertex coordinates, each value contains 3 floats

indices

per-face list of vertices, each value is a vector of ints

normals

per-vertex normals, each value contains 3 floats

colors

per-vertex colors, each value contains 3 floats

texCoords

per-vertex texture coordinates, each value contains 2 or 3 floats

loadPointCloud()#

void cv::loadPointCloud(
const String & filename,
OutputArray vertices,
OutputArray normals = noArray(),
OutputArray rgb = noArray() )

#include <opencv2/ptcloud.hpp>

Python:

cv.loadPointCloud(filename[, vertices[, normals[, rgb]]]) -> vertices, normals, rgb

Loads a point cloud from a file.

The function loads point cloud from the specified file and returns it. If the cloud cannot be read, throws an error. Vertex coordinates, normals and colors are returned as they are saved in the file even if these arrays have different sizes and their elements do not correspond to each other (which is typical for OBJ files for example)

Currently, the following file formats are supported:

Parameters

  • filename — Name of the file

  • vertices — vertex coordinates, each value contains 3 floats

  • normals — per-vertex normals, each value contains 3 floats

  • rgb — per-vertex colors, each value contains 3 floats

saveMesh()#

void cv::saveMesh(
const String & filename,
InputArray vertices,
InputArrayOfArrays indices,
InputArray normals = noArray(),
InputArray colors = noArray(),
InputArray texCoords = noArray() )

#include <opencv2/ptcloud.hpp>

Python:

cv.saveMesh(filename, vertices, indices[, normals[, colors[, texCoords]]])

Saves a mesh to a specified file.

The function saves mesh to the specified file. File format is chosen based on the filename extension.

Parameters

  • filename — Name of the file.

  • vertices — vertex coordinates, each value contains 3 floats

  • indices — per-face list of vertices, each value is a vector of ints

  • normals — per-vertex normals, each value contains 3 floats

  • colors — per-vertex colors, each value contains 3 floats

  • texCoords — per-vertex texture coordinates, each value contains 2 or 3 floats

savePointCloud()#

void cv::savePointCloud(
const String & filename,
InputArray vertices,
InputArray normals = noArray(),
InputArray rgb = noArray() )

#include <opencv2/ptcloud.hpp>

Python:

cv.savePointCloud(filename, vertices[, normals[, rgb]])

Saves a point cloud to a specified file.

The function saves point cloud to the specified file. File format is chosen based on the filename extension.

Parameters

  • filename — Name of the file

  • vertices — vertex coordinates, each value contains 3 floats

  • normals — per-vertex normals, each value contains 3 floats

  • rgb — per-vertex colors, each value contains 3 floats

triangleRasterize()#

void cv::triangleRasterize(
InputArray vertices,
InputArray indices,
InputArray colors,
InputOutputArray colorBuf,
InputOutputArray depthBuf,
InputArray world2cam,
double fovY,
double zNear,
double zFar,
const TriangleRasterizeSettings & settings = TriangleRasterizeSettings() )

#include <opencv2/ptcloud.hpp>

Python:

cv.triangleRasterize(vertices, indices, colors, colorBuf, depthBuf, world2cam, fovY, zNear, zFar[, settings]) -> colorBuf, depthBuf

Renders a set of triangles on a depth and color image.

Triangles can be drawn white (1.0, 1.0, 1.0), flat-shaded or with a color interpolation between vertices. In flat-shaded mode the 1st vertex color of each triangle is used to fill the whole triangle.

The world2cam is an inverted camera pose matrix in fact. It transforms vertices from world to camera coordinate system.

The camera coordinate system emulates the OpenGL’s coordinate system having coordinate origin in a screen center, X axis pointing right, Y axis pointing up and Z axis pointing towards the viewer except that image is vertically flipped after the render. This means that all visible objects are placed in z-negative area, or exactly in -zNear > z > -zFar since zNear and zFar are positive. For example, at fovY = PI/2 the point (0, 1, -1) will be projected to (width/2, 0) screen point, (1, 0, -1) to (width/2 + height/2, height/2). Increasing fovY makes projection smaller and vice versa.

The function does not create or clear output images before the rendering. This means that it can be used for drawing over an existing image or for rendering a model into a 3D scene using pre-filled Z-buffer.

Empty scene results in a depth buffer filled by the maximum value since every pixel is infinitely far from the camera. Therefore, before rendering anything from scratch the depthBuf should be filled by zFar values (or by ones in INVDEPTH mode).

There are special versions of this function named triangleRasterizeDepth and triangleRasterizeColor for cases if a user needs a color image or a depth image alone; they may run slightly faster.

Parameters

  • vertices — vertices coordinates array. Should contain values of CV_32FC3 type or a compatible one (e.g. cv::Vec3f, etc.)

  • indices — triangle vertices index array, 3 per triangle. Each index indicates a vertex in a vertices array. Should contain CV_32SC3 values or compatible

  • colors — per-vertex colors of CV_32FC3 type or compatible. Can be empty or the same size as vertices array. If the values are out of [0; 1] range, the result correctness is not guaranteed

  • colorBuf — an array representing the final rendered image. Should containt CV_32FC3 values and be the same size as depthBuf. Not cleared before rendering, i.e. the content is reused as there is some pre-rendered scene.

  • depthBuf — an array of floats containing resulting Z buffer. Should contain float values and be the same size as colorBuf. Not cleared before rendering, i.e. the content is reused as there is some pre-rendered scene. Empty scene corresponds to all values set to zFar (or to 1.0 in INVDEPTH mode)

  • world2cam — a 4x3 or 4x4 float or double matrix containing inverted (sic!) camera pose

  • fovY — field of view in vertical direction, given in radians

  • zNear — minimum Z value to render, everything closer is clipped

  • zFar — maximum Z value to render, everything farther is clipped

  • settings — see TriangleRasterizeSettings. By default the smooth shading is on, with CW culling and with disabled GL compatibility

triangleRasterizeColor()#

void cv::triangleRasterizeColor(
InputArray vertices,
InputArray indices,
InputArray colors,
InputOutputArray colorBuf,
InputArray world2cam,
double fovY,
double zNear,
double zFar,
const TriangleRasterizeSettings & settings = TriangleRasterizeSettings() )

#include <opencv2/ptcloud.hpp>

Python:

cv.triangleRasterizeColor(vertices, indices, colors, colorBuf, world2cam, fovY, zNear, zFar[, settings]) -> colorBuf

Overloaded version of triangleRasterize() with color-only rendering.

Parameters

  • vertices — vertices coordinates array. Should contain values of CV_32FC3 type or a compatible one (e.g. cv::Vec3f, etc.)

  • indices — triangle vertices index array, 3 per triangle. Each index indicates a vertex in a vertices array. Should contain CV_32SC3 values or compatible

  • colors — per-vertex colors of CV_32FC3 type or compatible. Can be empty or the same size as vertices array. If the values are out of [0; 1] range, the result correctness is not guaranteed

  • colorBuf — an array representing the final rendered image. Should containt CV_32FC3 values and be the same size as depthBuf. Not cleared before rendering, i.e. the content is reused as there is some pre-rendered scene.

  • world2cam — a 4x3 or 4x4 float or double matrix containing inverted (sic!) camera pose

  • fovY — field of view in vertical direction, given in radians

  • zNear — minimum Z value to render, everything closer is clipped

  • zFar — maximum Z value to render, everything farther is clipped

  • settings — see TriangleRasterizeSettings. By default the smooth shading is on, with CW culling and with disabled GL compatibility

triangleRasterizeDepth()#

void cv::triangleRasterizeDepth(
InputArray vertices,
InputArray indices,
InputOutputArray depthBuf,
InputArray world2cam,
double fovY,
double zNear,
double zFar,
const TriangleRasterizeSettings & settings = TriangleRasterizeSettings() )

#include <opencv2/ptcloud.hpp>

Python:

cv.triangleRasterizeDepth(vertices, indices, depthBuf, world2cam, fovY, zNear, zFar[, settings]) -> depthBuf

Overloaded version of triangleRasterize() with depth-only rendering.

Parameters

  • vertices — vertices coordinates array. Should contain values of CV_32FC3 type or a compatible one (e.g. cv::Vec3f, etc.)

  • indices — triangle vertices index array, 3 per triangle. Each index indicates a vertex in a vertices array. Should contain CV_32SC3 values or compatible

  • depthBuf — an array of floats containing resulting Z buffer. Should contain float values and be the same size as colorBuf. Not cleared before rendering, i.e. the content is reused as there is some pre-rendered scene. Empty scene corresponds to all values set to zFar (or to 1.0 in INVDEPTH mode)

  • world2cam — a 4x3 or 4x4 float or double matrix containing inverted (sic!) camera pose

  • fovY — field of view in vertical direction, given in radians

  • zNear — minimum Z value to render, everything closer is clipped

  • zFar — maximum Z value to render, everything farther is clipped

  • settings — see TriangleRasterizeSettings. By default the smooth shading is on, with CW culling and with disabled GL compatibility

depthTo3d()#

void cv::depthTo3d(
InputArray depth,
InputArray K,
OutputArray points3d,
InputArray mask = noArray() )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.depthTo3d(depth, K[, points3d[, mask]]) -> points3d

Converts a depth image to 3d points. If the mask is empty then the resulting array has the same dimensions as depth, otherwise it is 1d vector containing mask-enabled values only. The coordinate system is x pointing left, y down and z away from the camera

Parameters

  • depth — the depth image (if given as short int CV_U, it is assumed to be the depth in millimeters (as done with the Microsoft Kinect), otherwise, if given as CV_32F or CV_64F, it is assumed in meters)

  • K — The calibration matrix

  • points3d — the resulting 3d points (point is represented by 4 channels value [x, y, z, 0]). They are of the same depth as depth if it is CV_32F or CV_64F, and the depth of K if depth is of depth CV_16U or CV_16S

  • mask — the mask of the points to consider (can be empty)

depthTo3dSparse()#

void cv::depthTo3dSparse(
InputArray depth,
InputArray in_K,
InputArray in_points,
OutputArray points3d )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.depthTo3dSparse(depth, in_K, in_points[, points3d]) -> points3d

Parameters

  • depth — the depth image

  • in_K

  • in_points — the list of xy coordinates

  • points3d — the resulting 3d points (point is represented by 4 chanels value [x, y, z, 0])

findPlanes()#

void cv::findPlanes(
InputArray points3d,
InputArray normals,
OutputArray mask,
OutputArray plane_coefficients,
int block_size = 40,
int min_size = 40 *40,
double threshold = 0.01,
double sensor_error_a = 0,
double sensor_error_b = 0,
double sensor_error_c = 0,
RgbdPlaneMethod method = RGBD_PLANE_METHOD_DEFAULT )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.findPlanes(points3d, normals[, mask[, plane_coefficients[, block_size[, min_size[, threshold[, sensor_error_a[, sensor_error_b[, sensor_error_c[, method]]]]]]]]]) -> mask, plane_coefficients

Find the planes in a depth image

Parameters

  • points3d — the 3d points organized like the depth image: rows x cols with 3 channels

  • normals — the normals for every point in the depth image; optional, can be empty

  • mask — An image where each pixel is labeled with the plane it belongs to and 255 if it does not belong to any plane

  • plane_coefficients — the coefficients of the corresponding planes (a,b,c,d) such that ax+by+cz+d=0, norm(a,b,c)=1 and c < 0 (so that the normal points towards the camera)

  • block_size — The size of the blocks to look at for a stable MSE

  • min_size — The minimum size of a cluster to be considered a plane

  • threshold — The maximum distance of a point from a plane to belong to it (in meters)

  • sensor_error_a — coefficient of the sensor error. 0 by default, use 0.0075 for a Kinect

  • sensor_error_b — coefficient of the sensor error. 0 by default

  • sensor_error_c — coefficient of the sensor error. 0 by default

  • method — The method to use to compute the planes.

registerDepth()#

void cv::registerDepth(
InputArray unregisteredCameraMatrix,
InputArray registeredCameraMatrix,
InputArray registeredDistCoeffs,
InputArray Rt,
InputArray unregisteredDepth,
const Size & outputImagePlaneSize,
OutputArray registeredDepth,
bool depthDilation = false )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.registerDepth(unregisteredCameraMatrix, registeredCameraMatrix, registeredDistCoeffs, Rt, unregisteredDepth, outputImagePlaneSize[, registeredDepth[, depthDilation]]) -> registeredDepth

Registers depth data to an external camera Registration is performed by creating a depth cloud, transforming the cloud by the rigid body transformation between the cameras, and then projecting the transformed points into the RGB camera.

uv_rgb = K_rgb * [R | t] * z * inv(K_ir) * uv_ir

Currently does not check for negative depth values.

Parameters

  • unregisteredCameraMatrix — the camera matrix of the depth camera

  • registeredCameraMatrix — the camera matrix of the external camera

  • registeredDistCoeffs — the distortion coefficients of the external camera

  • Rt — the rigid body transform between the cameras. Transforms points from depth camera frame to external camera frame.

  • unregisteredDepth — the input depth data

  • outputImagePlaneSize — the image plane dimensions of the external camera (width, height)

  • registeredDepth — the result of transforming the depth into the external camera

  • depthDilation — whether or not the depth is dilated to avoid holes and occlusion errors (optional)

rescaleDepth()#

void cv::rescaleDepth(
InputArray in,
int type,
OutputArray out,
double depth_factor = 1000.0 )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.rescaleDepth(in_, type[, out[, depth_factor]]) -> out

If the input image is of type CV_16UC1 (like the Kinect one), the image is converted to floats, divided by depth_factor to get a depth in meters, and the values 0 are converted to std::numeric_limits::quiet_NaN() Otherwise, the image is simply converted to floats

Parameters

  • in — the depth image (if given as short int CV_U, it is assumed to be the depth in millimeters (as done with the Microsoft Kinect), it is assumed in meters)

  • type — the desired output depth (CV_32F or CV_64F)

  • out — The rescaled float depth image

  • depth_factor — (optional) factor by which depth is converted to distance (by default = 1000.0 for Kinect sensor)

warpFrame()#

void cv::warpFrame(
InputArray depth,
InputArray image,
InputArray mask,
InputArray Rt,
InputArray cameraMatrix,
OutputArray warpedDepth = noArray(),
OutputArray warpedImage = noArray(),
OutputArray warpedMask = noArray() )

#include <opencv2/ptcloud/depth.hpp>

Python:

cv.warpFrame(depth, image, mask, Rt, cameraMatrix[, warpedDepth[, warpedImage[, warpedMask]]]) -> warpedDepth, warpedImage, warpedMask

Warps depth or RGB-D image by reprojecting it in 3d, applying Rt transformation and then projecting it back onto the image plane. This function can be used to visualize the results of the Odometry algorithm.

Parameters

  • depth — Depth data, should be 1-channel CV_16U, CV_16S, CV_32F or CV_64F

  • image — RGB image (optional), should be 1-, 3- or 4-channel CV_8U

  • mask — Mask of used pixels (optional), should be CV_8UC1, CV_8SC1 or CV_BoolC1

  • Rt — Rotation+translation matrix (3x4 or 4x4) to be applied to depth points

  • cameraMatrix — Camera intrinsics matrix (3x3)

  • warpedDepth — The warped depth data (optional)

  • warpedImage — The warped RGB image (optional)

  • warpedMask — The mask of valid pixels in warped image (optional)