3D vision functionality#

Detailed Description#

Most of the functions in this section use a so-called pinhole camera model. The view of a scene is obtained by projecting a scene’s 3D point \(P_w\) into the image plane using a perspective transformation which forms the corresponding pixel \(p\). Both \(P_w\) and \(p\) are represented in homogeneous coordinates, i.e. as 3D and 2D homogeneous vector respectively. You will find a brief introduction to projective geometry, homogeneous vectors and homogeneous transformations at the end of this section’s introduction. For more succinct notation, we often drop the ‘homogeneous’ and say vector instead of homogeneous vector.

The distortion-free projective transformation given by a pinhole camera model is shown below.

\[ s \; p = A \begin{bmatrix} R|t \end{bmatrix} P_w, \]

where \(P_w\) is a 3D point expressed with respect to the world coordinate system, \(p\) is a 2D pixel in the image plane, \(A\) is the camera intrinsic matrix, \(R\) and \(t\) are the rotation and translation that describe the change of coordinates from world to camera coordinate systems (or camera frame) and \(s\) is the projective transformation’s arbitrary scaling and not part of the camera model.

The camera intrinsic matrix \(A\) (notation used as in [353] and also generally notated as \(K\)) projects 3D points given in the camera coordinate system to 2D pixel coordinates, i.e.

\[ p = A P_c. \]

The camera intrinsic matrix \(A\) is composed of the focal lengths \(f_x\) and \(f_y\), which are expressed in pixel units, and the principal point \((c_x, c_y)\), that is usually close to the image center:

\[ A = \vecthreethree{f_x}{0}{c_x}{0}{f_y}{c_y}{0}{0}{1}, \]

and thus

\[ s \vecthree{u}{v}{1} = \vecthreethree{f_x}{0}{c_x}{0}{f_y}{c_y}{0}{0}{1} \vecthree{X_c}{Y_c}{Z_c}. \]

The matrix of intrinsic parameters does not depend on the scene viewed. So, once estimated, it can be re-used as long as the focal length is fixed (in case of a zoom lens). Thus, if an image from the camera is scaled by a factor, all of these parameters need to be scaled (multiplied/divided, respectively) by the same factor.

The joint rotation-translation matrix \([R|t]\) is the matrix product of a projective transformation and a homogeneous transformation. The 3-by-4 projective transformation maps 3D points represented in camera coordinates to 2D points in the image plane and represented in normalized camera coordinates \(x' = X_c / Z_c\) and \(y' = Y_c / Z_c\):

\[\begin{split} Z_c \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} X_c \\ Y_c \\ Z_c \\ 1 \end{bmatrix}. \end{split}\]

The homogeneous transformation is encoded by the extrinsic parameters \(R\) and \(t\) and represents the change of basis from world coordinate system \(w\) to the camera coordinate sytem \(c\). Thus, given the representation of the point \(P\) in world coordinates, \(P_w\), we obtain \(P\)’s representation in the camera coordinate system, \(P_c\), by

\[\begin{split} P_c = \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix} P_w, \end{split}\]

This homogeneous transformation is composed out of \(R\), a 3-by-3 rotation matrix, and \(t\), a 3-by-1 translation vector:

\[\begin{split} \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_x \\ r_{21} & r_{22} & r_{23} & t_y \\ r_{31} & r_{32} & r_{33} & t_z \\ 0 & 0 & 0 & 1 \end{bmatrix}, \end{split}\]

and therefore

\[\begin{split} \begin{bmatrix} X_c \\ Y_c \\ Z_c \\ 1 \end{bmatrix} = \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_x \\ r_{21} & r_{22} & r_{23} & t_y \\ r_{31} & r_{32} & r_{33} & t_z \\ 0 & 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}. \end{split}\]

Combining the projective transformation and the homogeneous transformation, we obtain the projective transformation that maps 3D points in world coordinates into 2D points in the image plane and in normalized camera coordinates:

\[\begin{split} Z_c \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} R|t \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix} = \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_x \\ r_{21} & r_{22} & r_{23} & t_y \\ r_{31} & r_{32} & r_{33} & t_z \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}, \end{split}\]

with \(x' = X_c / Z_c\) and \(y' = Y_c / Z_c\). Putting the equations for instrincs and extrinsics together, we can write out \(s \; p = A \begin{bmatrix} R|t \end{bmatrix} P_w\) as

\[\begin{split} s \vecthree{u}{v}{1} = \vecthreethree{f_x}{0}{c_x}{0}{f_y}{c_y}{0}{0}{1} \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_x \\ r_{21} & r_{22} & r_{23} & t_y \\ r_{31} & r_{32} & r_{33} & t_z \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}. \end{split}\]

If \(Z_c \ne 0\), the transformation above is equivalent to the following,

\[\begin{split} \begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} f_x X_c/Z_c + c_x \\ f_y Y_c/Z_c + c_y \end{bmatrix} \end{split}\]

with

\[\begin{split} \vecthree{X_c}{Y_c}{Z_c} = \begin{bmatrix} R|t \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}. \end{split}\]

The following figure illustrates the pinhole camera model.

Pinhole camera model

Real lenses usually have some distortion, mostly radial distortion, and slight tangential distortion. So, the above model is extended as:

\[\begin{split} \begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} f_x x'' + c_x \\ f_y y'' + c_y \end{bmatrix} \end{split}\]

where

\[\begin{split} \begin{bmatrix} x'' \\ y'' \end{bmatrix} = \begin{bmatrix} x' \frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + 2 p_1 x' y' + p_2(r^2 + 2 x'^2) + s_1 r^2 + s_2 r^4 \\ y' \frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + p_1 (r^2 + 2 y'^2) + 2 p_2 x' y' + s_3 r^2 + s_4 r^4 \\ \end{bmatrix} \end{split}\]

with

\[ r^2 = x'^2 + y'^2 \]

and

\[\begin{split} \begin{bmatrix} x'\\ y' \end{bmatrix} = \begin{bmatrix} X_c/Z_c \\ Y_c/Z_c \end{bmatrix}, \end{split}\]

if \(Z_c \ne 0\).

The distortion parameters are the radial coefficients \(k_1\), \(k_2\), \(k_3\), \(k_4\), \(k_5\), and \(k_6\) , \(p_1\) and \(p_2\) are the tangential distortion coefficients, and \(s_1\), \(s_2\), \(s_3\), and \(s_4\), are the thin prism distortion coefficients. Higher-order coefficients are not considered in OpenCV.

The next figures show two common types of radial distortion: barrel distortion ( \( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6 \) monotonically decreasing) and pincushion distortion ( \( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6 \) monotonically increasing). Radial distortion is always monotonic for real lenses, and if the estimator produces a non-monotonic result, this should be considered a calibration failure. More generally, radial distortion must be monotonic and the distortion function must be bijective. A failed estimation result may look deceptively good near the image center but will work poorly in e.g. AR/SFM applications. The optimization method used in OpenCV camera calibration does not include these constraints as the framework does not support the required integer programming and polynomial inequalities. See issue #15992 for additional information.

In some cases, the image sensor may be tilted in order to focus an oblique plane in front of the camera (Scheimpflug principle). This can be useful for particle image velocimetry (PIV) or triangulation with a laser fan. The tilt causes a perspective distortion of \(x''\) and \(y''\). This distortion can be modeled in the following way, see e.g. [191].

\[\begin{split} \begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} f_x x''' + c_x \\ f_y y''' + c_y \end{bmatrix}, \end{split}\]

where

\[ s\vecthree{x'''}{y'''}{1} = \vecthreethree{R_{33}(\tau_x, \tau_y)}{0}{-R_{13}(\tau_x, \tau_y)} {0}{R_{33}(\tau_x, \tau_y)}{-R_{23}(\tau_x, \tau_y)} {0}{0}{1} R(\tau_x, \tau_y) \vecthree{x''}{y''}{1} \]

and the matrix \(R(\tau_x, \tau_y)\) is defined by two rotations with angular parameter \(\tau_x\) and \(\tau_y\), respectively,

\[ R(\tau_x, \tau_y) = \vecthreethree{\cos(\tau_y)}{0}{-\sin(\tau_y)}{0}{1}{0}{\sin(\tau_y)}{0}{\cos(\tau_y)} \vecthreethree{1}{0}{0}{0}{\cos(\tau_x)}{\sin(\tau_x)}{0}{-\sin(\tau_x)}{\cos(\tau_x)} = \vecthreethree{\cos(\tau_y)}{\sin(\tau_y)\sin(\tau_x)}{-\sin(\tau_y)\cos(\tau_x)} {0}{\cos(\tau_x)}{\sin(\tau_x)} {\sin(\tau_y)}{-\cos(\tau_y)\sin(\tau_x)}{\cos(\tau_y)\cos(\tau_x)}. \]

In the functions below the coefficients are passed or returned as

\[ (k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \tau_x, \tau_y]]]]) \]

vector. That is, if the vector contains four elements, it means that \(k_3=0\) . The distortion coefficients do not depend on the scene viewed. Thus, they also belong to the intrinsic camera parameters. And they remain the same regardless of the captured image resolution. If, for example, a camera has been calibrated on images of 320 x 240 resolution, absolutely the same distortion coefficients can be used for 640 x 480 images from the same camera while \(f_x\), \(f_y\), \(c_x\), and \(c_y\) need to be scaled appropriately.

The functions below use the above model to do the following:

Project 3D points to the image plane given intrinsic and extrinsic parameters.
Compute extrinsic parameters given intrinsic parameters, a few 3D points, and their projections.
Estimate intrinsic and extrinsic camera parameters from several views of a known calibration pattern (every view is described by several 3D-2D point correspondences).
Estimate the relative position and orientation of the stereo camera “heads” and compute the rectification* transformation that makes the camera optical axes parallel.

Homogeneous Coordinates

Homogeneous Coordinates are a system of coordinates that are used in projective geometry. Their use allows to represent points at infinity by finite coordinates and simplifies formulas when compared to the cartesian counterparts, e.g. they have the advantage that affine transformations can be expressed as linear homogeneous transformation.

One obtains the homogeneous vector \(P_h\) by appending a 1 along an n-dimensional cartesian vector \(P\) e.g. for a 3D cartesian vector the mapping \(P \rightarrow P_h\) is:

\[\begin{split} \begin{bmatrix} X \\ Y \\ Z \end{bmatrix} \rightarrow \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix}. \end{split}\]

For the inverse mapping \(P_h \rightarrow P\), one divides all elements of the homogeneous vector by its last element, e.g. for a 3D homogeneous vector one gets its 2D cartesian counterpart by:

\[\begin{split} \begin{bmatrix} X \\ Y \\ W \end{bmatrix} \rightarrow \begin{bmatrix} X / W \\ Y / W \end{bmatrix}, \end{split}\]

if \(W \ne 0\).

Due to this mapping, all multiples \(k P_h\), for \(k \ne 0\), of a homogeneous point represent the same point \(P_h\). An intuitive understanding of this property is that under a projective transformation, all multiples of \(P_h\) are mapped to the same point. This is the physical observation one does for pinhole cameras, as all points along a ray through the camera’s pinhole are projected to the same image point, e.g. all points along the red ray in the image of the pinhole camera model above would be mapped to the same image coordinate. This property is also the source for the scale ambiguity s in the equation of the pinhole camera model.

As mentioned, by using homogeneous coordinates we can express any change of basis parameterized by \(R\) and \(t\) as a linear transformation, e.g. for the change of basis from coordinate system 0 to coordinate system 1 becomes:

\[\begin{split} P_1 = R P_0 + t \rightarrow P_{h_1} = \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix} P_{h_0}. \end{split}\]

Note

Many functions in this module take a camera intrinsic matrix as an input parameter. Although all functions assume the same structure of this parameter, they may name it differently. The parameter’s description, however, will be clear in that a camera intrinsic matrix with the structure shown above is required.
A calibration sample for 3 cameras in a horizontal position can be found at opencv_source_code/samples/cpp/3calibration.cpp
A calibration sample based on a sequence of images can be found at opencv_source_code/samples/cpp/calibration.cpp
A calibration sample in order to do 3D reconstruction can be found at opencv_source_code/samples/cpp/build3dmodel.cpp
A calibration example on stereo calibration can be found at opencv_source_code/samples/cpp/stereo_calib.cpp
A calibration example on stereo matching can be found at opencv_source_code/samples/cpp/stereo_match.cpp
(Python) A camera calibration sample can be found at opencv_source_code/samples/python/calibrate.py

Namespaces#

cv::fisheye namespace

Classes#

Name	Description
`class` `cv::LevMarq`	Levenberg-Marquadt solver. View details
`struct` `cv::UsacParams`	View details

Enumerations#

enum cv {
    LMEDS = 4,
    RANSAC = 8,
    RHO = 16,
    USAC_DEFAULT = 32,
    USAC_PARALLEL = 33,
    USAC_FM_8PTS = 34,
    USAC_FAST = 35,
    USAC_ACCURATE = 36,
    USAC_PROSAC = 37,
    USAC_MAGSAC = 38
}

type of the robust estimation algorithm View details

enum cv {
    FM_7POINT = 1,
    FM_8POINT = 2,
    FM_LMEDS = 4,
    FM_RANSAC = 8
}

the algorithm for finding fundamental matrix View details

enum cv::LocalOptimMethod {
    cv::LOCAL_OPTIM_NULL =0,
    cv::LOCAL_OPTIM_INNER_LO =1,
    cv::LOCAL_OPTIM_INNER_AND_ITER_LO =2,
    cv::LOCAL_OPTIM_GC =3,
    cv::LOCAL_OPTIM_SIGMA =4
}

View details

enum struct cv::MatrixType {
    cv::MatrixType::AUTO = 0,
    cv::MatrixType::DENSE = 1,
    cv::MatrixType::SPARSE = 2
}

Type of matrix used in LevMarq solver. View details

enum cv::NeighborSearchMethod {
    cv::NEIGH_FLANN_KNN =0,
    cv::NEIGH_GRID =1,
    cv::NEIGH_FLANN_RADIUS =2
}

View details

enum cv::PolishingMethod {
    cv::NONE_POLISHER =0,
    cv::LSQ_POLISHER =1,
    cv::MAGSAC =2,
    cv::COV_POLISHER =3
}

View details

enum cv::SamplingMethod {
    cv::SAMPLING_UNIFORM =0,
    cv::SAMPLING_PROGRESSIVE_NAPSAC =1,
    cv::SAMPLING_NAPSAC =2,
    cv::SAMPLING_PROSAC =3
}

View details

enum cv::ScoreMethod {
    cv::SCORE_METHOD_RANSAC =0,
    cv::SCORE_METHOD_MSAC =1,
    cv::SCORE_METHOD_MAGSAC =2,
    cv::SCORE_METHOD_LMEDS =3
}

View details

enum cv::SolvePnPMethod {
    cv::SOLVEPNP_ITERATIVE = 0,
    cv::SOLVEPNP_EPNP = 1,
    cv::SOLVEPNP_P3P = 2,
    cv::SOLVEPNP_AP3P = 3,
    cv::SOLVEPNP_IPPE = 4,
    cv::SOLVEPNP_IPPE_SQUARE = 5,
    cv::SOLVEPNP_SQPNP = 6
}

View details

enum struct cv::VariableType {
    cv::VariableType::LINEAR = 0,
    cv::VariableType::SO3 = 1,
    cv::VariableType::SE3 = 2
}

Type of variables used in LevMarq solver. View details

`LMEDS`	least-median of squares algorithm
`RANSAC`	RANSAC algorithm.
`RHO`	RHO algorithm.
`USAC_DEFAULT`	USAC algorithm, default settings.
`USAC_PARALLEL`	USAC, parallel version.
`USAC_FM_8PTS`	USAC, fundamental matrix 8 points.
`USAC_FAST`	USAC, fast settings.
`USAC_ACCURATE`	USAC, accurate settings.
`USAC_PROSAC`	USAC, sorted points, runs PROSAC.
`USAC_MAGSAC`	USAC, runs MAGSAC++.

`FM_7POINT`	7-point algorithm
`FM_8POINT`	8-point algorithm
`FM_LMEDS`	least-median algorithm. 7-point algorithm is used.
`FM_RANSAC`	RANSAC algorithm. It needs at least 15 points. 7-point algorithm is used.

`SOLVEPNP_ITERATIVE` Python: `cv.SOLVEPNP_ITERATIVE`	Pose refinement using non-linear Levenberg-Marquardt minimization scheme [200] [88] Initial solution for non-planar “objectPoints” needs at least 6 points and uses the DLT algorithm. Initial solution for planar “objectPoints” needs at least 4 points and uses pose from homography decomposition.
`SOLVEPNP_EPNP` Python: `cv.SOLVEPNP_EPNP`	EPnP: Efficient Perspective-n-Point Camera Pose Estimation [176].
`SOLVEPNP_P3P` Python: `cv.SOLVEPNP_P3P`	Revisiting the P3P Problem [79].
`SOLVEPNP_AP3P` Python: `cv.SOLVEPNP_AP3P`	An Efficient Algebraic Solution to the Perspective-Three-Point Problem [163].
`SOLVEPNP_IPPE` Python: `cv.SOLVEPNP_IPPE`	Infinitesimal Plane-Based Pose Estimation [69] Object points must be coplanar.
`SOLVEPNP_IPPE_SQUARE` Python: `cv.SOLVEPNP_IPPE_SQUARE`	Infinitesimal Plane-Based Pose Estimation [69] This is a special case suitable for marker pose estimation. 4 coplanar object points must be defined in the following order: point 0: [-squareLength / 2, squareLength / 2, 0] point 1: [ squareLength / 2, squareLength / 2, 0] point 2: [ squareLength / 2, -squareLength / 2, 0] point 3: [-squareLength / 2, -squareLength / 2, 0]
`SOLVEPNP_SQPNP` Python: `cv.SOLVEPNP_SQPNP`	SQPnP: A Consistently Fast and Globally OptimalSolution to the Perspective-n-Point Problem [302].

3D vision functionality#

Detailed Description#

Namespaces#

Classes#

Enumerations#

Enumeration Type Documentation#

enum#

enum#

LocalOptimMethod#

MatrixType#

NeighborSearchMethod#

PolishingMethod#

SamplingMethod#

ScoreMethod#

SolvePnPMethod#

VariableType#

Function Documentation#

calibrationMatrixValues()#

composeRT()#

computeCorrespondEpilines()#

convertPointsFromHomogeneous()#

convertPointsHomogeneous()#

convertPointsToHomogeneous()#

correctMatches()#

decomposeEssentialMat()#

decomposeHomographyMat()#

decomposeProjectionMatrix()#

estimateAffine2D()#

estimateAffine2D()#

estimateAffine3D()#

estimateAffine3D()#

estimateAffinePartial2D()#

estimateTranslation2D()#

estimateTranslation3D()#

filterHomographyDecompByVisibleRefpoints()#

findEssentialMat()#

findEssentialMat()#

findEssentialMat()#

findEssentialMat()#

findFundamentalMat()#

findFundamentalMat()#

findFundamentalMat()#

findFundamentalMat()#

findHomography()#

findHomography()#

findHomography()#

getDefaultNewCameraMatrix()#

getOptimalNewCameraMatrix()#

getUndistortRectangles()#

matMulDeriv()#

projectPoints()#

projectPoints()#

recoverPose()#

recoverPose()#

recoverPose()#

recoverPose()#

Rodrigues()#

RQDecomp3x3()#

sampsonDistance()#

solveP3P()#

solvePnP()#

solvePnPGeneric()#

solvePnPRansac()#

solvePnPRansac()#

solvePnPRefineLM()#

solvePnPRefineVVS()#

triangulatePoints()#

undistortImagePoints()#

undistortPoints()#