OpenCV  4.8.0
Open Source Computer Vision
Detection of ArUco Boards

Prev Tutorial: Detection of ArUco Markers

Next Tutorial: Detection of ChArUco Boards

An ArUco Board is a set of markers that acts like a single marker in the sense that it provides a single pose for the camera.

The most popular board is the one with all the markers in the same plane, since it can be easily printed:

gboriginal.png

However, boards are not limited to this arrangement and can represent any 2d or 3d layout.

The difference between a Board and a set of independent markers is that the relative position between the markers in the Board is known a priori. This allows that the corners of all the markers can be used for estimating the pose of the camera respect to the whole Board.

When you use a set of independent markers, you can estimate the pose for each marker individually, since you dont know the relative position of the markers in the environment.

The main benefits of using Boards are:

Board Detection

A Board detection is similar to the standard marker detection. The only difference is in the pose estimation step. In fact, to use marker boards, a standard marker detection should be done before estimating the Board pose.

To perform pose estimation for boards, you should use cv::solvePnP function, as shown below:

cv::Mat inputImage;
// Camera parameters are read from somewhere
cv::Mat cameraMatrix, distCoeffs;
// You can read camera parameters from tutorial_camera_params.yml
readCameraParameters(filename, cameraMatrix, distCoeffs); // This function is implemented in aruco_samples_utility.hpp
// Assume we have a function to create the board object
cv::Ptr<cv::aruco::Board> board = cv::aruco::Board::create();
...
std::vector<int> markerIds;
std::vector<std::vector<cv::Point2f>> markerCorners;
cv::aruco::ArucoDetector detector(board.dictionary, detectorParams);
detector.detectMarkers(inputImage, markerCorners, markerIds);
cv::Vec3d rvec, tvec;
// If at least one marker detected
if(markerIds.size() > 0) {
// Get object and image points for the solvePnP function
cv::Mat objPoints, imgPoints;
board->matchImagePoints(markerCorners, markerIds, objPoints, imgPoints);
// Find pose
cv::solvePnP(objPoints, imgPoints, cameraMatrix, distCoeffs, rvec, tvec);
}

The parameters are:

The drawFrameAxes() function can be used to check the obtained pose. For instance:

gbmarkersaxis.jpg
Board with axis

And this is another example with the board partially occluded:

gbocclusion.png
Board with occlusions
Note
The center and direction of the axes has been changed

As it can be observed, although some markers have not been detected, the Board pose can still be estimated from the rest of markers.

Grid Board

Creating the Board object requires specifying the corner positions for each marker in the environment. However, in many cases, the board will be just a set of markers in the same plane and in a grid layout, so it can be easily printed and used.

Fortunately, the aruco module provides the basic functionality to create and print these types of markers easily.

The GridBoard class is a specialized class that inherits from the Board class and which represents a Board with all the markers in the same plane and in a grid layout, as in the following image:

gboriginal.png
Image with aruco board

Concretely, the coordinate system in a Grid Board is positioned in the board plane, centered in the bottom left corner of the board and with the Z pointing out, like in the following image (X:red, Y:green, Z:blue):

gbaxis.jpg
Board with axis

A GridBoard object can be defined using the following parameters:

This object can be easily created from these parameters using the cv::aruco::GridBoard::create() static function:

cv::aruco::GridBoard board = cv::aruco::GridBoard::create(5, 7, 0.04, 0.01, dictionary);

So, this board will be composed by 5x7=35 markers. The ids of each of the markers are assigned, by default, in ascending order starting on 0, so they will be 0, 1, 2, ..., 34.

After creating a Grid Board, we probably want to print it and use it. A function to generate the image of a GridBoard is provided in cv::aruco::GridBoard::generateImage(). For example:

cv::Ptr<cv::aruco::GridBoard> board = cv::aruco::GridBoard::create(5, 7, 0.04, 0.01, dictionary);
cv::Mat boardImage;
board->generateImage( cv::Size(600, 500), boardImage, 10, 1 );

The output image will be something like this:

board.png

A full working example of board creation is included in the create_board.cpp inside the modules/aruco/samples/.

Note: The samples now take input via commandline via the OpenCV Commandline Parser. For this file the example parameters will look like

"_output_path_/aboard.png" -w=5 -h=7 -l=100 -s=10 -d=10

Finally, a full example of board detection:

cv::VideoCapture inputVideo;
inputVideo.open(0);
cv::Mat cameraMatrix, distCoeffs;
// You can read camera parameters from tutorial_camera_params.yml
readCameraParameters(filename, cameraMatrix, distCoeffs); // This function is implemented in aruco_samples_utility.hpp
// To use tutorial sample, you need read custom dictionaty from tutorial_dict.yml
readDictionary(filename, dictionary); // This function is implemented in opencv/modules/objdetect/src/aruco/aruco_dictionary.cpp
cv::Ptr<cv::aruco::GridBoard> board = cv::aruco::GridBoard::create(5, 7, 0.04, 0.01, dictionary);
cv::aruco::ArucoDetector detector(dictionary, detectorParams);
while (inputVideo.grab()) {
cv::Mat image, imageCopy;
inputVideo.retrieve(image);
image.copyTo(imageCopy);
std::vector<int> ids;
std::vector<std::vector<cv::Point2f> > corners;
// Detect markers
detector.detectMarkers(image, corners, ids);
// If at least one marker detected
if (ids.size() > 0) {
cv::aruco::drawDetectedMarkers(imageCopy, corners, ids);
cv::Vec3d rvec, tvec;
// Get object and image points for the solvePnP function
cv::Mat objPoints, imgPoints;
board->matchImagePoints(corners, ids, objPoints, imgPoints);
// Find pose
cv::solvePnP(objPoints, imgPoints, cameraMatrix, distCoeffs, rvec, tvec);
// If at least one board marker detected
markersOfBoardDetected = (int)objPoints.total() / 4;
if(markersOfBoardDetected > 0)
cv::drawFrameAxes(imageCopy, cameraMatrix, distCoeffs, rvec, tvec, 0.1);
}
cv::imshow("out", imageCopy);
char key = (char) cv::waitKey(waitTime);
if (key == 27)
break;
}

Sample video:

A full working example is included in the detect_board.cpp inside the modules/aruco/samples/.

Note: The samples now take input via commandline via the OpenCV Commandline Parser. For this file the example parameters will look like

-w=5 -h=7 -l=100 -s=10
-v=/path_to_aruco_tutorials/aruco_board_detection/images/gboriginal.png
-c=/path_to_aruco_samples/tutorial_camera_params.yml
-cd=/path_to_aruco_samples/tutorial_dict.yml

Parameters for detect_board.cpp:

const char* keys =
"{w | | Number of squares in X direction }"
"{h | | Number of squares in Y direction }"
"{l | | Marker side length (in pixels) }"
"{s | | Separation between two consecutive markers in the grid (in pixels)}"
"{d | | dictionary: DICT_4X4_50=0, DICT_4X4_100=1, DICT_4X4_250=2,"
"DICT_4X4_1000=3, DICT_5X5_50=4, DICT_5X5_100=5, DICT_5X5_250=6, DICT_5X5_1000=7, "
"DICT_6X6_50=8, DICT_6X6_100=9, DICT_6X6_250=10, DICT_6X6_1000=11, DICT_7X7_50=12,"
"DICT_7X7_100=13, DICT_7X7_250=14, DICT_7X7_1000=15, DICT_ARUCO_ORIGINAL = 16}"
"{cd | | Input file with custom dictionary }"
"{c | | Output file with calibrated camera parameters }"
"{v | | Input from video or image file, if omitted, input comes from camera }"
"{ci | 0 | Camera id if input doesnt come from video (-v) }"
"{dp | | File of marker detector parameters }"
"{rs | | Apply refind strategy }"
"{r | | show rejected candidates too }";
}
Note
To work with examples from the tutorial, you can use camera parameters from tutorial_camera_params.yml and you need use custom dictionary from tutorial_dict.yml. An example of usage in detect_board.cpp.

Refine marker detection

ArUco boards can also be used to improve the detection of markers. If we have detected a subset of the markers that belongs to the board, we can use these markers and the board layout information to try to find the markers that have not been previously detected.

This can be done using the refineDetectedMarkers() function, which should be called after calling detectMarkers().

The main parameters of this function are the original image where markers were detected, the Board object, the detected marker corners, the detected marker ids and the rejected marker corners.

The rejected corners can be obtained from the detectMarkers() function and are also known as marker candidates. This candidates are square shapes that have been found in the original image but have failed to pass the identification step (i.e. their inner codification presents too many errors) and thus they have not been recognized as markers.

However, these candidates are sometimes actual markers that have not been correctly identified due to high noise in the image, very low resolution or other related problems that affect to the binary code extraction. The refineDetectedMarkers() function finds correspondences between these candidates and the missing markers of the board. This search is based on two parameters:

This is an example of using the refineDetectedMarkers() function:

cv::Ptr<cv::aruco::GridBoard> board = cv::aruco::GridBoard::create(5, 7, 0.04, 0.01, dictionary);
cv::aruco::ArucoDetector detector(dictionary, detectorParams);
std::vector<int> markerIds;
std::vector<std::vector<cv::Point2f>> markerCorners, rejectedCandidates;
detector.detectMarkers(inputImage, markerCorners, markerIds, rejectedCandidates);
detector.refineDetectedMarkers(inputImage, board, markerCorners, markerIds, rejectedCandidates);
// After calling this function, if any new marker has been detected it will be removed from rejectedCandidates and included
// at the end of markerCorners and markerIds

It must also be noted that, in some cases, if the number of detected markers in the first place is too low (for instance only 1 or 2 markers), the projections of the missing markers can be of bad quality, producing erroneous correspondences.

See module samples for a more detailed implementation.