3D object recognition and pose estimation API#

As CNN based learning algorithm shows better performance on the classification issues, the rich labeled data could be more useful in the training stage. 3D object classification and pose estimation is a jointed mission aiming at separate different posed apart in the descriptor form.

In the training stage, we prepare 2D training images generated from our module with their class label and pose label. We fully exploit the information lies in their labels by using a triplet and pair-wise jointed loss function in CNN training.

As CNN based learning algorithm shows better performance on the classification issues, the rich labeled data could be more useful in the training stage. 3D object classification and pose estimation is a jointed mission aiming at separate different posea apart in the descriptor form.

In the training stage, we prepare 2D training images generated from our module with their class label and pose label. We fully exploit the information that lies in their labels by using a triplet and pair-wise jointed loss function in CNN training.

Both class and pose label are in consideration in the triplet loss. The loss score will be smaller when features from the same class and same pose is more similar and features from different classes or different poses will lead to a much larger loss score.

This loss is also jointed with a pair wise component to make sure the loss is never be zero and have a restriction on the model scale.

About the training and feature extraction process, it is a rough implementation by using OpenCV and Caffe from the idea of Paul Wohlhart. The principal purpose of this API is constructing a well labeled database from .ply models for CNN training with triplet loss and extracting features with the constructed model for prediction or other purpose of pattern recognition, algorithms into two main Class:

icoSphere: methods belonging to this class generates 2D images from a 3D model, together with their class and pose from camera view labels.

descriptorExtractor: methods belonging to this class extract descriptors from 2D images which is discriminant on category prediction and pose estimation.

Note

This API need Caffe with triplet version which is designed for this module Wangyida/caffe.

Classes#

Name

Description

class cv::cnn_3dobj::descriptorExtractor

Caffe based 3D images descriptor. A class to extract features from an image. The so obtained descriptors can be used for classification and pose estimation goals [335].

class cv::cnn_3dobj::icoSphere

Icosohedron based camera view data generator. The class create some sphere views of camera towards a 3D object meshed from .ply files [140] .

Class cv::cnn_3dobj::descriptorExtractor#

Caffe based 3D images descriptor. A class to extract features from an image. The so obtained descriptors can be used for classification and pose estimation goals [335].

#include <opencv2/cnn_3dobj.hpp>

Detailed Description#

Caffe based 3D images descriptor. A class to extract features from an image. The so obtained descriptors can be used for classification and pose estimation goals [335].

Constructor & Destructor Documentation#

descriptorExtractor()#

cv::cnn_3dobj::descriptorExtractor::descriptorExtractor(
const String & device_type,
int device_id = 0 )

Set the device for feature extraction, if the GPU is used, there should be a device_id.

Parameters

  • device_type — CPU or GPU.

  • device_id — ID of GPU.

Member Function Documentation#

preprocess()#

void cv::cnn_3dobj::descriptorExtractor::preprocess(
const cv::Mat & img,
std::vector< cv::Mat > * input_channels )

Convert the input image to the input image format of the network.

setMean()#

void cv::cnn_3dobj::descriptorExtractor::setMean(const String & mean_file)

Load the mean file in binaryproto format if it is needed.

Parameters

  • mean_file — Path of mean file which stores the mean of training images, it is usually generated by Caffe tool.

wrapInput()#

void cv::cnn_3dobj::descriptorExtractor::wrapInput(std::vector< cv::Mat > * input_channels)

Wrap the input layer of the network in separate cv::Mat objects(one per channel). This way we save one memcpy operation and we don’t need to rely on cudaMemcpy2D. The last preprocessing operation will write the separate channels directly to the input layer.

extract()#

void cv::cnn_3dobj::descriptorExtractor::extract(
InputArrayOfArrays inputimg,
OutputArray feature,
String feature_blob )

Extract features from a single image or from a vector of images. If loadNet was not called before, this method invocation will fail.

Parameters

  • inputimg — Input images.

  • feature — Output features.

  • feature_blob — Layer which the feature is extracted from.

getDeviceId()#

int cv::cnn_3dobj::descriptorExtractor::getDeviceId()

Get device ID information for feature extraction.

getDeviceType()#

String cv::cnn_3dobj::descriptorExtractor::getDeviceType()

Get device type information for feature extraction.

loadNet()#

void cv::cnn_3dobj::descriptorExtractor::loadNet(
const String & model_file,
const String & trained_file,
const String & mean_file = “” )

Initiate a classification structure, the net work parameter is stored in model_file, the network structure is stored in trained_file, you can decide whether to use mean images or not.

Parameters

  • model_file — Path of caffemodel which including all parameters in CNN.

  • trained_file — Path of prototxt which defining the structure of CNN.

  • mean_file — Path of mean file(option).

setDeviceId()#

void cv::cnn_3dobj::descriptorExtractor::setDeviceId(const int & device_id)

Set device ID information for feature extraction. Useful to change device without the need to reload the net. Only used for GPU.

Parameters

  • device_id — ID of GPU.

setDeviceType()#

void cv::cnn_3dobj::descriptorExtractor::setDeviceType(const String & device_type)

Set device type information for feature extraction. Useful to change device without the need to reload the net.

Parameters

  • device_type — CPU or GPU.

Member Data Documentation#

convnet#

caffe::Net< float > * cv::cnn_3dobj::descriptorExtractor::convnet

deviceId#

int cv::cnn_3dobj::descriptorExtractor::deviceId

deviceType#

String cv::cnn_3dobj::descriptorExtractor::deviceType

input_geometry#

cv::Size cv::cnn_3dobj::descriptorExtractor::input_geometry

mean_#

cv::Mat cv::cnn_3dobj::descriptorExtractor::mean_

net_ready#

int cv::cnn_3dobj::descriptorExtractor::net_ready

net_set#

bool cv::cnn_3dobj::descriptorExtractor::net_set

num_channels#

int cv::cnn_3dobj::descriptorExtractor::num_channels

Source file#

The documentation for this class was generated from the following file:

  • opencv2/cnn_3dobj.hpp

Class cv::cnn_3dobj::icoSphere#

Icosohedron based camera view data generator. The class create some sphere views of camera towards a 3D object meshed from .ply files [140] .

#include <opencv2/cnn_3dobj.hpp>

Detailed Description#

Icosohedron based camera view data generator. The class create some sphere views of camera towards a 3D object meshed from .ply files [140] .

Constructor & Destructor Documentation#

icoSphere()#

cv::cnn_3dobj::icoSphere::icoSphere(
float radius_in,
int depth_in )

Generating a sphere by mean of a iteration based points selection process.

Parameters

  • radius_in — Another radius used for adjusting the view distance.

  • depth_in — Number of interations for increasing the points on sphere.

Member Function Documentation#

add()#

void cv::cnn_3dobj::icoSphere::add(float[] v)

Add a new view point.

norm()#

void cv::cnn_3dobj::icoSphere::norm(float[] v)

Make all view points having the same distance from the focal point used by the camera view.

subdivide()#

void cv::cnn_3dobj::icoSphere::subdivide(
float[] v1,
float[] v2,
float[] v3,
int depth )

Generate new view points from all triangles.

getCenter()#

cv::Point3d cv::cnn_3dobj::icoSphere::getCenter(cv::Mat cloud)

Get the center of points on surface in .ply model.

Parameters

  • cloud — Point cloud used for computing the center point.

getRadius()#

float cv::cnn_3dobj::icoSphere::getRadius(
cv::Mat cloud,
cv::Point3d center )

Get the proper camera radius from the view point to the center of model.

Parameters

  • cloud — Point cloud used for computing the center point.

  • center — center point of the point cloud.

createHeader()#

static void cv::cnn_3dobj::icoSphere::createHeader(
int num_item,
int rows,
int cols,
const char * headerPath )

Create header in binary files collecting the image data and label.

Parameters

  • num_item — Number of items.

  • rows — Rows of a single sample image.

  • cols — Columns of a single sample image.

  • headerPath — Path where the header will be stored.

swapEndian()#

static int cv::cnn_3dobj::icoSphere::swapEndian(int val)

Suit the position of bytes in 4 byte data structure for particular system.

writeBinaryfile()#

static void cv::cnn_3dobj::icoSphere::writeBinaryfile(
String filenameImg,
const char * binaryPath,
const char * headerPath,
int num_item,
int label_class,
int x,
int y,
int z,
int isrgb )

Write binary files used for training in other open source project including Caffe.

Parameters

  • filenameImg — Path which including a set of images.

  • binaryPath — Path which will output a binary file.

  • headerPath — Path which header belongs to.

  • num_item — Number of samples.

  • label_class — Class label of the sample.

  • x — Pose label of X.

  • y — Pose label of Y.

  • z — Pose label of Z.

  • isrgb — Option for choice of using RGB images or not.

Member Data Documentation#

CameraPos_temp#

std::vector< cv::Point3d > cv::cnn_3dobj::icoSphere::CameraPos_temp

Temp camera position for duplex position elimination.

diff#

float cv::cnn_3dobj::icoSphere::diff

A threshold for the dupicated points elimination.

X#

float cv::cnn_3dobj::icoSphere::X

X position of one base point on the initial Icosohedron sphere, Y is set to be 0 as default.

Z#

float cv::cnn_3dobj::icoSphere::Z

Z position of one base point on the initial Icosohedron sphere.

CameraPos#

std::vector< cv::Point3d > cv::cnn_3dobj::icoSphere::CameraPos

Camera position on the sphere after duplicated points elimination.

Source file#

The documentation for this class was generated from the following file:

  • opencv2/cnn_3dobj.hpp