OpenCV
Open Source Computer Vision
All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Modules Pages
Using DepthAI Hardware / OAK depth sensors

Table of Contents

Prev Tutorial: Implementing a face beautification algorithm with G-API

Oak-D and Oak-D-Light cameras

Depth sensors compatible with Luxonis DepthAI library are supported through OpenCV Graph API (or G-API) module. RGB image and some other formats of output can be retrieved by using familiar interface of G-API module.

In order to use DepthAI sensor with OpenCV you should do the following preliminary steps:

  1. Install Luxonis DepthAI library depthai-core.
  2. Configure OpenCV with DepthAI library support by setting WITH_OAK flag in CMake. If DepthAI library is found in install folders OpenCV will be built with depthai-core (see a status WITH_OAK in CMake log).
  3. Build OpenCV.

Source code

You can find source code how to process heterogeneous graphs in the modules/gapi/samples/oak_basic_infer.cpp of the OpenCV source code library.

#include <algorithm>
#include <iostream>
#include <sstream>
#include <opencv2/gapi.hpp>
const std::string keys =
"{ h help | | Print this help message }"
"{ detector | | Path to compiled .blob face detector model }"
"{ duration | 100 | Number of frames to pull from camera and run inference on }";
namespace custom {
G_API_NET(FaceDetector, <cv::GMat(cv::GFrame)>, "sample.custom.face-detector");
using GDetections = cv::GArray<cv::Rect>;
using GSize = cv::GOpaque<cv::Size>;
G_API_OP(BBoxes, <GPrims(GDetections)>, "sample.custom.b-boxes") {
static cv::GArrayDesc outMeta(const cv::GArrayDesc &) {
}
};
GAPI_OCV_KERNEL(OCVBBoxes, BBoxes) {
// This kernel converts the rectangles into G-API's
// rendering primitives
static void run(const std::vector<cv::Rect> &in_face_rcs,
std::vector<cv::gapi::wip::draw::Prim> &out_prims) {
out_prims.clear();
const auto cvt = [](const cv::Rect &rc, const cv::Scalar &clr) {
return cv::gapi::wip::draw::Rect(rc, clr, 2);
};
for (auto &&rc : in_face_rcs) {
out_prims.emplace_back(cvt(rc, CV_RGB(0,255,0))); // green
}
}
};
} // namespace custom
int main(int argc, char *argv[]) {
cv::CommandLineParser cmd(argc, argv, keys);
if (cmd.has("help")) {
cmd.printMessage();
return 0;
}
const auto det_name = cmd.get<std::string>("detector");
const auto duration = cmd.get<int>("duration");
if (det_name.empty()) {
std::cerr << "FATAL: path to detection model is not provided for the sample."
<< "Please specify it with --detector options."
<< std::endl;
return 1;
}
// Prepare G-API kernels and networks packages:
auto networks = cv::gapi::networks(detector);
auto args = cv::compile_args(kernels, networks);
// Initialize graph structure
cv::GFrame copy = cv::gapi::oak::copy(in); // NV12 transfered to host + passthrough copy for infer
// infer is not affected by the actual copy here
// FIXME: OAK infer detects faces slightly out of frame bounds
cv::GArray<cv::Rect> rcs = cv::gapi::parseSSD(blob, sz, 0.5f, true, false);
auto rendered = cv::gapi::wip::draw::renderFrame(copy, custom::BBoxes::on(rcs));
// on-the-fly conversion NV12->BGR
auto pipeline = cv::GComputation(cv::GIn(in), cv::GOut(out, rcs))
.compileStreaming(std::move(args));
// Graph execution
pipeline.start();
cv::Mat out_mat;
std::vector<cv::Rect> out_dets;
int frames = 0;
while (pipeline.pull(cv::gout(out_mat, out_dets))) {
std::string name = "oak_infer_frame_" + std::to_string(frames) + ".png";
cv::imwrite(name, out_mat);
if (!out_dets.empty()) {
std::cout << "Got " << out_dets.size() << " detections on frame #" << frames << std::endl;
}
++frames;
if (frames == duration) {
pipeline.stop();
break;
}
}
std::cout << "Pipeline finished. Processed " << frames << " frames" << std::endl;
return 0;
}
Designed for command line parsing.
Definition utility.hpp:890
cv::GArray<T> template class represents a list of objects of class T in the graph.
Definition garray.hpp:366
GComputation class represents a captured computation graph. GComputation objects form boundaries for ...
Definition gcomputation.hpp:121
GAPI_WRAP GStreamingCompiled compileStreaming(GMetaArgs &&in_metas, GCompileArgs &&args={})
Compile the computation for streaming mode.
GFrame class represents an image or media frame in the graph.
Definition gframe.hpp:61
GMat class represents image or tensor data in the graph.
Definition gmat.hpp:68
cv::GOpaque<T> template class represents an object of class T in the graph.
Definition gopaque.hpp:326
void setSource(GRunArgs &&ins)
Specify the input data to GStreamingCompiled for processing, a generic version.
n-dimensional dense array class
Definition mat.hpp:829
Template class for 2D rectangles.
Definition types.hpp:444
Definition infer.hpp:39
#define GAPI_OCV_KERNEL(Name, API)
Definition gcpukernel.hpp:488
#define G_API_OP
Definition gkernel.hpp:369
GKernelPackage kernels()
Create a kernel package object containing kernels and transformations specified in variadic template ...
Definition gkernel.hpp:678
GCompileArgs compile_args(Ts &&... args)
Wraps a list of arguments (a parameter pack) into a vector of compilation arguments (cv::GCompileArg)...
Definition gcommon.hpp:214
GFrame renderFrame(const GFrame &m_frame, const GArray< Prim > &prims)
Renders Media Frame.
GArrayDesc empty_array_desc()
Definition garray.hpp:45
CV_EXPORTS_W bool imwrite(const String &filename, InputArray img, const std::vector< int > &params=std::vector< int >())
Saves an image to a specified file.
#define CV_RGB(r, g, b)
Definition imgproc.hpp:4526
int main(int argc, char *argv[])
Definition highgui_qt.cpp:3
#define G_API_NET(Class, API, Tag)
Definition infer.hpp:452
cv::GKernelPackage kernels()
GFrame copy(const GFrame &in)
cv::gapi::GKernelPackage kernels()
GOpaque< Size > size(const GMat &src)
Gets dimensions from Mat.
cv::GMat BGR(const cv::GFrame &in)
Gets bgr plane from input frame.
IStreamSource::Ptr make_src(Args &&... args)
Definition source.hpp:55
std::tuple< GArray< Rect >, GArray< int > > parseSSD(const GMat &in, const GOpaque< Size > &inSz, const float confidenceThreshold=0.5f, const int filterLabel=-1)
Parses output of SSD network.
Net::Result infer(cv::GOpaque< cv::Rect > roi, T in)
Calculates response for the specified network (template parameter) for the specified region in the so...
Definition infer.hpp:474
cv::GKernelPackage combine(const cv::GKernelPackage &lhs, const cv::GKernelPackage &rhs)
cv::gapi::GNetPackage networks(Args &&... args)
Definition infer.hpp:703
GProtoInputArgs GIn(Ts &&... ts)
Definition gproto.hpp:96
GRunArgsP gout(Ts &... args)
Definition garg.hpp:280
GProtoOutputArgs GOut(Ts &&... ts)
Definition gproto.hpp:101
Definition garray.hpp:39
This structure represents a rectangle to draw.
Definition render_types.hpp:128