OpenCV  4.0.0-alpha
Open Source Computer Vision
Interactive Visual Debugging of Computer Vision applications

What is the most common way to debug computer vision applications? Usually the answer is temporary, hacked together, custom code that must be removed from the code for release compilation.

In this tutorial we will show how to use the visual debugging features of the cvv module (opencv2/cvv.hpp) instead.

Goals

In this tutorial you will learn how to:

Code

The example code

If the program is compiled without visual debugging (see CMakeLists.txt below) the only result is some information printed to the command line. We want to demonstrate how much debugging or development functionality is added by just a few lines of cvv commands.

1 // system includes
2 #include <iostream>
3 
4 // library includes
5 #include <opencv2/imgproc.hpp>
6 #include <opencv2/features2d.hpp>
8 #include <opencv2/videoio.hpp>
10 
11 #define CVVISUAL_DEBUGMODE
14 #include <opencv2/cvv/filter.hpp>
15 #include <opencv2/cvv/dmatch.hpp>
17 
18 using namespace std;
19 using namespace cv;
20 
21 template<class T> std::string toString(const T& p_arg)
22 {
23  std::stringstream ss;
24 
25  ss << p_arg;
26 
27  return ss.str();
28 }
29 
30 
31 void
32 usage()
33 {
34  printf("usage: cvv_demo [-r WxH]\n");
35  printf("-h print this help\n");
36  printf("-r WxH change resolution to width W and height H\n");
37 }
38 
39 
40 int
41 main(int argc, char** argv)
42 {
43  cv::Size* resolution = nullptr;
44 
45  // parser keys
46  const char *keys =
47  "{ help h usage ? | | show this message }"
48  "{ resolution r |0x0| resolution to width and height in the format WxH }";
49  CommandLineParser parser(argc, argv, keys);
50  string res(parser.get<string>("resolution"));
51  if (parser.has("help")) {
52  usage();
53  return 0;
54  }
55  if (res != "0x0") {
56  char dummych;
57  resolution = new cv::Size();
58  if (sscanf(res.c_str(), "%d%c%d", &resolution->width, &dummych, &resolution->height) != 3) {
59  cout << res << " not a valid resolution" << endl;
60  return 1;
61  }
62  }
63 
64  // setup video capture
65  cv::VideoCapture capture(0);
66  if (!capture.isOpened()) {
67  std::cout << "Could not open VideoCapture" << std::endl;
68  return 3;
69  }
70 
71  if (resolution) {
72  printf("Setting resolution to %dx%d\n", resolution->width, resolution->height);
73  capture.set(CV_CAP_PROP_FRAME_WIDTH, resolution->width);
74  capture.set(CV_CAP_PROP_FRAME_HEIGHT, resolution->height);
75  delete resolution;
76  }
77 
78 
79  cv::Mat prevImgGray;
80  std::vector<cv::KeyPoint> prevKeypoints;
81  cv::Mat prevDescriptors;
82 
83  int maxFeatureCount = 500;
84  Ptr<ORB> detector = ORB::create(maxFeatureCount);
85 
87 
88  for (int imgId = 0; imgId < 10; imgId++) {
89  // capture a frame
90  cv::Mat imgRead;
91  capture >> imgRead;
92  printf("%d: image captured\n", imgId);
93 
94  std::string imgIdString{"imgRead"};
95  imgIdString += toString(imgId);
96  cvv::showImage(imgRead, CVVISUAL_LOCATION, imgIdString.c_str());
97 
98  // convert to grayscale
99  cv::Mat imgGray;
100  cv::cvtColor(imgRead, imgGray, CV_BGR2GRAY);
101  cvv::debugFilter(imgRead, imgGray, CVVISUAL_LOCATION, "to gray");
102 
103  // detect ORB features
104  std::vector<cv::KeyPoint> keypoints;
105  cv::Mat descriptors;
106  detector->detectAndCompute(imgGray, cv::noArray(), keypoints, descriptors);
107  printf("%d: detected %zd keypoints\n", imgId, keypoints.size());
108 
109  // match them to previous image (if available)
110  if (!prevImgGray.empty()) {
111  std::vector<cv::DMatch> matches;
112  matcher.match(prevDescriptors, descriptors, matches);
113  printf("%d: all matches size=%zd\n", imgId, matches.size());
114  std::string allMatchIdString{"all matches "};
115  allMatchIdString += toString(imgId-1) + "<->" + toString(imgId);
116  cvv::debugDMatch(prevImgGray, prevKeypoints, imgGray, keypoints, matches, CVVISUAL_LOCATION, allMatchIdString.c_str());
117 
118  // remove worst (as defined by match distance) bestRatio quantile
119  double bestRatio = 0.8;
120  std::sort(matches.begin(), matches.end());
121  matches.resize(int(bestRatio * matches.size()));
122  printf("%d: best matches size=%zd\n", imgId, matches.size());
123  std::string bestMatchIdString{"best " + toString(bestRatio) + " matches "};
124  bestMatchIdString += toString(imgId-1) + "<->" + toString(imgId);
125  cvv::debugDMatch(prevImgGray, prevKeypoints, imgGray, keypoints, matches, CVVISUAL_LOCATION, bestMatchIdString.c_str());
126  }
127 
128  prevImgGray = imgGray;
129  prevKeypoints = keypoints;
130  prevDescriptors = descriptors;
131  }
132 
133  cvv::finalShow();
134 
135  return 0;
136 }
void sort(InputArray src, OutputArray dst, int flags)
Sorts each row or each column of a matrix.
void cvtColor(InputArray src, OutputArray dst, int code, int dstCn=0)
Converts an image from one color space to another.
Definition: videoio_c.h:170
Definition: videoio_c.h:169
Definition: base.hpp:199
std::shared_ptr< _Tp > Ptr
Definition: cvstd_wrapper.hpp:23
_Tp width
the width
Definition: types.hpp:339
STL namespace.
Definition: affine.hpp:51
static void showImage(cv::InputArray img, impl::CallMetaData metaData=impl::CallMetaData(), const char *description=nullptr, const char *view=nullptr)
Add a single image to debug GUI (similar to imshow <>).
Definition: show_image.hpp:38
Designed for command line parsing.
Definition: utility.hpp:806
Class for video capturing from video files, image sequences or cameras.
Definition: videoio.hpp:608
Template class for specifying the size of an image or rectangle.
Definition: types.hpp:315
void finalShow()
Passes the control to the debug-window for a last time.
Definition: final_show.hpp:23
static std::string toString(const MatShape &shape, const String &name="")
Definition: shape_utils.hpp:177
Brute-force descriptor matcher.
Definition: features2d.hpp:1078
InputOutputArray noArray()
static void debugFilter(cv::InputArray original, cv::InputArray result, impl::CallMetaData metaData=impl::CallMetaData(), const char *description=nullptr, const char *view=nullptr)
Use the debug-framework to compare two images (from which the second is intended to be the result of ...
Definition: filter.hpp:36
static void debugDMatch(cv::InputArray img1, std::vector< cv::KeyPoint > keypoints1, cv::InputArray img2, std::vector< cv::KeyPoint > keypoints2, std::vector< cv::DMatch > matches, const impl::CallMetaData &data, const char *description=nullptr, const char *view=nullptr, bool useTrainDescriptor=true)
Add a filled in DMatch <dmatch> to debug GUI.
Definition: dmatch.hpp:49
_Tp height
the height
Definition: types.hpp:340
Size2i Size
Definition: types.hpp:347
Definition: types_c.h:121
n-dimensional dense array class
Definition: mat.hpp:789
#define CVVISUAL_LOCATION
Creates an instance of CallMetaData with the location of the macro as value.
Definition: call_meta_data.hpp:65
bool empty() const
Returns true if the array has no elements.
cmake_minimum_required(VERSION 2.8)
project(cvvisual_test)
SET(CMAKE_PREFIX_PATH ~/software/opencv/install)
SET(CMAKE_CXX_COMPILER "g++-4.8")
SET(CMAKE_CXX_FLAGS "-std=c++11 -O2 -pthread -Wall -Werror")
# (un)set: cmake -DCVV_DEBUG_MODE=OFF ..
OPTION(CVV_DEBUG_MODE "cvvisual-debug-mode" ON)
if(CVV_DEBUG_MODE MATCHES ON)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DCVVISUAL_DEBUGMODE")
endif()
FIND_PACKAGE(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
add_executable(cvvt main.cpp)
target_link_libraries(cvvt
opencv_core opencv_videoio opencv_imgproc opencv_features2d
opencv_cvv
)

Explanation

  1. We compile the program either using the above CmakeLists.txt with Option CVV_DEBUG_MODE=ON (cmake -DCVV_DEBUG_MODE=ON) or by adding the corresponding define CVVISUAL_DEBUGMODE to our compiler (e.g. g++ -DCVVISUAL_DEBUGMODE).
  2. The first cvv call simply shows the image (similar to imshow) with the imgIdString as comment.

    cvv::showImage(imgRead, CVVISUAL_LOCATION, imgIdString.c_str());

    The image is added to the overview tab in the visual debug GUI and the cvv call blocks.

    01_overview_single.jpg
    image

    The image can then be selected and viewed

    02_single_image_view.jpg
    image

    Whenever you want to continue in the code, i.e. unblock the cvv call, you can either continue until the next cvv call (Step), continue until the last cvv call (*>>*) or run the application until it exists (Close).

    We decide to press the green Step button.

  3. The next cvv calls are used to debug all kinds of filter operations, i.e. operations that take a picture as input and return a picture as output.

    cvv::debugFilter(imgRead, imgGray, CVVISUAL_LOCATION, "to gray");

    As with every cvv call, you first end up in the overview.

    03_overview_two.jpg
    image

    We decide not to care about the conversion to gray scale and press Step.

    cvv::debugFilter(imgGray, imgGraySmooth, CVVISUAL_LOCATION, "smoothed");

    If you open the filter call, you will end up in the so called "DefaultFilterView". Both images are shown next to each other and you can (synchronized) zoom into them.

    04_default_filter_view.jpg
    image

    When you go to very high zoom levels, each pixel is annotated with its numeric values.

    05_default_filter_view_high_zoom.jpg
    image

    We press Step twice and have a look at the dilated image.

    cvv::debugFilter(imgEdges, imgEdgesDilated, CVVISUAL_LOCATION, "dilated edges");

    The DefaultFilterView showing both images

    06_default_filter_view_edges.jpg
    image

    Now we use the View selector in the top right and select the "DualFilterView". We select "Changed Pixels" as filter and apply it (middle image).

    07_dual_filter_view_edges.jpg
    image

    After we had a close look at these images, perhaps using different views, filters or other GUI features, we decide to let the program run through. Therefore we press the yellow *>>* button.

    The program will block at

    and display the overview with everything that was passed to cvv in the meantime.

    08_overview_all.jpg
    image
  4. The cvv debugDMatch call is used in a situation where there are two images each with a set of descriptors that are matched to each other.

    We pass both images, both sets of keypoints and their matching to the visual debug module.

    cvv::debugDMatch(prevImgGray, prevKeypoints, imgGray, keypoints, matches, CVVISUAL_LOCATION, allMatchIdString.c_str());

    Since we want to have a look at matches, we use the filter capabilities (*#type match*) in the overview to only show match calls.

    09_overview_filtered_type_match.jpg
    image

    We want to have a closer look at one of them, e.g. to tune our parameters that use the matching. The view has various settings how to display keypoints and matches. Furthermore, there is a mouseover tooltip.

    10_line_match_view.jpg
    image

    We see (visual debugging!) that there are many bad matches. We decide that only 70% of the matches should be shown - those 70% with the lowest match distance.

    11_line_match_view_portion_selector.jpg
    image

    Having successfully reduced the visual distraction, we want to see more clearly what changed between the two images. We select the "TranslationMatchView" that shows to where the keypoint was matched in a different way.

    12_translation_match_view_portion_selector.jpg
    image

    It is easy to see that the cup was moved to the left during the two images.

    Although, cvv is all about interactively seeing the computer vision bugs, this is complemented by a "RawView" that allows to have a look at the underlying numeric data.

    13_raw_view.jpg
    image
  5. There are many more useful features contained in the cvv GUI. For instance, one can group the overview tab.

    14_overview_group_by_line.jpg
    image

    Result

Enjoy computer vision!