Video Capture and Rectification

Abstract 概要: We provide a utility for opening capture device, capturing from it, and rectifying the image. This page introduce the usage of the capture utility.

Table of contents 目次

TCameraInfo Object and Related Functions
Image Rectification
- Setup
- Rectify
Implementing Video Capture and Rectification
Capture Utility
- capture.cpp
- capture_mult.cpp

TCameraInfo Object and Related Functions †

TCameraInfo is a structure to store the capture configuration, defined in ay_vision/vision_util.h:

struct TCameraInfo
{
  std::string Name;   // Camera name
  std::string DevID;  // Video device ID (int) or Video/stream path (string)
  std::string PixelFormat;  // Pixel format such as MJPG
  int Width, Height;  // Image size
  int CapWidth, CapHeight;  // Capture size (if zero, Width/Height is used; if not zero, captured image is resized to (Width,Height))
  int NRotate90;  // Rotating image NRotate90 * 90 degrees
  int Rectification;  // Whether rectify image or not
  double Alpha;     // Scaling factor
  cv::Mat K, D, R;  // Camera, distortion, and rectification matrices
};

DevID may be:

Camera device ID; e.g. "1" (camera device 1, i.e. /dev/video1).
Video stream; e.g. "http://localhost:8080/?action=stream&dummy=file.mjpg" (Use MJPG-Streamer for streaming video).
Video file; e.g. "../data/fv_demo/cam0_0001.m4v".

Some functions are defined to work with TCameraInfo. The important ones are as follows:

↑

ReadFromYAML †

void ReadFromYAML(std::vector<TCameraInfo> &cam_info, const std::string &file_name);

Load a vector of TCameraInfo objects from a YAML file. Each YAML file may contain multiple camera devices.

Examples:
fingervision/config/fv_1.yaml,
fingervision/config/fv_2_l.yaml,
standalone/cam_2.yaml,
standalone/file1.yaml.

↑

CapOpen †

bool CapOpen(TCameraInfo &info, cv::VideoCapture &cap);

Open the capture device, video stream, or file and assign the capture object to cap. It returns true if successfully opened, and false if there is an error.

↑

CapWaitReopen †

bool CapWaitReopen(TCameraInfo &info, cv::VideoCapture &cap, int ms_wait=1000, int max_count=0);

Re-open the capture device (or video stream or file). This is useful when the capture fails. It tries to reopen at most max_count times (or forever if max_count==0). At each trial of reopen, it sleeps ms_wait milliseconds with cv::waitKey. If q or Esc is pressed, it exits the function immediately.

↑

Image Rectification †

TCameraInfo may have camera rectification parameters. TCameraRectifier is a rectifier object defined in ay_vision/vision_util.h. It has two methods:

↑

Setup †

void Setup(const cv::Mat &K, const cv::Mat &D, const cv::Mat &R, const cv::Size &size_in, const double &alpha, const cv::Size &size_out);

Setup the rectifier object. The parameters can be filled with the member variables of a TCameraInfo object.

↑

Rectify †

void Rectify(cv::Mat &frame, const cv::Scalar& border=cv::Scalar());

Rectify the given image (frame) and replace frame by the rectified image.

↑

Implementing Video Capture and Rectification †

The standalone capture program standalone/capture.cpp and multiple camera version standalone/capture_mult.cpp are good examples of the capture utility.

The following codes are part of standalone/capture.cpp.

Load TCameraInfo from a YAML file:

std::vector<TCameraInfo> cam_info;
ReadFromYAML(cam_info, filename);  // filename is a path to YAML file (std::string)

Or we can create TCameraInfo manually:

std::vector<TCameraInfo> cam_info;
cam_info.push_back(TCameraInfo());
cam_info[0].DevID= cam;  // e.g. "0", "http://xxx/yyy.mjpg", "zzz.mjpg"
cam_info[0].Width= 320;  // e.g. 0, 320, 640
cam_info[0].Height= 240;  // e.g. 0, 240, 480

Open the capture device (we just use the first element of cam_info):

cv::VideoCapture cap;
if(!CapOpen(cam_info[0], cap))  return -1;  // Exit if failure.

Setup the rectifier (only when cam_info[0].Rectification is true):

TCameraRectifier cam_rectifier;
if(cam_info[0].Rectification)
{
  cv::Size size_in(cam_info[0].Width,cam_info[0].Height), size_out(cam_info[0].Width,cam_info[0].Height);
  cam_rectifier.Setup(cam_info[0].K, cam_info[0].D, cam_info[0].R, size_in, cam_info[0].Alpha, size_out);
}

Capture from cap, and if failure, try to reopen the camera device with CapWaitReopen:

cv::Mat frame;
while(!cap.read(frame))
{
  if(CapWaitReopen(cam_info[0],cap)) continue;
  else  return cv::Mat();
}

Resize, rotate, and rectify the image:

if(cam_info[0].CapWidth!=cam_info[0].Width || cam_info[0].CapHeight!=cam_info[0].Height)
  cv::resize(frame,frame,cv::Size(cam_info[0].Width,cam_info[0].Height));
Rotate90N(frame,frame,cam_info[0].NRotate90);
if(cam_info[0].Rectification)
  cam_rectifier.Rectify(frame, /*border=*/cv::Scalar(0,0,0));

↑

Capture Utility †

There are two capture utilities that are useful to check the video device and stream. It can also record the video.

standalone/capture.cpp: Capture images from a camera device, video stream, and a video file.
standalone/capture_mult.cpp: Multiple camera version.

↑

capture.cpp †

Compile:

 $ g++ -g -Wall -O2 -o capture.out capture.cpp -I../3rdparty -I../fv_core -lopencv_core -lopencv_imgproc -lopencv_calib3d -lopencv_highgui

Execution:

 $ ./capture.out [CAMERA_NUMBER [WIDTH HEIGHT]]
   CAMERA_NUMBER: Camera device number (e.g. 1).
 $ ./capture.out CONFIG_FILE
   CONFIG_FILE: Configuration file in YAML (e.g. fv_1.yaml).
   The configuration file may include camera configuration (CameraInfo).

Usage:

Press 'q' or Esc: Exit the program.
Press 'W': On/off video capture.

Example execution commands:

Open /dev/video0 with 320x240:

 $ ./capture.out 0 320 240

Open a video stream from localhost port 8080. cf. MJPG-Streamer

 $ ./capture.out "http://localhost:8080/?action=stream&dummy=file.mjpg"

Open a video device with the configuration file fv_1.yaml (image rectification is enabled):

 $ ./capture.out fv_1.yaml

↑

capture_mult.cpp †

Compile:

 g++ -g -Wall -O2 -o capture_mult.out capture_mult.cpp -I../3rdparty -I../fv_core -lopencv_core -lopencv_imgproc -lopencv_calib3d -lopencv_highgui

Execution:

 $ ./capture_mult.out [CAMERA_NUMBER [WIDTH HEIGHT [CAMERA_NUMBER [WIDTH HEIGHT ...]]]]
   CAMERA_NUMBER: Camera device number (e.g. 1).
   To capture from multiple cameras, repeat like:
   $ ./capture_mult.out 0 320 240 1 320 240
 $ ./capture_mult.out CONFIG_FILE
   CONFIG_FILE: Configuration file in YAML (e.g. fv_1.yaml).
   The configuration file may include camera configuration (CameraInfo).
   Put configurations of multiple cameras in the same CONFIG_FILE
   to capture from multiple cameras (e.g. cam_2.yaml).

Usage:

Press 'q' or Esc: Exit the program.
Press 'W': On/off video capture.