Software of FingerVision

Overview †

System Architecture †

The system architecture of FingerVision has four kinds; two connection types, and two interface types. The connection types are direct USB connection and streaming video over Ethernet network. The interface types are standalone and ROS (robot operating system). These four kinds are illustrated in the following figure.

The difference of the connection types is as follows:

• Direct USB connection: Directly plug in the FingerVision camera to the USB port of the PC where the image processing is done.
  • Pros: Simple setup. Maybe the fastest way.
  • Cons: When sharing the same video among different FingerVision-processing programs (plugins), the programming becomes complicated.
• Streaming video over Ethernet network: Using another computer such as Raspberry Pi to stream the video from the FingerVision camera to the Ethernet network. The PC receives the video data through the Ethernet network and processes it.
  • Pros: Easy to share the video among different FingerVision-processing programs (plugins). Ethernet network might be better for the system integration.
  • Cons: We need another computer, which increases the complexity of the system.

In general, we recommend the streaming type in order to take advantage of FingerVision multimodality; we can increase the functionality of FingerVision by adding other video processing plugins.

The difference of the interface types is as follows:

• Standalone: Just processing the video data from FingerVision.
  • Pros: The program is simple, which is good for introducing it into other system.
  • Cons: Not suite for integration without extra programming.
• ROS: The processing program is implemented as a ROS node, and the processed result is sent as ROS topics.
  • Pros: Integrating with other programs (ROS nodes) is easy. We can use the program as is. We can also use the FingerVision-based manipulation programs implemented as ROS programs.
  • Cons: We need ROS.

Recently ROS is becoming much popular as a standard platform of robot programming. ROS version would be a good starting point. On the other side, if you wish to use FingerVision on other middleware platform, starting from the standalone programs would be better.

FingerVision Video Processing Program †

Currently we have two core computer vision programs for FingerVision video processing:

• Marker tracking: Tracking the markers (black dots) on the surface of FingerVision for estimating the force distribution.
• Proximity vision: Detecting an object, estimating the object pose (position and orientation on image), and detecting the object movement (which is caused by slippage or deformation).

As we described in the previous section, there are standalone and ROS versions of FingerVision video processing programs. The following figure breaks down the processes.

There are two standalone programs, marker tracking (blob_tracker2_test) and proximity vision (prox_vision_test). In the ROS program (fv_core_node), they are unified.

Among the standalone and ROS programs, many of the components are common, such as the camera rectification (Rectify), except for publishing ROS topics (and providing ROS services). Thus,

• For understanding FingerVision processing programs, reading the codes of standalone version is a good starting point.
• For making a new FingerVision processing program (plugin), starting as a standalone program would be a good practice.

Another remark is that a filter script (FV-Filter) is also provided for the ROS version. The marker tracker and the proximity vision are designed to be general; the output of them is a kind of low-level values. On the other side, FV-Filter converts these values to ones that are easy to use in control. For example, the marker movements are converted to the force estimate.

Source Code †

The source code of the standalone and ROS programs is available on GitHub:
https://github.com/akihikoy/fingervision

Note

The repository name optskin was changed to fingervision on Aug 09, 2018, and largely modified for better quality.

There are directories:

• fv_core: Core programs of processing FingerVision data, that are shared among the standalone programs and the ROS package.

• standalone: Standalone programs of processing FingerVision data.

• fingervision: The main ROS package to process data of FingerVision.

• fingervision_msgs: The messages and services used in fingervision.

• 3rdparty: Utility programs that are copied from other projects.

• cad: CAD models of FingerVision.

• tools: Useful tools, such as configuring a USB camera device, and a helper of MJPG-Streamer.

Reference †

• Installation
  • OpenCV Version
  • Installation
• Utility
  • Video capture and rectification
  • MJPG-Streamer: Streaming video over the Ethernet network.
  • Raspberry Pi Setup: Setting up Raspberry Pi with MJPG-Streamer.
• Marker tracking (algorithm and standalone program)
• Proximity vision (algorithm and standalone program)
• ROS package
• FV-Filter (ROS)
• Data
  • Dummy FingerVision Data

Tutorial †

• Writing your own FV plugin (standalone)
• Writing your own FV plugin (ROS)