A real-time fuzzy hardware structure for disparity map computation

Stereo images acquired by a stereo camera setup provide depth estimation of a scene. Numerous machine vision applications deal with retrieval of 3D information. Disparity map recovery from a stereo image pair involves computationally complex algorithms. Previous methods of disparity map computation are mainly restricted to software-based techniques on general-purpose architectures, presenting relatively high execution time. In this paper, a new hardware-implemented real-time disparity map computation module is realized. This enables a hardware-based fuzzy inference system parallel-pipelined design, for the overall module, implemented on a single FPGA device with a typical operating frequency of 138 MHz. This provides accurate disparity map computation at a rate of nearly 440 frames per second, given a stereo image pair with a disparity range of 80 pixels and 640 × 480 pixels spatial resolution. The proposed method allows a fast disparity map computational module to be built, enabling a suitable module for real-time stereo vision applications.     

A high-performance fully reconfigurable FPGA-based 2D convolution processor

This paper presents a new fully reconfigurable 2D convolver designed for FPGA-based image and video processors. The proposed architecture operates on image pixels coded with different bit resolutions and varying kernel weights avoiding power and time-consuming reconfiguration. This is made possible by using new SIMD arithmetic modules purposely designed for the new circuit. When optimized for the XILINX VIRTEX device family, the convolver presented in this work requires just 18.4 ms to perform a 5×5 convolution on a 1024×1024 8-bit pixels image and dissipates only 102.1 mW/MHz.The new circuit can be exploited in all the real-time applications in which adaptive convolutions are required and it can be realized also in many other FPGA device families.     

A real-time global stereo-matching on FPGA

An improved global stereo matching algorithm is implemented on a single FPGA for real-time applications. Stereo matching is widely used in stereo vision systems, i.e. objects detection and autonomous vehicles. Global algorithms have much more accurate results than local algorithms, but global algorithms are not implemented on FPGA since they rely over high-end hardware resources. In this implementation the stereo pairs are divided into blocks, the hardware resources are reduced by processing one block once. The hardware implementation is based on a Xilinx Kintex 7 FPGA. Experiment results show that the proposed implementation has an accurate result for the Middlebury benchmarks and 30 frames per second (fps) @1920 × 1680 is achieved.     

An FPGA-based RGBD imager

This paper describes a trinocular stereo vision system using a single chip of FPGA to generate the composite color (RGB) and disparity data stream at video rate, called the RGBD imager. The system uses the triangular configuration of three cameras for synchronous image capture and the trinocular adaptive cooperative algorithm based on local aggregation for smooth and accurate dense disparity mapping. We design a fine-grain parallel and pipelining architecture in FPGA for implementation to achieve a high computational and real-time throughput. A binary floating-point format is customized for data representation to satisfy the wide data range and high computation precision demands in the disparity calculation. Memory management and data bit-width control are applied in the system to reduce the hardware resource consumption and accelerate the processing speed. The system is able to produce dense disparity maps with 320 × 240 pixels in a disparity search range of 64 pixels at the rate of 30 frames per second.     

Adaptive Census Transform: A novel hardware-oriented stereovision algorithm

This paper presents a new hardware-oriented approach for the extraction of disparity maps from stereo images. The proposed method is based on the herein named Adaptive Census Transform that exploits adaptive support weights during the image transformation; the adaptively weighted sum of SADs is then used as the dissimilarity metric. Quality tests show that the proposed method reaches significantly better accuracy than alternative hardware-oriented approaches. To demonstrate the practical hardware feasibility, a specific architecture has been designed and its implementation has been carried out using a single FPGA chip. Such a VLSI implementation allows a frame rate up to 68 fps to be reached for 640 × 480 stereo images, using just 80,000 slices and 32 RAM blocks of a Virtex6 chip.     

FPGA based real-time on-road stereo vision system

Accurate and Real-Time stereo vision is an essential need for many computer vision applications, such as On-Road stereo vision system. In this paper, a fast and accurate system for On-Road stereo vision application is presented. In order to achieve this purpose, first, an algorithm is presented that is optimized for hardware implementation and On-Road application, then an appropriate hardware architecture for this algorithm is proposed. The approach uses gradient and Census transform as initial cost function, then cost is aggregate in cross-based supported reign. LR-check and disparity refinement is also utilized to improve final disparity. The proposed design is a standalone stereo vision system where all steps of algorithm is implemented on a hardware platform. Overall system is implemented on FPGA platform and it is tested on KITTI Database. The proposed system is also tested on Middlebury database without any changes in parameters. Experimental results show that the proposed system has high accuracy in KITTI and Middlebury database. In term of hardware resource, the system occupy %%66 of XC6VLX240T FPGA device for 1920 × 1080 image with 96 disparity levels that operated at 53.5 Fps.     

An FPGA stereo matching unit based on fuzzy logic

Stereo matching is one of the most used algorithms in real-time image processing applications such as positioning systems for mobile robots, three-dimensional building mapping and recognition, detection and three-dimensional reconstruction of objects. In order to improve the performance, stereo matching algorithms often have been implemented in dedicated hardware such as FPGA or GPU devices. In this paper an FPGA stereo matching unit based on fuzzy logic is described. The proposed algorithm consists of three stages. First, three similarity parameters inherent to each pixel contained in the input stereo pair are computed. Then, the similarity parameters are sent to a fuzzy inference system which determines a fuzzy-similarity value. Finally, the disparity value is defined as the index which maximizes the fuzzy-similarity values (zero up to d_max). Dense disparity maps are computed at a rate of 76 frames per second for input stereo pairs of 1280 × 1024 pixel resolution and a maximum expected disparity equal to 15. The developed FPGA architecture provides reduction of the hardware resource demand compared to other FPGA-based stereo matching algorithms: near to 72.35% for logic units and near to 32.24% for bits of memory. In addition, the developed FPGA architecture increases the processing speed: near to 34.90% pixels per second and outperforms the accuracy of most of real-time stereo matching algorithms in the state of the art.     

FPGA implementation of reversible watermarking in digital images using reversible contrast mapping

Reversible contrast mapping (RCM) and its various modified versions are used extensively in reversible watermarking (RW) to embed secret information into the digital contents. RCM based RW accomplishes a simple integer transform applied on pair of pixels and their least significant bits (LSB) are used for data embedding. It is perfectly invertible even if the LSBs of the transformed pixels are lost during data embedding. RCM offers high embedding rate at relatively low visual distortion (embedding distortion). Moreover, low computation cost and ease of hardware realization make it attractive for real-time implementation. To this aim, this paper proposes a field programmable gate array (FPGA) based very large scale integration (VLSI) architecture of RCM-RW algorithm for digital images that can serve the purpose of media authentication in real-time environment. Two architectures, one for block size (8 × 8) and the other one for (32 × 32) block are developed. The proposed architecture allows a 6-stage pipelining technique to speed up the circuit operation. For a cover image of block size (32 × 32), the proposed architecture requires 9881 slices, 9347 slice flip-flops, 11291 number 4-input LUTs, 3 BRAMs and a data rate of 1.0395 Mbps at an operating frequency as high as 98.76 MHz.     

基于FPGA的双目立体视觉系统

立体视觉的目的之一就是为了获得周围场景的3维信息,其关键在于匹配算法。然而即便是使用目前先进的通用处理器,其计算致密视差图所需的时间仍无法满足高速自主导航的需求。为了解决这个问题,提出了一种基于现场可编程门阵列(FPGA)的双目立体视觉系统的设计方案,同时介绍了系统的硬件结构,并在讨论区域匹配的快速算法的基础上,提出了基于FPGA的像素序列和并行窗口算法框架,用以实现零均值像素灰度差平方和(ZSSD)的匹配算法。该算法是先将视频信号经解码芯片生成场景立体图像对,并由FPGA来完成立体图像对的几何校正和ZSSD匹配算法,然后将获得的致密视差图通过PC I总线发送至上位机。实践表明,该算法效果好、速度快,不仅具有较强的鲁棒性,并且硬件系统性能稳定、可靠。此外,该方案还适用于像素灰度差的绝对值和(SAD)和像素灰度差的平方和(SSD)等多种传统区域匹配算法的快速实现和实时处理。     

一种用于立体图像匹配的改进稀疏匹配算法

立体匹配有着广泛的应用前景,是计算机视觉领域的研究热点。立体匹配是立体视觉中最为关键和困难的一步,它的目标是计算标识匹配像素位置的视差图。文中提出的立体匹配算法基于置信传播（Belief Propagation,BP）。左图像首先经过非均匀采样,得到一个内容自适应的网格近似表示。算法的关键是使用基于置信传播的立体匹配算法,匹配稀疏的左图像和右图像得到稀疏视差图。通过左图像得到网格,稀疏视差图可以经过简单的插值得到稠密视差图。实验结果表明,该方法与现有稀疏立体匹配技术相比在视差图质量上平均有40％的提高。     

Automatically controlled pan–tilt smart camera with FPGA based image analysis system dedicated to real-time tracking of a moving object

This paper presents an intelligent, automatically controlled camera based on visual feedback. The camera housing contains actuators that change the orientation of the camera – enabling a full rotation around the vertical axis (pan) and 90° around the horizontal axis (tilt). A system for acquisition, processing image analysis and a camera driver are implemented in the FPGA Xilinx Spartan-6 device. An original, innovative reconfigurable system architecture has been developed. The FPGA device is connected directly to the eight independently operated SRAM memory banks. A prototype device has been constructed with a real-time tracking algorithm, enabling an automatically control of the position of the camera. The device has been tested indoors and outdoors. The camera is able to keep a tracked object close to the center of its field of view. The power consumption of the control system is 2 W. A reconfigurable part reaches the computing performance of 3200 MOPS.     

A Performance Study on Different Cost Aggregation Approaches Used in Real-Time Stereo Matching

Many vision applications require high-accuracy dense disparity maps in real-time and online. Due to time constraint, most real-time stereo applications rely on local winner-takes-all optimization in the disparity computation process. These local approaches are generally outperformed by offline global optimization based algorithms. However, recent research shows that, through carefully selecting and aggregating the matching costs of neighboring pixels, the disparity maps produced by a local approach can be more accurate than those generated by many global optimization techniques. We are therefore motivated to investigate whether these cost aggregation approaches can be adopted in real-time stereo applications and, if so, how well they perform under the real-time constraint. The evaluation is conducted on a real-time stereo platform, which utilizes the processing power of programmable graphics hardware. Six recent cost aggregation approaches are implemented and optimized for graphics hardware so that real-time speed can be achieved. The performances of these aggregation approaches in terms of both processing speed and result quality are reported.     

融合边缘保持与改进代价聚合的立体匹配算法

目的 立体匹配是计算机双目视觉的重要研究方向,主要分为全局匹配算法与局部匹配算法两类。传统的局部立体匹配算法计算复杂度低,可以满足实时性的需要,但是未能充分利用图像的边缘纹理信息,因此在非遮挡、视差不连续区域的匹配精度欠佳。为此,提出了融合边缘保持与改进代价聚合的立体匹配。方法 首先利用图像的边缘空间信息构建权重矩阵,与灰度差绝对值和梯度代价进行加权融合,形成新的代价计算方式,同时将边缘区域像素点的权重信息与引导滤波的正则化项相结合,并在多分辨率尺度的框架下进行代价聚合。所得结果经过视差计算,得到初始视差图,再通过左右一致性检测、加权中值滤波等视差优化步骤获得最终的视差图。结果 在Middlebury立体匹配平台上进行实验,结果表明,融合边缘权重信息对边缘处像素点的代价量进行了更加有效地区分,能够提升算法在各区域的匹配精度。其中,未加入视差优化步骤的21组扩展图像对的平均误匹配率较改进前减少3.48%,峰值信噪比提升3.57 dB,在标准4幅图中venus上经过视差优化后非遮挡区域的误匹配率仅为0.18%。结论 融合边缘保持的多尺度立体匹配算法有效提升了图像在边缘纹理处的匹配精度,进一步降低了非遮挡区域与视差不连续区域的误匹配率。     

3D visual comfort assessment by measuring the vertical disparity tolerance

Misalignment in stereo images leads to 3D discomfort, but the visual tolerance for disparities varies with viewing environment and stimulus. The aim of the study was twofold: first, to assess if vertical disparity tolerance (VDT) could be a reliable indicator of 3D visual comfort under certain restrained condition when vertical disparity is induced; second, to be able to predict how viewing conditions can affect visual comfort using an analytical model. Two viewing condition parameters were considered: luminance and stimulus angular size. The study was carried out in two experiments involving 17 subjects. In Experiment 1, visual comfort and vertical disparity tolerance were measured by a series of psychophysical tests for different stimulus angular sizes and luminance. Based on a regression analysis of this data, a model was proposed to estimate VDT as a function of luminance and stimulus angular size. In Experiment 2, a validation test was carried out to assess the quality of the model. Results confirm that for given viewing conditions (luminance, angular size, induced vertical disparity), the visual comfort measured is in agreement with the one predicted (ρ = 1.0008, p = 0.0026). VDT is a recognized reliable indicator of visual comfort due to vertical disparity and the model can be used to predict visual comfort for given viewing conditions.     

A CMOS image sensor with optical and potential dual imaging function for on-chip bioscientific applications

An optical and potential dual imaging CMOS sensor for bioscientific applications was proposed and fabricated. The CMOS image sensor has the capability to simultaneously capture optical and on-chip potential images. The sensor is designed with target applications of on-chip DNA (or protein) microarray analysis and on-chip neural imaging. A potential imaging function was implemented onto a CMOS image sensor with a simple pixel circuitry that is compatible with optical image sensor pixels. The basic properties of the potential-sensing pixel were characterized. By choosing an appropriate operating sequence and off-chip configuration, the sensor can be operated in either a wide-range potential imaging mode (>5 V) or a high-resolution potential imaging mode (1.6 mV). The sensor is applicable for most of the target applications and is capable of detecting a pH change in the solution placed on the surface. Two-dimensional optical and potential dual imaging was successfully demonstrated, and the profile of a potential spot smaller than 50 μm was clearly observed.     

Analog VLSI implementation for stereo correspondence between 2-Dimages

Many robotics and navigation systems utilizing stereopsis to determine depth have rigid size and power constraints and require direct physical implementation of the stereo algorithm. The main challenges lie in managing the communication between image sensor and image processor arrays, and in parallelizing the computation to determine stereo correspondence between image pixels in real-time. This paper describes the first comprehensive system level demonstration of a dedicated low-power analog VLSI (very large scale integration) architecture for stereo correspondence suitable for real-time implementation. The inputs to the implemented chip are the ordered pixels from a stereo image pair, and the output is a two-dimensional disparity map. The approach combines biologically inspired silicon modeling with the necessary interfacing options for a complete practical solution that can be built with currently available technology in a compact package. Furthermore, the strategy employed considers multiple factors that may degrade performance, including the spatial correlations in images and the inherent accuracy limitations of analog hardware, and augments the design with countermeasures.     

Experimental combination of intensity and stereo edges for improved snake segmentation

In this paper, we present an algorithm to combine edge information from stereo-derived disparity maps with edges from the original intensity/color image to improve the contour detection in images of natural scenes. After computing the disparity map, we generate a so-called “edge-combination image,” which relies on those edges of the original image that are also present in the stereo map. We describe an algorithm to identify corresponding intensity and disparity edges, which are usually not perfectly aligned due to errors in the stereo reconstruction. Our experiments show that the proposed edge-combination approach can significantly improve the segmentation results of an active contour algorithm. The text was submitted by the authors in English. Danijela Markovic graduated from the Faculty of Electronic Engineering, University of Nis, Serbia in 1997. She is currently a PhD student at the Institute for Software Technology and Interactive Systems, Vienna University of Technology. Her research interests are in computer vision and computer graphics, including stereo vision and curve/surface modeling. Particularly, she is interested in object segmentation, feature extraction, and tracking. Margrit Gelautz received her PhD degree in computer science from Graz University of Technology, Austria. She worked on stereo and interferometric image processing for radar remote sensing applications during a postdoctoral stay at Stanford University. Her current research interests include image and video processing for multimedia applications, with a focus on 3D vision and rendering techniques.     

Stereo vision for planetary rovers: Stochastic modeling to near real-time implementation

NASA scenarios for lunar and planetary missions include robotic vehicles that function in both teleoperated and semi-autonomous modes. Under teleoperation, on-board stereo cameras may provide 3-D scene information to human operators via stereographic displays; likewise, under semi-autonomy, machine stereo vision may provide 3-D information for obstacle avoidance. In the past, the slow speed of machine stereo vision systems has posed a hurdle to the semi-autonomous scenario; however, recent work at JPL and other laboratories has produced stereo systems with high reliability and near real-time performance for low-resolution image pairs. In particular, JPL has taken a significant step by achieving the first autonomous, cross-country robotic traverses (of up to 100 meters) to use stereo vision, with all computing on-board the vehicle. Here, we describe the stereo vision system, including the underlying statistical model and the details of the implementation. The statistical and algorithmic aspects employ random field models of the disparity map, Bayesian formulations of single-scale matching, and area-based image comparisons. The implementation builds bandpass image pyramids and produces disparity maps from the 60×64 level of the pyramids at rates of up to two seconds per image pair. All vision processing is done in one 68020 augmented with Datacube image processing boards. We argue that the overall approach provides a unifying paradigm for practical, domain-independent stereo ranging. We close with a discussion of practical and theoretical issues involved in evaluating and extending the performance of the stereo system.     

Real-Time Visual Sensing for Task Planning in a Field Navigation Vehicle

Some agricultural tasks consist of applying chemical treatment to the crops, but the products are applied throughout the field in most cases. In order to automate these tasks, and to apply the products more accurately, an autonomous vehicle can be used. The vehicle is intended to navigate autonomously through a field where plants are arranged in rows, following the crop rows. Its main sensor is a camera which takes perspective images of the area close to and in front of the vehicle. The vehicle is equipped with a treating device consisting of a bar with 30 spray nozzles. When the plants, which have been identified at some distance in front of the vehicle, reach the position of the treating device, the nozzles which are over a plant are switched on, thus applying the treatment. The typical vehicle speed is 1 m/s and the video frame rate is 10 Hz. A distributed system, consisting of several processors interconnected by serial links, is used, where each module accomplishes a different task: vehicle control, guidance information, image segmentation, map building and nozzle switching. The work explained in this paper is mainly focused on the latter parts of the system, map building and nozzle switching, and stresses the methods used to fit the time requirements. Real-time performance is achieved. Some results and time measurements are given.