Real-time parallel computation of disparity and optical flow using phase difference

A phase-difference-based algorithm for disparity and optical flow estimation is implemented on a TI-C40-based parallel DSP system. The module performs real-time computation of disparity maps on images of size 128 × 128 pixels and computation of optical flows on images of size 64 × 64 pixels. This paper describes the algorithm and its parallel implementation. Processing times required for the computation of disparity maps and velocity fields and measures of the algorithm's performance are reported in detail.     

FPGA based disparity map computation with vergence control

Depth estimation in a scene using image pairs acquired by a stereo camera setup, is one of the important tasks of stereo vision systems. The disparity between the stereo images allows for 3D information acquisition which is indispensable in many machine vision applications. Practical stereo vision systems involve wide ranges of disparity levels. Considering that disparity map extraction of an image is a computationally demanding task, practical real-time FPGA based algorithms require increased device utilization resource usage, depending on the disparity levels operational range, which leads to significant power consumption. In this paper a new hardware-efficient real-time disparity map computation module is developed. The module constantly estimates the precisely required range of disparity levels upon a given stereo image set, maintaining this range as low as possible by verging the stereo setup cameras axes. This enables a parallel-pipelined design, for the overall module, realized on a single FPGA device of the Altera Stratix IV family. Accurate disparity maps are computed at a rate of more than 320 frames per second, for a stereo image pair of 640 × 480 pixels spatial resolution with a disparity range of 80 pixels. The presented technique provides very good processing speed at the expense of accuracy, with very good scalability in terms of disparity levels. The proposed method enables a suitable module delivering high performance in real-time stereo vision applications, where space and power are significant concerns.     

Real-time computation of disparity for hand-pair gesture recognition using a stereo webcam

This paper presents an algorithm for the real-time computation of disparity using video stereo images captured by a stereo webcam. This algorithm is designed to provide both real-time throughput and robust disparity estimation for real-world applications where computation is limited to a pre-defined region-of-interest (ROI). More specifically, this algorithm is used as part of a hand-pair gesture recognition application where the disparity is computed for two ROI around a hand-pair identified by the segmentation component of the recognition application. The developed algorithm provides the required relative difference in disparity with background at high frame rates for the hand-pair gesture recognition application. The results obtained with an inexpensive commercial VGA stereo webcam show a robust disparity computation of 20?ms/frame enabling real-time hand-pair gesture recognition at 25?fps with >90% recognition rate for a maximum hand speed of 40?cm/s and for hand distances between 30 and 150?cm away from the camera.     

A real-time fuzzy hardware structure for disparity map computation

Stereo images acquired by a stereo camera setup provide depth estimation of a scene. Numerous machine vision applications deal with retrieval of 3D information. Disparity map recovery from a stereo image pair involves computationally complex algorithms. Previous methods of disparity map computation are mainly restricted to software-based techniques on general-purpose architectures, presenting relatively high execution time. In this paper, a new hardware-implemented real-time disparity map computation module is realized. This enables a hardware-based fuzzy inference system parallel-pipelined design, for the overall module, implemented on a single FPGA device with a typical operating frequency of 138 MHz. This provides accurate disparity map computation at a rate of nearly 440 frames per second, given a stereo image pair with a disparity range of 80 pixels and 640 × 480 pixels spatial resolution. The proposed method allows a fast disparity map computational module to be built, enabling a suitable module for real-time stereo vision applications.     

Scalable hardware architecture for disparity map computation and object location in real-time

We present the disparity map computation core of a hardware system for isolating foreground objects in stereoscopic video streams. The operation is based on the computation of dense disparity maps using block-matching algorithms and two well-known metrics: sum of absolute differences and Census transform. Two sets of disparity maps are computed by taking each of the images as reference so that a consistency check can be performed to identify occluded pixels and eliminate spurious foreground pixels. Taking advantage of parallelism, the proposed architecture is highly scalable and provides numerous degrees of adjustment to different application needs, performance levels and resource usage. A version of the system for 640 × 480 images and a maximum disparity of 135 pixels was implemented in a system based on a Xilinx Virtex II-Pro FPGA and two cameras with a frame rate of 25 fps (less than the maximum supported frame rate of 40 fps on this platform). Implementation of the same system on a Virtex-5 FPGA is estimated to achieve 80 fps, while a version with increased parallelism is estimated to run at 140 fps (which corresponds to the calculation of more than 5.9 × 10⁹ disparity-pixels per second).     

Adaptive disparity computation using local and non-local cost aggregations

A new method is proposed to adaptively compute the disparity of stereo matching by choosing one of the alternative disparities from local and non-local disparity maps. The initial two disparity maps can be obtained from state-of-the-art local and non-local stereo algorithms. Then, the more reasonable disparity is selected. We propose two strategies to select the disparity. One is based on the magnitude of the gradient in the left image, which is simple and fast. The other utilizes the fusion move to combine the two proposal labelings (disparity maps) in a theoretically sound manner, which is more accurate. Finally, we propose a texture-based sub-pixel refinement to refine the disparity map. Experimental results using Middlebury datasets demonstrate that the two proposed selection strategies both perform better than individual local or non-local algorithms. Moreover, the proposed method is compatible with many local and non-local algorithms that are widely used in stereo matching.     

Real-time rear obstacle detection using reliable disparity for driver assistance

A vision based real-time rear obstacle detection system is one of the most essential technologies, which can be used in many applications such as a parking assistance systems and intelligent vehicles. Although disparity is a useful feature for detecting obstacles, estimating a correct disparity map is a hard problem due to the matching ambiguity and noise sensitivity, especially in homogeneous regions. To overcome these problems, we leverage reliable disparities only for obstacle detection. A reliability factor is introduced to measure an inhomogeneity of the regions quantitatively. It is computed at each superpixel to consider the noise sensitivity of pixel-wise gradients and to assign similar reliability value within a same object. It includes two major components: firstly, In a feature extraction and combining stage, we extract three features from stereo images such as disparity, superpixel segments and pixel-wise gradient and compute the reliability of disparity from superpixel segments and the pixel-wise gradient. Secondly, In an obstacle detection stage, a disparity feature with reliability votes for localizing obstacles and dominant candidates in voting map are selected as initial obstacle region. The initial obstacle regions are expanded into their neighbor superpixels based on CIELAB color similarity and distance similarity between superpixels. Experimental results show satisfactory performance under various real parking environments. Its detection rate is at least 4% higher than those of other existing methods, and its false detection rate is more than 10% lower and thus, can be used for parking assistance system.     

Stereo matching using iterative reliable disparity map expansion in the color–spatial–disparity space

In this paper, we propose a new stereo matching algorithm using an iterated graph cuts and mean shift filtering technique. Our algorithm estimates the disparity map progressively through the following two steps. In the first step, with a previously estimated RDM (reliable disparity map) that consists of sparse ground control points, an updated dense disparity map is constructed through a RDM constrained energy minimization framework that can cope with occlusion. The graph cuts technique is employed for the solution of the proposed energy model. In the second step, more accurate and denser RDM is estimated through the disparity crosschecking technique and the mean shift filtering in the CSD (color–spatial–disparity) space. The proposed algorithm expands the reliable disparities in RDM repeatedly through the above two steps until it converges. Experimental results on the standard data set demonstrate that the proposed algorithm achieves comparable performance to the state-of-the-arts, and gives excellent results especially in the areas such as the disparity discontinuous boundaries and occluded regions, where the conventional methods usually suffer.     

Geometric constraints and stereo disparity computation

Most stereo techniques compute disparity assuming that it varies slowly along surfaces. We quantify and justify this assumption, using weak assumptions about surface orientation distributions in the world to derive the density of disparity surface orientations. The small disparity change assumption is justified by the orientation density's heavy bias toward disparity surfaces that are nearly parallel to the image plane. In addition, the bias strengthens with smaller baselines, larger focal lengths, and as surfaces move farther from the cameras. To analyze current stereo techniques, we derive three densities from the first density, those of the disparity gradient magnitude, the directional derivative of disparity, and the difference in disparity between neighboring surface points. The latter may be used in Bayesian algorithms computing dense disparity fields. The directional derivative density and the disparity difference density both show that feature-based algorithms should strongly favor small disparity changes, contrary to several well-known algorithms. Finally, we use our original surface orientation density and the gradient magnitude density to derive two new surfaces-from-stereo techniques, techniques combining feature-based matching and surface reconstruction. The first uses the densities to severely restrict the search range for the optimum fit. The second incorporates the surface orientation density into the optimization criteria, producing a Bayesian formulation. Both algorithms are shown to be efficient and effective.The authors would like to thank James Miller for helpful discussions and comments on earlier versions of this paper and Wes Turner for making the camera system work. They would also like to acknowledge the financial support of the National Science Foundation under grant IRI-9217195.     

A hybrid approach for stereo disparity computation

An alternative, hybrid approach for disparity estimation, based on the phase difference technique, is presented. The proposed technique combines the robustness of the matching method with the sub-pixel accuracy of the phase difference approach. A matching between the phases of the left and right signals is introduced in order to allow the phase difference method to work in a reduced disparity range. In this framework, a new criterion to detect signal singularities is proposed. The presented test cases show that the performance of the proposed technique in terms of accuracy and density of the disparity estimates has greatly improved. Received: 24 June 1997 / Accepted: 15 September 1998     

Real-time joint disparity and disparity flow estimation on programmable graphics hardware

Disparity flow depicts the 3D motion of a scene in the disparity space of a given view and can be considered as view-dependent scene flow. A novel algorithm is presented to compute disparity maps and disparity flow maps in an integrated process. Consequently, the disparity flow maps obtained helps to enforce the temporal consistency between disparity maps of adjacent frames. The disparity maps found also provides the spatial correspondence information that can be used to cross-validate disparity flow maps of different views. Two different optimization approaches are integrated in the presented algorithm for searching optimal disparity values and disparity flows. The local winner-take-all approach runs faster, whereas the global dynamic programming based approach produces better results. All major computations are performed in the image space of the given view, leading to an efficient implementation on programmable graphics hardware. Experimental results on captured stereo sequences demonstrate the algorithm’s capability of estimating both 3D depth and 3D motion in real-time. Quantitative performance evaluation using synthetic data with ground truth is also provided.     

Leveraging cost matrix structure for hardware implementation of stereo disparity computation using dynamic programming

Dynamic programming is a powerful method for solving energy minimisation problems in computer vision, for example stereo disparity computations. While it may be desirable to implement this algorithm in hardware to achieve frame-rate processing, a na?¨ve implementation may fail to meet timing requirements. In this paper, the structure of the cost matrix is examined to provide improved methods of hardware implementation. It is noted that by computing cost matrix entries along anti-diagonals instead of rows, the cost matrix entries can be computed in a pipelined architecture. Further, if only a subset of the cost matrix needs to be considered, for example by placing limits on the disparity range (include neglecting negative disparities by assuming rectified images), the resources required to compute the cost matrix in parallel can be reduced. Boundary conditions required to allow computing a subset of the cost matrix are detailed. Finally, a hardware solution of Cox’s maximum-likelihood, dynamic programming stereo disparity algorithm is implemented to demonstrate the performance achieved. The design provides high frame rate (>123 fps) estimates for a large disparity range (e.g. 128 pixels), for image sizes of 640 × 480 pixels, and can be simply extended to work well over 200 fps.     

Accelerating computation of Euclidean distance map using the GPU with efficient memory access

Recent graphics processing units (GPUs), which have many processing units, can be used for general purpose parallel computation. To utilise the powerful computing ability, GPUs are widely used for general purpose processing. Since GPUs have very high memory bandwidth, the performance of GPUs greatly depends on memory access. The main contribution of this paper is to present a GPU implementation of computing Euclidean distance map (EDM) with efficient memory access. Given a two-dimensional (2D) binary image, EDM is a 2D array of the same size such that each element stores the Euclidean distance to the nearest black pixel. In the proposed GPU implementation, we have considered many programming issues of the GPU system such as coalesced access of global memory and shared memory bank conflicts, and so on. To be concrete, by transposing 2D arrays, which are temporal data stored in the global memory, with the shared memory, the main access from/to the global memory enables to be performed by coalesced access. In practice, we have implemented our parallel algorithm in the following three modern GPU systems: Tesla C1060, GTX 480 and GTX 580. The experimental results have shown that, for an input binary image with size of 9216 × 9216, our implementation can achieve a speedup factor of 54 over the sequential algorithm implementation.     

Service‐oriented middleware for financial Monte Carlo simulations on the cell broadband engine

Financial Monte Carlo simulations are computationally intensive applications that must meet tight deadlines in terms of job completion times. The completion time might have a huge impact on the financial profits made from decisions derived from the simulation results. Naturally, there is a huge interest in being able to simulate as fast as possible. While single simulations can be done on one machine, decisions often depend on portfolios of simulations. Distributing the workload among resources is crucial to achieve low latency. In this article we present a combination of a middleware with a high‐performance implementation of an Asian options evaluation code on the Cell Broadband Engine (CBE). We handle workload distribution with our PHASTGrid middleware and provide users with a web service interface to the whole infrastructure. The CBE is particularly suitable for Monte Carlo simulations. We implemented a well‐known algorithm on both the CBE and the Intel x86 multicore architectures. Both codes are integrated in our middleware, allowing a direct comparison of the performance and scalability. In addition to the Monte Carlo simulation, we also use different applications and compare our middleware with Globus. Copyright © 2009 John Wiley & Sons, Ltd.     

Relationship between phase and energy methods for disparity computation

The phase and energy methods for computing binocular disparity maps from stereograms are motivated differently, have different physiological relevances, and involve different computational steps. Nevertheless, we demonstrate that at the final stages where disparity values are made explicit, the simplest versions of the two methods are exactly equivalent. The equivalence also holds when the quadrature-pair construction in the energy method is replaced with a more physiologically plausible phase-averaging step. The equivalence fails, however, when the phase-difference receptive field model is replaced by the position-shift model. Additionally, intermediate results from the two methods are always quite distinct. In particular, the energy method generates a distributed disparity representation similar to that found in the visual cortex, while the phase method does not. Finally, more elaborate versions of the two methods are in general not equivalent. We also briefly compare these two methods with some other stereo models in the literature.     

Gauss map computation for free-form surfaces

The Gauss map of a smooth doubly-curved surface characterizes the range of variation of the surface normal as an area on the unit sphere. An algorithm to approximate the Gauss map boundary to any desired accuracy is presented, in the context of a tensor-product polynomial surface patch, r(u,v) for (u,v)[0,1]×[0,1]. Boundary segments of the Gauss map correspond to variations of the normal along the patch boundary or the parabolic lines (loci of vanishing Gaussian curvature) on the surface. To compute the latter, points of vanishing Gaussian curvature are identified with the zero-set of a bivariate polynomial, expressed in the numerically-stable Bernstein basis—the subdivision and variation-diminishing properties then govern an adaptive quadtree decomposition of the (u,v) parameter domain that captures the zero-set of this polynomial to any desired accuracy. Loci on the unit sphere corresponding to the patch boundaries and parabolic lines are trimmed at their mutual or self-intersection points (if any), and the resulting segments are arranged in a graph structure with the segment end-points as nodes. By appropriate traversal of this graph, the Gauss map boundary segments may then be identified in proper order, and extraneous segments (lying in the Gauss map interior) are discarded. The symmetrization of the Gauss map (by identification of antipodal points) and its stereographic projection onto a plane are also discussed.     

Minimum disparity computation via the iteratively reweighted least integrated squares algorithms

Minimum disparity estimation is appealing in that the estimates it provides are simultaneously robust and efficient. This paper presents a family of algorithms called iteratively reweighted least integrated squares for minimum disparity computation. This family of algorithms, indexed by a real parameter α, approximates the disparity measure by quadratic functions, in a form of integrated weighted squared errors, and minimizes the quadratic functions conveniently by using weighted least squares linear regression algorithms. Among all potential values of α, we advocate the use of α=1 from the consideration of robust estimation, which results in an algorithm similar in spirit to the Fisher scoring method for maximum likelihood computation. Numerical studies show that the new algorithms, especially the one that uses α=1, give competitive or better performance over the other algorithms available in the literature.     

On the computation of the temperature distribution at the combustion chamber of a diesel engine using adinat

This paper reports on the computation of the temperature distribution at the surface of the combustion chamber of a direct-injection diesel engine. The aim of these computations with ADINAT is to determine the temperature distribution, perturbated by the presence of measurement sensors, for the numerical calibration of these sensors. In addition, the temperature deviations caused by soot deposits on the surface are investigated; here the ADINAT results are compared to an analytical solution by a Fourier series approach.

Furthermore software aspects are discussed that arise from the relatively large problems analysed here. A software environment for the new program system ADINAT in connection with existing pre- and postprocessors is described, the special requirements of large-scale problems and the resulting necessary modifications in the program are summarized, and finally the performance of ADINAT with respect to computational costs is studied in test computations.     

Real-time map labelling for mobile applications

It is essential to label roads, landmarks, and other important features on maps for mobile applications to help users to understand their location and the environment. This paper aims to examine real-time map labelling methods suitable for the small screen on mobile devices. A slider method with a continuous search space was proposed to sequentially label both line and point features. The method starts with defining a range box for possible locations of the label. Then a search is performed, and the range box is reduced, if there are any cartographic objects that overlap the range box. Finally, the label is placed, at the best possible position in the reduced range box according to normal cartographic preferences, where it does not obscure any cartographic object. We implemented this method in a Java environment using the open source library JTS Topology Suite. A case study showed sound cartographic results of the labelling and acceptable computational efficiency.