Using Multiple-Hypothesis Disparity Maps and Image Velocity for 3-D Motion Estimation

In this paper we explore a multiple hypothesis approach to estimating rigid motion from a moving stereo rig. More precisely, we introduce the use of Gaussian mixtures to model correspondence uncertainties for disparity and image velocity estimation. We show some properties of the disparity space and show how rigid transformations can be represented. An algorithm derived from standard random sampling-based robust estimators, that efficiently estimates rigid transformations from multi-hypothesis disparity maps and velocity fields is given.     

On 3-D scene flow and structure recovery from multiview image sequences

Two novel systems computing dense three-dimensional (3-D) scene flow and structure from multiview image sequences are described in this paper. We do not assume rigidity of the scene motion, thus allowing for nonrigid motion in the scene. The first system, integrated model-based system (IMS), assumes that each small local image region is undergoing 3-D affine motion. Non-linear motion model fitting based on both optical flow constraints and stereo constraints is then carried out on each local region in order to simultaneously estimate 3-D motion correspondences and structure. The second system is based on extended gradient-based system (EGS), a natural extension of two-dimensional (2-D) optical flow computation. In this method, a new hierarchical rule-based stereo matching algorithm is first developed to estimate the initial disparity map. Different available constraints under a multiview camera setup are further investigated and utilized in the proposed motion estimation. We use image segmentation information to adopt and maintain the motion and depth discontinuities. Within the framework for EGS, we present two different formulations for 3-D scene flow and structure computation. One formulation assumes that initial disparity map is accurate, while the other does not. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3-D motion and structure recovery schemes. Empirical comparison between IMS and EGS is also reported.     

Utilization of stereo disparity and optical flow information for the computer analysis of human interactions

We propose a system that simultaneously utilizes the stereo disparity and optical flow information of real-time stereo grayscale multiresolution images for the recognition of objects and gestures in human interactions. For real-time calculation of the disparity and optical flow information of a stereo image, the system first creates pyramid images using a Gaussian filter. The system then determines the disparity and optical flow of a low-density image and extracts attention regions in a high-density image. The three foremost regions are recognized using higher-order local autocorrelation features and linear discriminant analysis. As the recognition method is view based, the system can process the face and hand recognitions simultaneously in real time. The recognition features are independent of parallel translations, so the system can use unstable extractions from stereo depth information. We demonstrate that the system can discriminate the users, monitor the basic movements of the user, smoothly learn an object presented by users, and can communicate with users by hand signs learned in advance. Received: 31 January 2000 / Accepted: 1 May 2001 Correspondence to: I. Yoda (e-mail: yoda@ieee.org, Tel.: +81-298-615941, Fax: +81-298-613313)     

The analytic structure of image flows: Deformation and segmentation

Time-varying imagery is often described in terms of image flow fields (i.e., image motion), which correspond to the perceptive projection of feature motions in three dimensions (3D). In the case of multiple moving objects with smooth surfaces, the image flow possesses an analytic structure that reflects these 3D properties. This paper describes the analytic structure of image flow fields in the image space-time domain, and its use for segmentation and 3D motion computation. First we discuss thelocal flow structure as embodied in the concept ofneighborhood deformation. The local image deformation is effectively represented by a set of 12 basis deformations, each of which is responsible for an independent deformation. This local representation provides us with sufficient information for the recovery of 3D object structure and motion, in the case of relative rigid body motions. We next discuss theglobal flow structure embodied in the partitioning of the entire image plane intoanalytic regions separated byboundaries of analyticity, such that each small neighborhood within the analytic region is described in terms of deformation bases. This analysis reveals an effective mechanism for detecting the analytic boundaries of flow fields, thereby segmenting the image into meaningful regions. The notion ofconsistency which is often used in the image segmentation is made explicit by the mathematical notion ofanalyticity derived from the projection relation of 3D object motion. The concept of flow analyticity is then extended to the temporal domain, suggesting a more robust algorithm for recovering image flow from multiple frames. Finally, we argue that the process of flow segmentation can be understood in the framework of grouping process. The general concept ofcoherence orgrouping through local support (such as the second-order flows in our case) is discussed.     

Qualitative egomotion

Due to the aperture problem, the only motion measurement in images, whose computation does not require any assumptions about the scene in view, is normal flow—the projection of image motion on the gradient direction. In this paper we show how a monocular observer can estimate its 3D motion relative to the scene by using normal flow measurements in a global and qualitative way. The problem is addressed through a search technique. By checking constraints imposed by 3D motion parameters on the normal flow field, the possible space of solutions is gradually reduced. In the four modules that comprise the solution, constraints of increasing restriction are considered, culminating in testing every single normal flow value for its consistency with a set of motion parameters. The fact that motion is rigid defines geometric relations between certain values of the normal flow field. The selected values form patterns in the image plane that are dependent on only some of the motion parameters. These patterns, which are determined by the signs of the normal flow values, are searched for in order to find the axes of translation and rotation. The third rotational component is computed from normal flow vectors that are only due to rotational motion. Finally, by looking at the complete data set, all solutions that cannot give rise to the given normal flow field are discarded from the solution space.Research supported in part by NSF (Grant IRI-90-57934), ONR (Contract N00014-93-1-0257) and ARPA (Order No. 8459).     

Moving object detection and segmentation in urban environments from a moving platform

This paper proposes an effective approach to detect and segment moving objects from two time-consecutive stereo frames, which leverages the uncertainties in camera motion estimation and in disparity computation. First, the relative camera motion and its uncertainty are computed by tracking and matching sparse features in four images. Then, the motion likelihood at each pixel is estimated by taking into account the ego-motion uncertainty and disparity in computation procedure. Finally, the motion likelihood, color and depth cues are combined in the graph-cut framework for moving object segmentation. The efficiency of the proposed method is evaluated on the KITTI benchmarking datasets, and our experiments show that the proposed approach is robust against both global (camera motion) and local (optical flow) noise. Moreover, the approach is dense as it applies to all pixels in an image, and even partially occluded moving objects can be detected successfully. Without dedicated tracking strategy, our approach achieves high recall and comparable precision on the KITTI benchmarking sequences.     

Binocular image flows: steps toward stereo-motion fusion

The analyses of visual data by stereo and motion modules have typically been treated as separate parallel processes which both feed a common viewer-centered 2.5-D sketch of the scene. When acting separately, stereo and motion analyses are subject to certain inherent difficulties; stereo must resolve a combinatorial correspondence problem and is further complicated by the presence of occluding boundaries, motion analysis involves the solution of nonlinear equations and yields a 3-D interpretation specified up to an undetermined scale factor. A new module is described here which unifies stereo and motion analysis in a manner in which each helps to overcome the other's short-comings. One important result is a correlation between relative image flow (i.e., binocular difference flow) and stereo disparity; it points to the importance of the ratio ? ?, rate of change of disparity ? to disparity ?, and its possible role in establishing stereo correspondence. The importance of such ratios was first pointed out by Richards [19]. Our formulation may reflect the human perception channel probed by Regan and Beverley [18].     

Recovering the Three-Dimensional Motion and Structure of Multiple Moving Objects from Binocular Image Flows

A new method is presented for recovering the three-dimensional motion and structure of multiple, independently moving, rigid objects through the analysis of binocular image flow fields. The input to the algorithm is the image location and image velocity of a sparse set of feature points in the stereo image pair. The algorithm analyzes one rigid object at a time by simultaneously segmenting the associated feature points from the input data set, establishing the stereo correspondence of these feature points and determining the three-dimensional motion of the object. The solution method is iterative and is based on the stereo-motion algorithm presented in J. H. Duncan, L. Li, and W. Wang, “Recovering Three-Dimensional Velocity and Established Stereo Correspondence from Binocular Image Flows” (Opt. Eng.34(7), July 1995, 2157–2167.) for the analysis of scenes with only one set of three-dimensional motion components. No restrictions on the three-dimensional structure of the scene are required by the theory. Experimental results with numerically generated and laboratory image sequences are given to verify the method.     

基于秩空间的二次立体匹配

提出了一种基于秩空间的区域立体匹配算法。首先对立体图像进行秩（rank）变换,将图像从灰度空间变换到秩空间,消除因噪声和两个摄像机参数不一致产生的干扰,再根据自然视频序列每幅图像多数景物景深变化不大的事实,把视差分解为全局视差与局部视差之和,在秩空间进行二次立体匹配：先在最大窗口内估计全局视差,然后在这个最大窗口内采用块匹配方式进行二次匹配求各点的实际视差。该二次立体匹配算法有效地消除了误匹配,提高正确匹配率。实验结果证明,提出的算法优于传统的基于区域的立体匹配方法。     

基于角点跟踪的光流场计算

在综合视觉运动分析中的两类处理方法，选取图像中的角点作为特征点，并检测和跟踪图像序列中的角点。记录检测到的角点在图像序列中的位移，在理论上证明了时变图像的光流场可以近似地用角点的位移场代替，同时给出这种替代的两个前提条件。本文用真实图像序列验证提出的算法，实验结果表明该算法取得了较好的效果。     

Real-time joint disparity and disparity flow estimation on programmable graphics hardware

Disparity flow depicts the 3D motion of a scene in the disparity space of a given view and can be considered as view-dependent scene flow. A novel algorithm is presented to compute disparity maps and disparity flow maps in an integrated process. Consequently, the disparity flow maps obtained helps to enforce the temporal consistency between disparity maps of adjacent frames. The disparity maps found also provides the spatial correspondence information that can be used to cross-validate disparity flow maps of different views. Two different optimization approaches are integrated in the presented algorithm for searching optimal disparity values and disparity flows. The local winner-take-all approach runs faster, whereas the global dynamic programming based approach produces better results. All major computations are performed in the image space of the given view, leading to an efficient implementation on programmable graphics hardware. Experimental results on captured stereo sequences demonstrate the algorithm’s capability of estimating both 3D depth and 3D motion in real-time. Quantitative performance evaluation using synthetic data with ground truth is also provided.     

Analysis of motion of a rigid body from its image. Orthogonal projection

We consider the determination of the three-dimensional structure and motion of a rigid body from the velocity and displacement fields of its orthogonal projection. A relationship between these techniques is established. Two approaches to the solution of the problem are examined.Translated from Kibernetika, No. 1, pp. 109–115, January–February, 1991.     

Determining three-dimensional motion and structure from optical flow generated by several moving objects

A new approach for the interpretation of optical flow fields is presented. The flow field, which can be produced by a sensor moving through an environment with several independently moving, rigid objects, is allowed to be sparse, noisy, and partially incorrect. The approach is based on two main stages. In the first stage, the flow field is partitioned into connected segments of flow vectors, where each segment is consistent with a rigid motion of a roughly planar surface. In the second stage, segments are grouped under the hypothesis that they are induced by a single, rigidly moving object. Each hypothesis is tested by searching for three-dimensional (3-D) motion parameters which are compatible with all the segments in the corresponding group. Once the motion parameters are recovered, the relative environmental depth can be estimated as well. Experiments based on real and simulated data are presented.     

融合边缘保持与改进代价聚合的立体匹配算法

目的 立体匹配是计算机双目视觉的重要研究方向,主要分为全局匹配算法与局部匹配算法两类。传统的局部立体匹配算法计算复杂度低,可以满足实时性的需要,但是未能充分利用图像的边缘纹理信息,因此在非遮挡、视差不连续区域的匹配精度欠佳。为此,提出了融合边缘保持与改进代价聚合的立体匹配。方法 首先利用图像的边缘空间信息构建权重矩阵,与灰度差绝对值和梯度代价进行加权融合,形成新的代价计算方式,同时将边缘区域像素点的权重信息与引导滤波的正则化项相结合,并在多分辨率尺度的框架下进行代价聚合。所得结果经过视差计算,得到初始视差图,再通过左右一致性检测、加权中值滤波等视差优化步骤获得最终的视差图。结果 在Middlebury立体匹配平台上进行实验,结果表明,融合边缘权重信息对边缘处像素点的代价量进行了更加有效地区分,能够提升算法在各区域的匹配精度。其中,未加入视差优化步骤的21组扩展图像对的平均误匹配率较改进前减少3.48%,峰值信噪比提升3.57 dB,在标准4幅图中venus上经过视差优化后非遮挡区域的误匹配率仅为0.18%。结论 融合边缘保持的多尺度立体匹配算法有效提升了图像在边缘纹理处的匹配精度,进一步降低了非遮挡区域与视差不连续区域的误匹配率。     

Scene Flow-Based Environment 3D Digitalization for Mobile Robot Navigation

This paper presents an effective 3D digitalization technique to reconstruct an accurate and reliable 3D environment model from multi-view stereo for an environment-learning mobile robot. The novelty of this paper lies in the introduction of nonrigid motion analysis to stereo reconstruction routine. In our proposed scheme, reconstruction task is decoupled into two stages. Firstly, range depth of feature points is recovered and in turn is used for building a polygonal mesh and secondly, projection feedback on comparison views, which is generated on assumption of the established coarse mesh model, is carefully introduced to deform the primitive mesh model so as to improve its quality dramatically. The discrepancy of observation on comparison views and the corresponding predictive feedback is quantitatively evaluated by optical flow field and is employed to derive the corresponding scene flow vector field subsequently, which is then used for surface deformation. As optical flow vector field estimation outperforms traditional dense disparity for its inherent advantage of being robust to illumination change and being optimized and smoothed in global sense, the deformed surface can be improved in accuracy, which is validated by experimental results.     

Generating near-spherical range panoramas by fusing optical flow and stereo from a single-camera folded catadioptric rig

We design a novel “folded” spherical catadioptric rig (formed by two coaxially-aligned spherical mirrors of distinct radii and a single perspective camera) to recover near-spherical range panoramas (about 360° × 153°) from the fusion of depth given by optical flow and stereoscopy. We observe that for rigid motion that is parallel to a plane, optical flow and stereo generate nearly complementary distributions of depth resolution. While optical flow provides strong depth cues in the periphery and near the poles of the view-sphere, stereo generates reliable depth in a narrow band about the equator instead. We exploit this dual-modality principle by modeling (separately) the depth resolution of optical flow and stereo in order to fuse them later on a probabilistic spherical panorama. We achieve a desired vertical field-of-view and optical resolution by deriving a linearized model of the rig in terms of three parameters (radii of the two mirrors plus axial distance between the mirrors’ centers). We analyze the error due to the violation of the single viewpoint constraint and formulate additional constraints on the design to minimize this error. We evaluate our proposed method via a synthetic model and with real-world prototypes by computing dense spherical panoramas of depth from cluttered indoor environments after fusing the two modalities (stereo and optical flow).     

3D Scene Reconstruction from Multiple Spherical Stereo Pairs

We propose a 3D environment modelling method using multiple pairs of high-resolution spherical images. Spherical images of a scene are captured using a rotating line scan camera. Reconstruction is based on stereo image pairs with a vertical displacement between camera views. A 3D mesh model for each pair of spherical images is reconstructed by stereo matching. For accurate surface reconstruction, we propose a PDE-based disparity estimation method which produces continuous depth fields with sharp depth discontinuities even in occluded and highly textured regions. A full environment model is constructed by fusion of partial reconstruction from spherical stereo pairs at multiple widely spaced locations. To avoid camera calibration steps for all camera locations, we calculate 3D rigid transforms between capture points using feature matching and register all meshes into a unified coordinate system. Finally a complete 3D model of the environment is generated by selecting the most reliable observations among overlapped surface measurements considering surface visibility, orientation and distance from the camera. We analyse the characteristics and behaviour of errors for spherical stereo imaging. Performance of the proposed algorithm is evaluated against ground-truth from the Middlebury stereo test bed and LIDAR scans. Results are also compared with conventional structure-from-motion algorithms. The final composite model is rendered from a wide range of viewpoints with high quality textures.     

Stereoscopic Scene Flow Computation for 3D Motion Understanding

Building upon recent developments in optical flow and stereo matching estimation, we propose a variational framework for the estimation of stereoscopic scene flow, i.e., the motion of points in the three-dimensional world from stereo image sequences. The proposed algorithm takes into account image pairs from two consecutive times and computes both depth and a 3D motion vector associated with each point in the image. In contrast to previous works, we partially decouple the depth estimation from the motion estimation, which has many practical advantages. The variational formulation is quite flexible and can handle both sparse or dense disparity maps. The proposed method is very efficient; with the depth map being computed on an FPGA, and the scene flow computed on the GPU, the proposed algorithm runs at frame rates of 20 frames per second on QVGA images (320×240 pixels). Furthermore, we present solutions to two important problems in scene flow estimation: violations of intensity consistency between input images, and the uncertainty measures for the scene flow result.     

VisionBug: A Hexapod Robot Controlled by Stereo Cameras

A six-legged robot system demonstrating reactive behaviors of simple organisms—walking between two walls, steering around obstacles, making turns at corners, and making U-turns at pathway dead-ends—is described. The system, named VisionBug, uses no active range sensor but only a stereo pair of cameras for sensing the surroundings. The system assumes the surroundings to be consisting of mostly a ground surface, although the surface could have a varying geometric relationship with the robot due to the jiggling nature of legged motion. By the use of the image-to-image mapping induced to the stereo images by the ground surface, the system is able to avoid explicit 3D reconstruction and the use of optical flow altogether in locating through-ways. Specifically, it regards image features not respecting the ground-induced mapping as obstacles, and express the obstacles in terms of a 2D distribution on the ground. Based on the distribution information, which is further time-delay compensated, a simple fuzzy control mechanism is used to command the legged motion. Experiments show that the system is effective for demonstrating the above-mentioned behaviors in textured environment, at a speed fast enough for many applications.