立体视频对象分割及其三维重建算法研究*

为更加有效分析立体视频对象,本文提出了一种基于离散冗余小波变换的立体视频对象分割算法,首先采用离散冗余小波变换提取特征点结合DT网格技术的视差估计方法,获得了可靠的视差场,再利用视差信息对立体视频中静止对象进行分割。对于立体视频序列中的运动对象,采用离散冗余小波提取运动区域的方法进行分割。实验结果表明,本算法对有重叠的多视频对象具有较好的分割效果,可同时分割静止物体和运动物体,具有较好的精确性和鲁棒性。对于分割出的立体视频对象,结合深度信息对其进行三维重建,得到较好的三维效果。     

A LSS-based registration of stereo thermal–visible videos of multiple people using belief propagation

In this paper, we propose a novel stereo method for registering foreground objects in a pair of thermal and visible videos of close-range scenes. In our stereo matching, we use Local Self-Similarity (LSS) as similarity metric between thermal and visible images. In order to accurately assign disparities to depth discontinuities and occluded Region Of Interest (ROI), we have integrated color and motion cues as soft constraints in an energy minimization framework. The optimal disparity map is approximated for image ROIs using a Belief Propagation (BP) algorithm. We tested our registration method on several challenging close-range indoor video frames of multiple people at different depths, with different clothing, and different poses. We show that our global optimization algorithm significantly outperforms the existing state-of-the art method, especially for disparity assignment of occluded people at different depth in close-range surveillance scenes and for relatively large camera baseline.     

Stereo disparity optimization with depth change constraint based on a continuous video

Three-dimensional reconstruction based on stereo vision technology is an important research direction in the field of computer vision, and has a wide range of applications in industrial measurement, medical image reconstruction, cultural relic preservation, robot navigation, virtual reality and other fields. However, the three-dimensional reconstruction of moving objects usually has poor accuracy, low efficiency and poor visualization effect due to the image noise, motion blur, complex and time-consuming calculation etc. In this article, a disparity optimization method based on depth change constraint is proposed, which utilizes the correlation of the adjacent frames in the continuous video sequence to eliminate mismatches and correct the wrong disparity values by introducing a depth change constraint threshold. The experiments on the video images which are taken by a binocular stereo vision system demonstrate that our method of removing incorrect matches bears satisfactory results and it can greatly improve the effect of the three-dimensional reconstruction of the moving objects.     

视频图像序列运动参数估计与动态拼接

本文采用多重分层叠代算法来估计全局运动参数，并提出应用于动态拼接的运动分割新方法，实现既有摄像机运动又有物体运动的视频图像序列自动拼接。我们的方法基本步骤如下：首先进行全局运动参数的初始估计，并且在分层叠代过程中进行区域分类，得到初始运动模板。接着空间分割原始图像，先根据图像的空间属性由底向上分层合并图像空间区域，再利用视频图像时间属性进一步向上合并，得到图像空间分割结果。然后结合初始运动模板和图像空间分割结果，采用区域分类新方法重新对图像空间分割结果的每个区域进行分类。然后根据分类结果逐步精确求解全局运动参数。最后进行图像合成，得到全景拼接图像。我们的方法利用了多重分层叠代的优点，并且充分考虑到视频图像空间和时间上的属性，实现了运动物体和覆盖背景的精确分割，避免了遮挡问题对全局运动参数估计精度的影响。而且在图像合成时我们解决了拼接图可能产生模糊或某些区域不连续等问题。实验结果表明我们的方法实现了动态视频图像序列高质量的全景拼接。     

Non-Rigid Stereo Factorization

In this paper we address the problem of recovering 3D non-rigid structure from a sequence of images taken with a stereo pair. We have extended existing non-rigid factorization algorithms to the stereo camera case and presented an algorithm to decompose the measurement matrix into the motion of the left and right cameras and the 3D shape, represented as a linear combination of basis-shapes. The added constraints in the stereo camera case are that both cameras are viewing the same structure and that the relative orientation between both cameras is fixed. Our focus in this paper is on the recovery of flexible 3D shape rather than on the correspondence problem. We propose a method to compute reliable 3D models of deformable structure from stereo images. Our experiments with real data show that improved reconstructions can be achieved using this method. The algorithm includes a non-linear optimization step that minimizes image reprojection error and imposes the correct structure to the motion matrix by choosing an appropriate parameterization. We show that 3D shape and motion estimates can be successfully disambiguated after bundle adjustment and demonstrate this on synthetic and real image sequences. While this optimization step is proposed for the stereo camera case, it can be readily applied to the case of non-rigid structure recovery using a monocular video sequence. Electronic supplementary material Electronic supplementary material is available for this article at and accessible for authorised users.     

Depth extraction using a single moving camera: an integration of depth from motion and depth from stereo

An integrated approach to extract depth, efficiently and accurately, from a sequence of images is presented in this paper. The method combines the ability of the stereo processing to acquire highly accurate depth measurements and the efficiency of spatial and temporal gradient analysis. As a result of this integration, depth measurements of high quality are obtained at a speed approximately ten times greater than that of stereo processing. Without any a priori information of the locations of the points in the scene, the correspondence problem in stereo processing is computationally expensive. In our approach, we use spatial and temporal gradient (STG) analysis, which has been shown to provide depth with great efficiency, but limited accuracy, to guide the matching process of stereo. The camera motion used in the approach can be either lateral or axial. Extensive experiments on real scenes have shown the ability of the integrated approach to acquire depth with a mean error of less than 3%.     

立体视频误码恢复技术综述

随着立体视频技术的发展,大量视频数据的传输面临着巨大的挑战。当经过错误信道传输时,视频数据常常面临出错或丢失的现象,因而,有效的误码恢复技术变得极为重要。首先介绍了一些常见的立体视频编码结构及其对应的误码现象,并针对目前已有的立体视频误码恢复技术的编码结构,对相应的误码恢复方法进行了研究和分析。最后,指出了进一步发展的前景。     

Generating near-spherical range panoramas by fusing optical flow and stereo from a single-camera folded catadioptric rig

We design a novel “folded” spherical catadioptric rig (formed by two coaxially-aligned spherical mirrors of distinct radii and a single perspective camera) to recover near-spherical range panoramas (about 360° × 153°) from the fusion of depth given by optical flow and stereoscopy. We observe that for rigid motion that is parallel to a plane, optical flow and stereo generate nearly complementary distributions of depth resolution. While optical flow provides strong depth cues in the periphery and near the poles of the view-sphere, stereo generates reliable depth in a narrow band about the equator instead. We exploit this dual-modality principle by modeling (separately) the depth resolution of optical flow and stereo in order to fuse them later on a probabilistic spherical panorama. We achieve a desired vertical field-of-view and optical resolution by deriving a linearized model of the rig in terms of three parameters (radii of the two mirrors plus axial distance between the mirrors’ centers). We analyze the error due to the violation of the single viewpoint constraint and formulate additional constraints on the design to minimize this error. We evaluate our proposed method via a synthetic model and with real-world prototypes by computing dense spherical panoramas of depth from cluttered indoor environments after fusing the two modalities (stereo and optical flow).     

Bilayer segmentation of webcam videos using tree-based classifiers

This paper presents an automatic segmentation algorithm for video frames captured by a (monocular) webcam that closely approximates depth segmentation from a stereo camera. The frames are segmented into foreground and background layers that comprise a subject (participant) and other objects and individuals. The algorithm produces correct segmentations even in the presence of large background motion with a nearly stationary foreground. This research makes three key contributions: First, we introduce a novel motion representation, referred to as "motons," inspired by research in object recognition. Second, we propose estimating the segmentation likelihood from the spatial context of motion. The estimation is efficiently learned by random forests. Third, we introduce a general taxonomy of tree-based classifiers that facilitates both theoretical and experimental comparisons of several known classification algorithms and generates new ones. In our bilayer segmentation algorithm, diverse visual cues such as motion, motion context, color, contrast, and spatial priors are fused by means of a conditional random field (CRF) model. Segmentation is then achieved by binary min-cut. Experiments on many sequences of our videochat application demonstrate that our algorithm, which requires no initialization, is effective in a variety of scenes, and the segmentation results are comparable to those obtained by stereo systems.     

一种基于3DTV编解码技术的视频流传输方法*

3DTV编解码技术及其立体视频流的优化传输方法是构建实时立体多视角3DTV系统的关键所在。通过对立体视频压缩编码和优化技术、立体视频解码技术,以及运动补偿预测与视差补偿预测技术的分析,在基于DSP平台TMS320DM642上进行AVS标清立体视频编解码及视频流优化传输,得出将互补的流媒体内容分发技术加以融合的立体视频流传输方法。     

Spatial error concealment for stereoscopic video coding based on pixel matching

In stereoscopic video coding, the interview correlation between the stereo image pair can be used for error concealment. A new spatial error concealment method for stereoscopic video coding based on pixel matching in the decoder is proposed in this paper. The lost macroblocks are recovered by utilizing disparity matching between two-view images on a pixel-by-pixel basis. Firstly, we get the candidate disparity vectors of the four neighboring pixels of the lost pixel by disparity matching in the decoder. Secondly, by calculating the boundary pixel difference, we determine an optimal replacing pixel in the reference image, and then we recover the lost pixel by the optimal pixel in the reference image. Experimental results show that the proposed algorithm performs better comparing to the previous technique.     

视觉关注度的立体图像质量评价

立体图像质量评价对立体视频技术的发展起着非常重要的作用。常用的PSNR(峰值信噪比)方法不能反映人类视觉感知特性,也不能直接应用到立体图像质量评价中。针对人类视觉对立体图像深度感知和重点关注感兴趣区的特点,提出基于纹理图和深度图感兴趣区的质量评价方法。首先对纹理图和对应的深度图利用视觉关注度提取工具提取感兴趣区,然后在评价的过程中对各感兴趣区根据感兴趣程度分配基于像素的权重系数,最后将权重系数应用在立体图像的各个区域中进行评价。实验结果表明该方法得到的立体图像质量客观评价结果与主观评价结果之间具有更好的一致性,符合人眼视觉系统感知特性。     

Analysis of depth estimation error for cylindrical stereo imaging

Conventional stereo imaging uses area CCDs for which depth perception error has been analyzed in our past research (Comput. Vision Image understand. 63 (3) (1996) 447). In this work we analyze the depth perception error for a stereo system that uses two rotating linear CCD cameras to create cylindrical stereo images. Theoretical analysis shows certain advantages to cylindrical stereo imaging over conventional methods. Initial experimental results are presented to validate the theoretical results. Unlike normal area CCDs, cylindrical images captured by rotating linear CCD cameras can produce very high-resolution images thereby substantially reducing the error in 3D estimation.     

面向监视视频实时分析的快速行人检测方法

为解决监视视频实时分析应用中行人检测效率低的问题,提出一种快速行人检测方法。首先,采用运动侦测方法提取运动区域,并结合行人检测要求对运动区域进行尺寸扩展、归一化和拼接操作;然后,在拼接图像上结合积分图快速提取各运动区域的Haar特征,并采用双支持向量机实现快速的特征分类;最后,结合包围盒相交策略进行帧间滤波,降低行人误检现象。实验表明,本文方法不仅可以实时检测行人目标,而且检测错误率低于现有主流方法。     

Video 2‐D—to—3‐D conversion based on hybrid depth cueing

Abstract— With the maturation of three‐dimensional (3‐D) technologies, display systems can provide higher visual quality to enrich the viewer experience. However, the depth information required for 3‐D displays is not available in conventional 2‐D recorded contents. Therefore, the conversion of existing 2‐D video to 3‐D video becomes an important issue for emerging 3‐D applications. This paper presents a system which automatically converts 2‐D videos to 3‐D format. The proposed system combines three major depth cues: the depth from motion, the scene depth from geometrical perspective, and the fine‐granularity depth from the relative position. The proposed system uses a block‐based method incorporating a joint bilateral filter to efficiently generate visually comfortable depth maps and to diminish the blocky artifacts. By means of the generated depth map, 2‐D videos can be readily converted into 3‐D format. Moreover, for conventional 2‐D displays, a 2‐D image/video depth perception enhancement application is also presented. With the depth‐aware adjustment of color saturation, contrast, and edge, the stereo effect of the 2‐D content can be enhanced. A user study on subjective quality shows that the proposed method has promising results on depth quality and visual comfort.     

A Database and Evaluation Methodology for Optical Flow

The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance. The challenges for optical flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms. To that end, we contribute four types of data to test different aspects of optical flow algorithms: (1) sequences with nonrigid motion where the ground-truth flow is determined by tracking hidden fluorescent texture, (2) realistic synthetic sequences, (3) high frame-rate video used to study interpolation error, and (4) modified stereo sequences of static scenes. In addition to the average angular error used by Barron et al., we compute the absolute flow endpoint error, measures for frame interpolation error, improved statistics, and results at motion discontinuities and in textureless regions. In October 2007, we published the performance of several well-known methods on a preliminary version of our data to establish the current state of the art. We also made the data freely available on the web at . Subsequently a number of researchers have uploaded their results to our website and published papers using the data. A significant improvement in performance has already been achieved. In this paper we analyze the results obtained to date and draw a large number of conclusions from them.     

基于压缩域信息的立体视频对象分割方法

立体视频对象分割是交互式多视点视频应用的关键技术。为了提高对象分割的时效性和精确性,提出了一种利用压缩域视差和运动信息的立体视频对象分割算法。该算法首先对运动矢量场和视差场进行提取和修正处理,然后对视频帧进行分割作为初始值,最后用均值偏移算法聚类得到最终的对象分割结果。实验结果表明,对于纹理复杂的场景有很好的分割效果,可以获得与语义一致的对象。     

Analysis of Error in Depth Perception with Vergence and Spatially Varying Sensing

In stereo vision the depth of a 3-D point is estimated based on the position of its projections on the left and right images. The image plane of cameras that produces the images consists of discrete pixels. This discretization of images generates uncertainty in estimation of the depth at each 3-D point. In this paper, we investigate the effect of vergence and spatially varying resolution on the depth estimation error. First, vergence is studied when pairs of stereo images with uniform resolution are used. Then the problem is studied for a stereo system similar to that of humans, in which cameras have high resolution in the center and nonlinearly decreasing resolution toward the periphery. In this paper we are only concerned with error in depth perception, assuming that stereo matching is already done.     

A multicamera setup for generating stereo panoramic video

Traditional visual communication systems convey only two-dimensional (2-D) fixed field-of-view (FOV) video information. The viewer is presented with a series of flat, nonstereoscopic images, which fail to provide a realistic sense of depth. Furthermore, traditional video is restricted to only a small part of the scene, based on the director's discretion and the user is not allowed to "look around" in an environment. The objective of this work is to address both of these issues and develop new techniques for creating stereo panoramic video sequences. A stereo panoramic video sequence should be able to provide the viewer with stereo vision at any direction (complete 360-degree FOV) at video rates. In this paper, we propose a new technique for creating stereo panoramic video using a multicamera approach, thus creating a high-resolution output. We present a setup that is an extension of a previously known approach, developed for the generation of still stereo panoramas, and demonstrate that it is capable of creating high-resolution stereo panoramic video sequences. We further explore the limitations involved in a practical implementation of the setup, namely the limited number of cameras and the nonzero physical size of real cameras. The relevant tradeoffs are identified and studied.     

立体视觉里程计中基于微粒群优化的初始运动估计和内点检测

初始运动估计和内点检测是影响立体视觉里程计定位精度的重要因素.目前,立体视觉里程计都采用基于3点线性运动估计的随机采样一致性(random sample consensus,RANSAC)方法.本文分析了随机采样一致性方法在初始运动估计中的性能:该方法对排除误匹配点是有效的,但在一定采样次数下采样到特征点提取误差和立体匹配误差都很小的匹配点的概率是很小的,所以通过该方法得到的初始运动参数和匹配内点不够精确。本文提出了采用微粒群优化的初始运动估计和内点检测新方法,该方法收敛速度快,搜索精确解的能力强,能够获得高精度的运动参数和匹配内点.立体视觉里程计仿真实验和真实智能车实验表明:和随机采样一致性方法相比,本文方法在运行时间、定位精度方面都更优越.