Multiplane Video Stabilization

This paper presents a novel video stabilization approach by leveraging the multiple planes structure of video scene to stabilize inter‐frame motion. As opposed to previous stabilization procedure operating in a single plane, our approach primarily deals with multiplane videos and builds their multiple planes structure for performing stabilization in respective planes. Hence, a robust plane detection scheme is devised to detect multiple planes by classifying feature trajectories according to reprojection errors generated by plane induced homographies. Then, an improved planar stabilization technique is applied by conforming to the compensated homography in each plane. Finally, multiple stabilized planes are coherently fused by content‐preserving image warps to obtain the output stabilized frames. Our approach does not need any stereo reconstruction, yet is able to produce commendable results due to awareness of multiple planes structure in the stabilization. Experimental results demonstrate the effectiveness and efficiency of our approach to robust stabilization on multiplane videos.     

Acquiring a radiance distribution to superimpose virtual objectsonto a real scene

This paper describes a new method for superimposing virtual objects with correct shadings onto an image of a real scene. Unlike the previously proposed methods, our method can measure a radiance distribution of a real scene automatically and use it for superimposing virtual objects appropriately onto a real scene. First, a geometric model of the scene is constructed from a pair of omnidirectional images by using an omnidirectional stereo algorithm. Then, radiance of the scene is computed from a sequence of omnidirectional images taken with different shutter speeds and mapped onto the constructed geometric model. The radiance distribution mapped onto the geometric model is used for rendering virtual objects superimposed onto the scene image. As a result, even for a complex radiance distribution, our method can superimpose virtual objects with convincing shadings and shadows cast onto the real scene. We successfully tested the proposed method by using real images to show its effectiveness     

Interactive object modelling based on piecewise planar surface patches

Detecting elements such as planes in 3D is essential to describe objects for applications such as robotics and augmented reality. While plane estimation is well studied, table-top scenes exhibit a large number of planes and methods often lock onto a dominant plane or do not estimate 3D object structure but only homographies of individual planes. In this paper we introduce MDL to the problem of incrementally detecting multiple planar patches in a scene using tracked interest points in image sequences. Planar patches are reconstructed and stored in a keyframe-based graph structure. In case different motions occur, separate object hypotheses are modelled from currently visible patches and patches seen in previous frames. We evaluate our approach on a standard data set published by the Visual Geometry Group at the University of Oxford [24] and on our own data set containing table-top scenes. Results indicate that our approach significantly improves over the state-of-the-art algorithms.     

Spatial layout recovery from a single omnidirectional image and its matching-free sequential propagation

The goal of this work is to recover the spatial layout of indoor environments from omnidirectional images assuming a Manhattan world structure. We propose a new method for scene structure recovery from a single image. This method is based on the line extraction for omnidirectional images, line classification, and vanishing points estimation combined with a new hierarchical expansion procedure for detecting floor and wall boundaries. Each single omnidirectional image independently provides a useful hypothesis of the 3D scene structure. In order to enhance the robustness and accuracy of this single image-based hypothesis, we extend this estimation with a new homography-based procedure applied to the various hypotheses obtained along the sequence of consecutive images. A key point in this contribution is the use of geometrical constraints for computing the homographies from a single line of the floor. The homography parametrization proposed allows the design of a matching-free method for spatial layout propagation along a sequence of images. Experimental results show single image layout recovery performance and the improvement obtained with the propagation of the hypothesis through the image sequence.     

Multiview constraints on homographies

The image motion of a planar surface between two camera views is captured by a homography (a 2D projective transformation). The homography depends on the intrinsic and extrinsic camera parameters, as well as on the 3D plane parameters. While camera parameters vary across different views, the plane geometry remains the same. Based on this fact, we derive linear subspace constraints on the relative homographies of multiple (⩾ 2) planes across multiple views. The paper has three main contributions: 1) We show that the collection of all relative homographies (homologies) of a pair of planes across multiple views, spans a 4-dimensional linear subspace. 2) We show how this constraint can be extended to the case of multiple planes across multiple views. 3) We show that, for some restricted cases of camera motion, linear subspace constraints apply also to the set of homographies of a single plane across multiple views. All the results derived are true for uncalibrated cameras. The possible utility of these multiview constraints for improving homography estimation and for detecting nonrigid motions are also discussed     

虚平面映射:深度图像内部视点的绘制技术

首先推导与归纳了图像三维变换中像素深度场的变换规律,同时提出了基于深度场和极线原则的像素可见性别方法,根据上述理论和方法,提出一种基于深度图像的建模与绘制(image-based modeling and rendering,简称IBMR)技术,称为虚平面映射.该技术可以基于图像空间内任意视点对场景进行绘制.绘制时,先在场景中根据视线建立若干虚拟平面,将源深度图像中的像素转换到虚平面上,然后通过对虚平面上像素的中间变换,将虚平面转换成平面纹理,再利用虚平面的相互拼接,将视点的成像以平面纹理映射的方式完成.新方法还能在深度图像内侧,基于当前视点快速获得该视点的全景图,从而实现视点的实时漫游.新方法视点运动空间大、存储需求小,且可以发挥图形硬件的纹理映射功能,并能表现物体表面的三维凹凸细节和成像视差效果,克服了此前类似算法的局限和不足.     

Vision-based multi-UAV position estimation

This paper describes a method for vision-based unmanned aerial vehicle (UAV) motion estimation from multiple planar homographies. The paper also describes the determination of the relative displacement between different UAVs employing techniques for blob feature extraction and matching. It then presents and shows experimental results of the application of the proposed technique to multi-UAV detection of forest fires.     

Superresolution modeling using an omnidirectional image sensor

Recently, many virtual reality and robotics applications have been called on to create virtual environments from real scenes. A catadioptric omnidirectional image sensor composed of a convex mirror can simultaneously observe a 360-degree field of view making it useful for modeling man-made environments such as rooms, corridors, and buildings, because any landmarks around the sensor can be taken in and tracked in its large field of view. However, the angular resolution of the omnidirectional image is low because of the large field of view captured. Hence, the resolution of surface texture patterns on the three-dimensional (3-D) scene model generated is not sufficient for monitoring details. To overcome this, we propose a high resolution scene texture generation method that combines an omnidirectional image sequence using image mosaic and superresolution techniques.     

From lines to epipoles through planes in two views

This paper addresses the computation of the fundamental matrix between two views, when camera motion and 3D structure are unknown, but planar surfaces can be assumed. We use line features which are automatically matched in two steps. Firstly, with image based parameters, a set of matches are obtained to secondly compute homographies, which allows to reject wrong ones, and to grow good matches in a final stage. The inclusion of projective transformations gives much better results to match features with short computing overload. When two or more planes are observed, different homographies can be computed, segmenting simultaneously the corresponding planar surfaces. These can be used to obtain the fundamental matrix, which gives constraints for the whole scene. The results show that the global process is robust enough, turning out stable and useful to obtain matches and epipolar geometry from lines in man made environments.     

Plane-based camera self-calibration by metric rectification of images

Plane-based self-calibration aims at the computation of camera intrinsic parameters from homographies relating multiple views of the same unknown planar scene. This paper proposes a straightforward geometric statement of plane-based self-calibration, through the concept of metric rectification of images. A set of constraints is derived from a decomposition of metric rectification in terms of intrinsic parameters and planar scene orientation. These constraints are then solved using an optimization framework based on the minimization of a geometrically motivated cost function. The link with previous approaches is demonstrated and our method appears to be theoretically equivalent but conceptually simpler. Moreover, a solution dealing with radial distortion is introduced. Experimentally, the method is compared with plane-based calibration and very satisfactory results are obtained. Markerless self-calibration is demonstrated using an intensity-based estimation of the inter-image homographies.     

遥感图像辅助的折反射全向图三维重构

提出了一种针对室外场景的单幅折反射全向图三维重构方法,能够自动重构出全向图中360°视野内景物的三维模型,并实现自由漫游。基于全向图与遥感图匹配把全向图分为水平地面、垂直建筑物立面和垂直背景景物面三类区域,得到全向图场景的基本结构;在此基础上利用折反射光路投射模型计算出全向图中每个像素点的三维几何位置,从而实现了折反射全向图的重构。实验证明该方法具有采集简单、视野大、处理过程全自动化、能够重构非平面场景等特点。     

Merging artificial objects with marker-less video sequences based on the interacting multiple model method

Inserting synthetic objects into video sequences has gained much interest in recent years. Fast and robust vision-based algorithms are necessary to make such an application possible. Traditional pose tracking schemes using recursive structure from motion techniques adopt one Kalman filter and thus only favor a certain type of camera motion. We propose a robust simultaneous pose tracking and structure recovery algorithm using the interacting multiple model (IMM) to improve performance. In particular, a set of three extended Kalman filters (EKFs), each describing a frequently occurring camera motion in real situations (general, pure translation, pure rotation), is applied within the IMM framework to track the pose of a scene. Another set of EKFs,one filter for each model point, is used to refine the positions of the model features in the 3-D space. The filters for pose tracking and structure refinement are executed in an interleaved manner. The results are used for inserting virtual objects into the original video footage. The performance of the algorithm is demonstrated with both synthetic and real data. Comparisons with different approaches have been performed and show that our method is more efficient and accurate.     

A novel plane extraction approach using supervised learning

This paper presents a novel approach for the classification of planar surfaces in an unorganized point clouds. A feature-based planner surface detection method is proposed which classifies a point cloud data into planar and non-planar points by learning a classification model from an example set of planes. The algorithm performs segmentation of the scene by applying a graph partitioning approach with improved representation of association among graph nodes. The planarity estimation of the points in a scene segment is then achieved by classifying input points as planar points which satisfy planarity constraint imposed by the learned model. The resultant planes have potential application in solving simultaneous localization and mapping problem for navigation of an unmanned-air vehicle. The proposed method is validated on real and synthetic scenes. The real data consist of five datasets recorded by capturing three-dimensional(3D) point clouds when a RGBD camera is moved in five different indoor scenes. A set of synthetic 3D scenes are constructed containing planar and non-planar structures. The synthetic data are contaminated with Gaussian and random structure noise. The results of the empirical evaluation on both the real and the simulated data suggest that the method provides a generalized solution for plane detection even in the presence of the noise and non-planar objects in the scene. Furthermore, a comparative study has been performed between multiple plane extraction methods.     

多机空战战法训练飞行视景仿真系统研究

多机空战战法训练设备研制是适应大规模分布式空战仿真训练的必然趋势.分析了战法训练的虚拟现实技术应用及其优势,对多机空战战法训练设备中飞机仿真平台的功能模型、视景流程和帧渲染顺序进行了总体设计.通过三维建模、虚拟场景配置、虚拟场景实时驱动三个步骤对视景系统进行实现,开发出一套适用于多机空战战法训练的飞行视景仿真系统.测试结果表明,所建立的飞行仿真视景系统能够满足飞行战法训练的要求.     

多平面多视点单应矩阵间的约束

用代数方法系统地讨论了多平面多视点下单应矩阵间的约束关系.主要结论有(A)如 果视点间摄像机的运动为纯平移运动,则1)对于所有平面关于两视点间的单应矩阵的集合,或 单个平面关于所有视点的单应矩阵的集合的秩均等于4,2)对于多平面多视点的标准单应矩阵 的集合其秩仍等于4,3)根据以上结论可推出现有文献中关于"相对单应矩阵"约束的所有结 果;(B)如果视点间摄像机的运动为一般运动,则1)对于所有平面关于两个视点间的单应矩阵 集合的秩等于4的结论仍成立,2)对于其它情况其秩不再等于4而是等于9.     

Toward coherent object detection and scene layout understanding

Detecting objects in complex scenes while recovering the scene layout is a critical functionality in many vision-based applications. In this work, we advocate the importance of geometric contextual reasoning for object recognition. We start from the intuition that objects' location and pose in the 3D space are not arbitrarily distributed but rather constrained by the fact that objects must lie on one or multiple supporting surfaces. We model such supporting surfaces by means of hidden parameters (i.e. not explicitly observed) and formulate the problem of joint scene reconstruction and object recognition as the one of finding the set of parameters that maximizes the joint probability of having a number of detected objects on K supporting planes given the observations. As a key ingredient for solving this optimization problem, we have demonstrated a novel relationship between object location and pose in the image, and the scene layout parameters (i.e. normal of one or more supporting planes in 3D and camera pose, location and focal length). Using a novel probabilistic formulation and the above relationship our method has the unique ability to jointly: i) reduce false alarm and false negative object detection rate; ii) recover object location and supporting planes within the 3D camera reference system; iii) infer camera parameters (view point and the focal length) from just one single uncalibrated image. Quantitative and qualitative experimental evaluation on two datasets (desk-top dataset [1] and LabelMe [2]) demonstrates our theoretical claims.     

由平行平面的投影确定无穷远平面的单应矩阵

在三维计算机视觉中,无穷远平面的单应矩阵扮演了极其重要的角色,可使众多视觉问题的求解得到简化.主要讨论如何利用平行平面的投影来求解两个视点间的无穷远平面的单应矩阵,用代数方法构造性地证明了下述结论:(1) 如果场景中含有一组平行平面,则可以通过求解一个一元4次方程来确定两个视点间的无穷远平面对应的单应矩阵;(2) 如果场景中含有两组平行平面,则可以线性地确定两个视点间的无穷远平面对应的单应矩阵.并对上述结果给出了相应的几何解释和具体算法.所给出的结果在三维计算机视觉,特别是摄像机自标定中具有一定的理论意义和应用价值.     

Automatic camera control in virtual environments augmented using multiple sparse videos

Automated virtual camera control has been widely used in animation and interactive virtual environments. We have developed a multiple sparse camera based free view video system prototype that allows users to control the position and orientation of a virtual camera, enabling the observation of a real scene in three dimensions (3D) from any desired viewpoint. Automatic camera control can be activated to follow selected objects by the user. Our method combines a simple geometric model of the scene composed of planes (virtual environment), augmented with visual information from the cameras and pre-computed tracking information of moving targets to generate novel perspective corrected 3D views of the virtual camera and moving objects. To achieve real-time rendering performance, view-dependent textured mapped billboards are used to render the moving objects at their correct locations and foreground masks are used to remove the moving objects from the projected video streams. The current prototype runs on a PC with a common graphics card and can generate virtual 2D views from three cameras of resolution 768×576 with several moving objects at about 11 fps.     

A performance study for camera pose estimation using visual marker based tracking

Vision-based tracking systems are widely used for augmented reality (AR) applications. Their registration can be very accurate and there is no delay between real and virtual scene. However, vision-based tracking often suffers from limited range, errors, heavy processing time and present erroneous behavior due to numerical instability. To address these shortcomings, robust method are required to overcome these problems. In this paper, we survey classic vision-based pose computations and present a method that offers increased robustness and accuracy in the context of real-time AR tracking. In this work, we aim to determine the performance of four pose estimation methods in term of errors and execution time. We developed a hybrid approach that mixes an iterative method based on the extended Kalman filter (EKF) and an analytical method with direct resolution of pose parameters computation. The direct method initializes the pose parameters of the EKF algorithm which performs an optimization of these parameters thereafter. An evaluation of the pose estimation methods was obtained using a series of tests and an experimental protocol. The analysis of results shows that our hybrid algorithm improves stability, convergence and accuracy of the pose parameters.