Manipulating refractive and reflective binocular disparity

Presenting stereoscopic content on 3D displays is a challenging task, usually requiring manual adjustments. A number of techniques have been developed to aid this process, but they account for binocular disparity of surfaces that are diffuse and opaque only. However, combinations of transparent as well as specular materials are common in the real and virtual worlds, and pose a significant problem. For example, excessive disparities can be created which cannot be fused by the observer. Also, multiple stereo interpretations become possible, e. g., for glass, that both reflects and refracts, which may confuse the observer and result in poor 3D experience. In this work, we propose an efficient method for analyzing and controlling disparities in computer‐generated images of such scenes where surface positions and a layer decomposition are available. Instead of assuming a single per‐pixel disparity value, we estimate all possibly perceived disparities at each image location. Based on this representation, we define an optimization to find the best per‐pixel camera parameters, assuring that all disparities can be easily fused by a human. A preliminary perceptual study indicates, that our approach combines comfortable viewing with realistic depiction of typical specular scenes.     

Calibration-free augmented reality in perspective

This paper deals with video-based augmented reality and proposes an algorithm for augmenting a real video sequence with views of graphics objects without metric calibration of the video camera by representing the motion of the video camera in projective space. A virtual camera, by which views of graphics objects are generated, is attached to a real camera by specifying image locations of the world coordinate system of the virtual world. The virtual camera is decomposed into calibration and motion components in order to make full use of graphics tools. The projective motion of the real camera recovered from image matches has the function of transferring the virtual camera and makes the virtual camera move according to the motion of the real camera. The virtual camera also follows the change of the internal parameters of the real camera. This paper shows the theoretical and experimental results of our application of nonmetric vision to augmented reality     

Local Shape from Mirror Reflections

We study the problem of recovering the 3D shape of an unknown smooth specular surface from a single image. The surface reflects a calibrated pattern onto the image plane of a calibrated camera. The pattern is such that points are available in the image where position, orientations, and local scale may be measured (e.g. checkerboard). We first explore the differential relationship between the local geometry of the surface around the point of reflection and the local geometry in the image.We then study the inverse problem and give necessary and sufficient conditions for recovering surface position and shape.We prove that surface position and shape up to third order can be derived as a function of local position, orientation and local scale measurements in the image when two orientations are available at the same point (e.g. a corner). Information equivalent to scale and orientation measurements can be also extracted from the reflection of a planar scene patch of arbitrary geometry, provided that the reflections of (at least) 3 distinctive points may be identified.We validate our theoretical results with both numerical simulations and experiments with real surfaces.     

Visual Modeling with a Hand-Held Camera

In this paper a complete system to build visual models from camera images is presented. The system can deal with uncalibrated image sequences acquired with a hand-held camera. Based on tracked or matched features the relations between multiple views are computed. From this both the structure of the scene and the motion of the camera are retrieved. The ambiguity on the reconstruction is restricted from projective to metric through self-calibration. A flexible multi-view stereo matching scheme is used to obtain a dense estimation of the surface geometry. From the computed data different types of visual models are constructed. Besides the traditional geometry- and image-based approaches, a combined approach with view-dependent geometry and texture is presented. As an application fusion of real and virtual scenes is also shown.     

Epipolar geometry from profiles under circular motion

Addresses the problem of motion estimation from profiles (apparent contours) of an object rotating on a turntable in front of a single camera. A practical and accurate technique for solving this problem from profiles alone is developed. It is precise enough to reconstruct the shape of the object. No correspondences between points or lines are necessary. Symmetry of the surface of revolution swept out by the rotating object is exploited to obtain the image of the rotation axis and the homography relating epipolar lines in two views robustly and elegantly. These, together with geometric constraints for images of rotating objects, are used to obtain first the image of the horizon, which is the projection of the plane that contains the camera centers, and then the epipoles, thus fully determining the epipolar geometry of the image sequence. The estimation of this geometry by this sequential approach avoids many of the problems found in other algorithms. The search for the epipoles, by far the most critical step, is carried out as a simple 1D optimization. Parameter initialization is trivial and completely automatic at all stages. After the estimation of the epipolar geometry, the Euclidean motion is recovered using the fixed intrinsic parameters of the camera obtained either from a calibration grid or from self-calibration techniques. Finally, the spinning object is reconstructed from its profiles using the motion estimated in the previous stage. Results from real data are presented, demonstrating the efficiency and usefulness of the proposed methods     

An adaptive observer framework for accurate feature depth estimation using an uncalibrated monocular camera

This paper presents a novel solution to the problem of depth estimation using a monocular camera undergoing known motion. Such problems arise in machine vision where the position of an object moving in three-dimensional space has to be identified by tracking motion of its projected feature on the two-dimensional image plane. The camera is assumed to be uncalibrated, and an adaptive observer yielding asymptotic estimates of focal length and feature depth is developed that precludes prior knowledge of scene geometry and is simpler than alternative designs. Experimental results using real camera imagery are obtained with the current scheme as well as the extended Kalman filter, and performance of the proposed observer is shown to be better than the extended Kalman filter-based framework.     

Visually guided ranging from observations of points, lines and curves via an identifier based nonlinear observer

In this paper we consider the problem of estimating the range information of features on an affine plane in by observing its image with the aid of a CCD camera, wherein we assume that the camera is undergoing a known motion. The features considered are points, lines and planar curves located on planar surfaces of static objects. The dynamics of the moving projections of the features on the image plane have been described as a suitable differential equation on an appropriate feature space. This dynamics is used to estimate feature parameters from which the range information is readily available. In this paper the proposed identification has been carried out via a newly introduced identifier based observer. Performance of the observer has been studied via simulation.     

Shape from Specular Reflection and Optical Flow

Inferring scene geometry from a sequence of camera images is one of the central problems in computer vision. While the overwhelming majority of related research focuses on diffuse surface models, there are cases when this is not a viable assumption: in many industrial applications, one has to deal with metal or coated surfaces exhibiting a strong specular behavior. We propose a novel and generalized constrained gradient descent method to determine the shape of a purely specular object from the reflection of a calibrated scene and additional data required to find a unique solution. This data is exemplarily provided by optical flow measurements obtained by small scale motion of the specular object, with camera and scene remaining stationary. We present a non-approximative general forward model to predict the optical flow of specular surfaces, covering rigid body motion as well as elastic deformation, and allowing for a characterization of problematic points. We demonstrate the applicability of our method by numerical experiments on synthetic and real data.     

Motion from occlusions

In computer vision, occlusions are almost always seen as undesirable singularities that pose difficult challenges to image motion analysis problems, such as optic flow computation, motion segmentation, disparity estimation, or egomotion estimation. However, it is well known that occlusions are extremely powerful cues for depth or motion perception, and could be used to improve those methods.

In this paper, we propose to recover camera motion information based uniquely on occlusions, by observing two specially useful properties: occlusions are independent of the camera rotation, and reveal direct information about the camera translation.

We assume a monocular observer, undergoing general rotational and translational motion in a static environment. We present a formal model for occlusion points and develop a method suitable for occlusion detection. Through the classification and analysis of the detected occlusion points, we show how to retrieve information about the camera translation (FOE). Experiments with real images are presented and discussed in the paper.     

一个基于全景图的虚拟环境漫游系统

虚拟现实的实现有两种方法。传统上,使用三维图形学方法进行建模和绘制,这种方法需要繁琐的建模工作和昂贵的专用绘图硬件,而且用三维模型很难真实表现自然景观。基于图像的绘制是实现虚拟现实系统的一种新方法,它克服了三维图形方法的上述缺点,近年来得到了日益广泛的应用。文章讨论了一个基于图像的虚拟环境漫游系统的实现,分析了此类系统的模型,介绍了系统实现中摄像机定标、图像拼接、实时图像变换等关键技术。     

利用CCD摄像机求解飞行器运动参数

针对飞行器的视觉导航应用,提出了利用CCD摄像机来估算飞行器运动参数的实时方案。在利用CCD摄像机所拍摄的序列图像中,首先,自动提取出第1帧图像中比较明显的特征点,并在后续的图像中对所提取的特征点进行追踪。然后,利用多视图几何技术来标定摄像机的外参数。所标定的摄像机的外参数经过转换就可以获得飞行器的运动参数。通过对实际拍摄的图像进行计算,反投影误差的最大值为0.03911,而处理26帧图像需要的时间为0.9194s,说明本方案在计算精度和实时性上是可行的。     

抗高光的光场深度估计方法

目的 光场相机一次成像可以同时获取场景中光线的空间和角度信息,为深度估计提供了条件。然而,光场图像场景中出现高光现象使得深度估计变得困难。为了提高算法处理高光问题的可靠性,本文提出了一种基于光场图像多视角上下文信息的抗高光深度估计方法。方法 本文利用光场子孔径图像的多视角特性,创建多视角输入支路,获取不同视角下图像的特征信息;利用空洞卷积增大网络感受野,获取更大范围的图像上下文信息,通过同一深度平面未发生高光的区域的深度信息,进而恢复高光区域深度信息。同时,本文设计了一种新型的多尺度特征融合方法,串联多膨胀率空洞卷积特征与多卷积核普通卷积特征,进一步提高了估计结果的精度和平滑度。结果 实验在3个数据集上与最新的4种方法进行了比较。实验结果表明,本文方法整体深度估计性能较好,在4D light field benchmark合成数据集上,相比于性能第2的模型,均方误差（mean square error,MSE）降低了20.24%,坏像素率（bad pixel,BP）降低了2.62%,峰值信噪比（peak signal-to-noise ratio,PSNR）提高了4.96%。同时,通过对CVIA （computer vision and image analysis） Konstanz specular dataset合成数据集和Lytro Illum拍摄的真实场景数据集的定性分析,验证了本文算法的有效性和可靠性。消融实验结果表明多尺度特征融合方法改善了深度估计在高光区域的效果。结论 本文提出的深度估计模型能够有效估计图像深度信息。特别地,高光区域深度信息恢复精度高、物体边缘区域平滑,能够较好地保存图像细节信息。     

基于真实环境图象的三维动态仿真系统研究

针对在城市规划中,建筑设计方案的计算机三维图形与真实环境图象进行匹配及合成的问题,研究和提出了一种基于摄影测量学返透视原理的,对实际环境摄像经抽样采集的图象序列进行亮度、方差及平均色度合成的图象分析方法,并研究了一种基于单象空间后方交会解析摄影测量技术一阶渐近迭代计算的求解虚拟摄像参数的算法,同时提出了基于关键帧技术的适合于图象合成时,计算分割区域随时间变化的动态掩模算法,还在太阳系行星运动规律的基础上,建立了适合于计算机虚拟三维世界坐标系的日照模型,以及相应的求解任意时刻、任意经纬度虚拟世界坐标位置处的虚拟太阳光源方向和强度的算法。这些算法已被应用到基于真实环境图象的三维动态仿真系统中,实验结果表明,这些算法是行这有效的。     

Visual motion and structure estimation using sliding mode observers

The problem of estimating motion and structure from a sequence of images has been a major research theme in machine vision for many years and remains one of the most challenging ones. In this work, we use sliding mode observers to estimate the motion and the structure of a moving body with the aid of a change-coupled device (CCD) camera. We consider a variety of dynamical systems which arise in machine vision applications and develop a novel identification procedure for the estimation of both constant and time-varying parameters. The basic procedure introduced for parameter estimation is to recast image feature dynamics linearly in terms of unknown parameters and construct a sliding mode observer to produce asymptotically correct estimates of the observed image features, and then use the observer input to compute parameters. Much of our analysis has been substantiated by computer simulations and real experiments.     

3D Surface Reconstruction Using Occluding Contours

This paper addresses the problem of 3D surface reconstruction using image sequences. It has been shown that shape recovery from three or more occluding contours of the surface is possible given a known camera motion. Several algorithms, which have been recently proposed, allow such a reconstruction under the assumption of a linear camera motion. A new approach is presented which deals with the reconstruction problem directly from a discrete point of view. First, a theoretical study of the epipolar correspondence between occluding contours is achieved. A correct depth formulation is then derived from a local approximation of the surface up to order two. This allows the local shape to be estimated, given three consecutive contours, without any constraints on the camera motion. Experimental results are presented for both synthetic and real data.     

An Analytically Stable Structure and Motion Observer Based on Monocular Vision

This paper presents a novel nonlinear sliding-mode differentiator-based complete-order observer for structure and motion identification with a calibrated monocular camera. In comparison with earlier work that requires prior knowledge of either the Euclidean geometry of the observed object or the linear acceleration of the camera and is restricted to establishing stability and convergence from image-plane measurements of a single tracked feature, the proposed scheme assumes partial velocity state feedback to asymptotically identify the true-scale Euclidean coordinates of numerous observed object features and the unknown motion parameters. The dynamics of the motion parameters are assumed to be described by a model with unknown parameters that incorporates a bounded uncertainty, and a Lyapunov analysis is provided to prove that the observer yields exponentially convergent estimates that converge to a uniform ultimate bound under a generic persistency of excitation condition. Numerical and experimental results are obtained that demonstrate the robust performance of the current scheme in the presence of model error and measurement noise.     

Detection and localization of specular surfaces using image motion cues

Successful identification of specularities in an image can be crucial for an artificial vision system when extracting the semantic content of an image or while interacting with the environment. We developed an algorithm that relies on scale and rotation invariant feature extraction techniques and uses motion cues to detect and localize specular surfaces. Appearance change in feature vectors is used to quantify the appearance distortion on specular surfaces, which has previously been shown to be a powerful indicator for specularity (Doerschner et al. in Curr Biol, 2011). The algorithm combines epipolar deviations (Swaminathan et al. in Lect Notes Comput Sci 2350:508–523, 2002) and appearance distortion, and succeeds in localizing specular objects in computer-rendered and real scenes, across a wide range of camera motions and speeds, object sizes and shapes, and performs well under image noise and blur conditions.     

Self-calibration of a 1D projective camera and its application to the self-calibration of a 2D projective camera

We introduce the concept of self-calibration of a 1D projective camera from point correspondences, and describe a method for uniquely determining the two internal parameters of a 1D camera, based on the trifocal tensor of three 1D images. The method requires the estimation of the trifocal tensor which can be achieved linearly with no approximation unlike the trifocal tensor of 2D images and solving for the roots of a cubic polynomial in one variable. Interestingly enough, we prove that a 2D camera undergoing planar motion reduces to a 1D camera. From this observation, we deduce a new method for self-calibrating a 2D camera using planar motions. Both the self-calibration method for a 1D camera and its applications for 2D camera calibration are demonstrated on real image sequences.     

一种真实物体表面反射属性采集与建模方法

为了快速获取物体表面反射属性数据和三维几何数据,研制了一套包括相机、光源、三维扫描仪、控制系统等的物体表面反射属性数据采集系统,在此基础上,提出一种单材质物体表面反射属性建模方法.该方法根据物体反射特性对漫反射与镜面反射分别建模,能够逼真地构造出物体表面反射属性模型,并且保证了参数求解的稳定性.最后设计并实现了反射属性模型编辑工具.实验结果表明,利用该编辑工具可以实时、可视地调整模型参数,从而满足用户对反射属性模型编辑的需求.     

The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences

The aim of this work is the recovery of 3D structure and camera projection matrices for each frame of an uncalibrated image sequence. In order to achieve this, correspondences are required throughout the sequence. A significant and successful mechanism for automatically establishing these correspondences is by the use of geometric constraints arising from scene rigidity. However, problems arise with such geometry guided matching if general viewpoint and general structure are assumed whilst frames in the sequence and/or scene structure do not conform to these assumptions. Such cases are termed degenerate.In this paper we describe two important cases of degeneracy and their effects on geometry guided matching. The cases are a motion degeneracy where the camera does not translate between frames, and a structure degeneracy where the viewed scene structure is planar. The effects include the loss of correspondences due to under or over fitting of geometric models estimated from image data, leading to the failure of the tracking method. These degeneracies are not a theoretical curiosity, but commonly occur in real sequences where models are statistically estimated from image points with measurement error.We investigate two strategies for tackling such degeneracies: the first uses a statistical model selection test to identify when degeneracies occur: the second uses multiple motion models to overcome the degeneracies. The strategies are evaluated on real sequences varying in motion, scene type, and length from 13 to 120 frames.