Augmented reality camera tracking with homographies

To realistically integrate 3D graphics into an unprepared environment, camera position must be estimated by tracking natural image features. We apply our technique to cases where feature positions in adjacent frames of an image sequence are related by a homography, or projective transformation. We describe this transformation's computation and demonstrate several applications. First, we use an augmented notice board to explain how a homography, between two images of a planar scene, completely determines the relative camera positions. Second, we show that the homography can also recover pure camera rotations, and we use this to develop an outdoor AR tracking system. Third, we use the system to measure head rotation and form a simple low-cost virtual reality (VR) tracking solution.     

基于关键点及光流的平面物体跟踪算法

为提高复杂场景下基于关键点的平面物体跟踪算法的鲁棒性,提出一种融合光流的平面物体跟踪算法。检测目标物体与输入图像的关键点及其对应描述符,由最近邻匹配方法构建目标与图像间关键点匹配集合,通过光流法构建相邻两张图像间关键点的对应关系,将已构建的关键点匹配集合与基于光流的对应关系通过加权平均的策略进行融合,得出修正的关键点匹配集合,根据关键点匹配估计目标物体在当前图像的单应性变换矩阵,从而完成目标跟踪。在POT数据集上的实验结果表明,与SIFT、FERNS等算法相比,在校正误差阈值为5时,该算法在所有图像序列上的平均跟踪精度达到66.67%,具有较好的跟踪性能。     

由平行平面的投影确定无穷远平面的单应矩阵

在三维计算机视觉中,无穷远平面的单应矩阵扮演了极其重要的角色,可使众多视觉问题的求解得到简化.主要讨论如何利用平行平面的投影来求解两个视点间的无穷远平面的单应矩阵,用代数方法构造性地证明了下述结论:(1) 如果场景中含有一组平行平面,则可以通过求解一个一元4次方程来确定两个视点间的无穷远平面对应的单应矩阵;(2) 如果场景中含有两组平行平面,则可以线性地确定两个视点间的无穷远平面对应的单应矩阵.并对上述结果给出了相应的几何解释和具体算法.所给出的结果在三维计算机视觉,特别是摄像机自标定中具有一定的理论意义和应用价值.     

The Geometry and Matching of Lines and Curves Over Multiple Views

This paper describes the geometry of imaged curves in two and three views. Multi-view relationships are developed for lines, conics and non-algebraic curves. The new relationships focus on determining the plane of the curve in a projective reconstruction, and in particular using the homography induced by this plane for transfer from one image to another. It is shown that given the fundamental matrix between two views, and images of the curve in each view, then the plane of a conic may be determined up to a two fold ambiguity, but local curvature of a curve uniquely determines the plane. It is then shown that given the trifocal tensor between three views, this plane defines a homography map which may be used to transfer a conic or the curvature from two views to a third. Simple expressions are developed for the plane and homography in each case.A set of algorithms are then described for automatically matching individual line segments and curves between images. The algorithms use both photometric information and the multiple view geometric relationships. For image pairs the homography facilitates the computation of a neighbourhood cross-correlation based matching score for putative line/curve correspondences. For image triplets cross-correlation matching scores are used in conjunction with line/curve transfer based on the trifocal geometry to disambiguate matches. Algorithms are developed for both short and wide baselines. The algorithms are robust to deficiencies in the segment extraction and partial occlusion.Experimental results are given for image pairs and triplets, for varying motions between views, and for different scene types. The methods are applicable to line/curve matching in stereo and trinocular rigs, and as a starting point for line/curve matching through monocular image sequences.     

借助大位移视图进行图像修补的透视畸变最小化算法

提出一种由粗到精的透视畸变最小化算法,借助大位移视图来修补目标图像.它先对大位移视点图像进行透视畸变校正后再用来补全目标图像上的丢失信息区域.首先,在平面场景的假设下,大位移视点图像通过单应矩阵进行全局变形得到初始的畸变校正.然后,由误匹配识别机制检测出初始校正的大位移视图中的残余畸变.在颜色一致性和位移场光滑性的期望下,残余畸变通过基于能量优化的重叠像素对应算法得到进一步的松弛.最后,在极线几何以及像素邻域中的位移场光滑性和颜色一致性的约束下,信息丢失像素按照特别定义的修补优先级函数依次得到恢复.泊松图像融合算法被用于消除修补区域与其周围像素之间可能存在的鬼影现象从而得到无缝的修补效果.实验表明,该方法优于已有的图像修补算法,且能够修补含有复杂结构信息的较大受损区域.     

Distortion Optimization based Image Completion from a Large Displacement View

We present a new image completion method based on an additional large displacement view (LDV) of the same scene for faithfully repairing large missing regions on the target image in an automatic way. A coarse‐to‐fine distortion correction algorithm is proposed to minimize the perspective distortion in the corresponding parts for the common scene regions on the LDV image. First, under the assumption of a planar scene, the LDV image is warped according to a homography to generate the initial correction result. Second, the residual distortions in the common known scene regions are revealed by means of a mismatch detection mechanism and relaxed by energy optimization of overlap correspondences, with the expectations of color constancy and displacement field smoothness. The fundamental matrix for the two views is then computed based on the reliable correspondence set. Third, under the constraints of epipolar geometry, displacement field smoothness and color consistency of the neighboring pixels, the missing pixels are orderly restored according to a specially defined repairing priority function. We finally eliminate the ghost effect between the repaired region and its surroundings by Poisson image blending. Experimental results demonstrate that our method outperforms recent state‐of‐the‐art image completion methods for repairing large missing area with complex structure information.     

Camera calibration based on arbitrary parallelograms

Existing algorithms for camera calibration and metric reconstruction are not appropriate for image sets containing geometrically transformed images for which we cannot apply the camera constraints such as square or zero-skewed pixels. In this paper, we propose a framework to use scene constraints in the form of camera constraints. Our approach is based on image warping using images of parallelograms. We show that the warped image using parallelograms constrains the camera both intrinsically and extrinsically. Image warping converts the calibration problems of transformed images into the calibration problem with highly constrained cameras. In addition, it is possible to determine affine projection matrices from the images without explicit projective reconstruction. We introduce camera motion constraints of the warped image and a new parameterization of an infinite homography using the warping matrix. Combining the calibration and the affine reconstruction results in the fully metric reconstruction of scenes with geometrically transformed images. The feasibility of the proposed algorithm is tested with synthetic and real data. Finally, examples of metric reconstructions are shown from the geometrically transformed images obtained from the Internet.     

场景无关约束下的特征匹配算法

鉴于对极约束是立体图像中完全不依赖于场景的重要几何约束，因此在特征匹配中起着很重要的作用，而且由于同形映射描述了平面场景的立体图像之间的对应关系，故大量文献中利用它对平面场景的立体图像对进行特征匹配。为了提高立体图像匹配精度和速度，提出了一种改进的场景无关约束下的特征匹配算法，该算法针对用对极约束和同形映射来进行曲面场景匹配的过程中同形估计容易出现降阶的情况，通过引入区域面积检测法来避免降阶情况的发生，以改善匹配结果；同时，由于在同形矩阵估计中，通过加入基础矩阵和同形矩阵本质上的约束关系，可使得原本独立的同形约束和对极约柬关系很好地融入到匹配的整个过程中，从而快速有效地抑制了错误匹配的发生。对真实图像的实验分析证明，该改进算法具有迭代次数少、速度更快和匹配精度高的良好性能。     

Linearized motion estimation for articulated planes

In this paper, we describe the explicit application of articulation constraints for estimating the motion of a system of articulated planes. We relate articulations to the relative homography between planes and show that these articulations translate into linearized equality constraints on a linear least-squares system, which can be solved efficiently using a Karush-Kuhn-Tucker system. The articulation constraints can be applied for both gradient-based and feature-based motion estimation algorithms and to illustrate this, we describe a gradient-based motion estimation algorithm for an affine camera and a feature-based motion estimation algorithm for a projective camera that explicitly enforces articulation constraints. We show that explicit application of articulation constraints leads to numerically stable estimates of motion. The simultaneous computation of motion estimates for all of the articulated planes in a scene allows us to handle scene areas where there is limited texture information and areas that leave the field of view. Our results demonstrate the wide applicability of the algorithm in a variety of challenging real-world cases such as human body tracking, motion estimation of rigid, piecewise planar scenes, and motion estimation of triangulated meshes.     

Improved frame-by-frame object pose tracking in complex environments

Pose tracking is an important task in Augmented Reality (AR), interactive systems, and robotic systems. The frame-by-frame pose tracking that is effective in many cases still faces challenges in complex environments such as occlusions, illumination changes and flipping. In this paper, based on the optimization model offered by Ye et al. J Vis Commun Image Represent 44:72–81 (2017), three improvements are further proposed. First, a feature adjustment strategy based on a group of neighbors is offered to alleviate a sharp reduction of features. Then, when the features are no longer well representing the scene of interest, a score model based on a weighted histogram for result evaluations is presented to realize an adaptive interval. Besides, a forward-backward algorithm is provided to improve the accuracy by replacing the detection method with the tracking method. Experimental results manifest the effectiveness of the proposed algorithms.     

一种基于平面模板的摄像机自定标方法

给出仿射坐标系下场景中平面与像平面的单应关系、绝对二次曲线及其图像的表示．通过对场景中一个平面模板获取3幅图像(该模板是由含内切圆的等边三角形构成)，利用上述单应关系并结合圆环点对摄像机内参数的约束，获得一组线性方程，进而确定摄像机的内参数．实验结果表明，所给出的方法切实可行，且具有较高的求解精度．     

基于单应矩阵的摄像机标定方法及应用

提出了一种基于单应矩阵的摄像机标定方法,并应用标定结果成功完成了移动机器人的视觉伺服任务。该方法首先根据图像平面和标定板平面之间特征点的对应关系,对单应矩阵进行了估计,进而利用旋转矩阵的单位正交性得到了其对摄像机内参数的约束条件。然后把摄像机内参数矩阵分解为有效焦距与主点位置两部分,并利用最小二乘法分别对其进行求解。针对镜头的径向畸变,恰当地选取了一种畸变模型,并由此得到了一种新的目标函数来对摄像机的所有参数进行非线性优化,从而使获得的畸变系数更适合于从二维图像信号中提取三维位姿信息。最后将标定结果成功应用于移动机器人视觉伺服系统之中,实验结果验证了该标定算法具有简单易用、精度较高等优良性能。     

Benchmarking template-based tracking algorithms

For natural interaction with augmented reality (AR) applications, good tracking technology is key. But unlike dense stereo, optical flow or multi-view stereo, template-based tracking which is most commonly used for AR applications lacks benchmark datasets allowing a fair comparison between state-of-the-art algorithms. Until now, in order to evaluate objectively and quantitatively the performance and the robustness of template-based tracking algorithms, mainly synthetically generated image sequences were used. The evaluation is therefore often intrinsically biased. In this paper, we describe the process we carried out to perform the acquisition of real-scene image sequences with very precise and accurate ground truth poses using an industrial camera rigidly mounted on the end effector of a high-precision robotic measurement arm. For the acquisition, we considered most of the critical parameters that influence the tracking results such as: the texture richness and the texture repeatability of the objects to be tracked, the camera motion and speed, and the changes of the object scale in the images and variations of the lighting conditions over time. We designed an evaluation scheme for object detection and interframe tracking algorithms suited for AR and other computer vision applications and used the image sequences to apply this scheme to several state-of-the-art algorithms. The image sequences are freely available for testing, submitting and evaluating new template-based tracking algorithms, i.e. algorithms that detect or track a planar object in an image sequence given only one image of the object (called the template).     

Spatial layout recovery from a single omnidirectional image and its matching-free sequential propagation

The goal of this work is to recover the spatial layout of indoor environments from omnidirectional images assuming a Manhattan world structure. We propose a new method for scene structure recovery from a single image. This method is based on the line extraction for omnidirectional images, line classification, and vanishing points estimation combined with a new hierarchical expansion procedure for detecting floor and wall boundaries. Each single omnidirectional image independently provides a useful hypothesis of the 3D scene structure. In order to enhance the robustness and accuracy of this single image-based hypothesis, we extend this estimation with a new homography-based procedure applied to the various hypotheses obtained along the sequence of consecutive images. A key point in this contribution is the use of geometrical constraints for computing the homographies from a single line of the floor. The homography parametrization proposed allows the design of a matching-free method for spatial layout propagation along a sequence of images. Experimental results show single image layout recovery performance and the improvement obtained with the propagation of the hypothesis through the image sequence.     

Preliminary coarse image registration by using straight lines found on them for constructing super resolution mosaics and 3D scene recovery

An algorithm of coarse image registration of a 3D scene taken from different camera perspectives is proposed. The algorithm uses information on geometrical parameters of straight lines found on the images and on distribution of color and/or brightness around these lines. Colors are taken into account by using the fuzzy logic technique. The result of the algorithm operation is a planar projective transformation (planar homography) matching approximately the images. In order to use the technique in algorithms of 3D scene reconstruction, an estimate of size of the window used for searching correspondent points after the coarse image registration is obtained.     

Iterative learning control for constrained linear systems

This article considers iterative learning control (ILC) for linear systems with convex control input constraints. First, the constrained ILC problem is formulated in a novel successive projection framework. Then, based on this projection method, two algorithms are proposed to solve this constrained ILC problem. The results show that, when perfect tracking is possible, both algorithms can achieve perfect tracking. The two algorithms differ, however, in that one algorithm needs much less computation than the other. When perfect tracking is not possible, both algorithms can exhibit a form of practical convergence to a ‘best approximation’. The effect of weighting matrices on the performance of the algorithms is also discussed and finally, numerical simulations are given to demonstrate the effectiveness of the proposed methods.     

基于帧间差分和边缘差分的遗留物检测算法

遗留物检测是智能视频监控系统的核心功能,遗留物一般较小,所处环境复杂,传统的运动目标检测算法直接用于遗留物检测效果一般.提出了一种基于帧间差分与边缘差分的遗留物检测算法,首先进行帧间差分得到运动目标区域,然后将当前帧图像和前一帧的背景图像进行边缘差分运算得到运动目标的边缘,融合二次差分的结果即可得到运动目标的完整轮廓特征,最终通过判断运动目标在场景中的滞留时间是否达到或超过报警系统设置的阈值来标示遗留物,供智能视频监控系统处理.实验结果证明该算法实时性好且识别率较高.     

Visual map‐less navigation based on homographies

We introduce a method for autonomous robot navigation based on homographies computed between the current image and images taken in a previous teaching phase with a monocular vision system. The features used to estimate the homography are vertical lines automatically extracted and matched. From homography, the underlying motion correction between the reference path and the current robot location is computed. The proposed method, which uses a sole calibration parameter, has turned out to be especially useful to correct heading and lateral displacement, which are critical in systems based on odometry. We have tested the proposal in simulation, and with real images. Besides, the visual system has been integrated into an autonomous wheelchair for handicapped, working in real time with robustness. © 2005 Wiley Periodicals, Inc.     

Using pedestrians walking on uneven terrains for camera calibration

A calibrated camera is essential for computer vision systems: the prime reason being that such a camera acts as an angle measuring device. Once the camera is calibrated, applications like three-dimensional reconstruction or metrology or other applications requiring real world information from the video sequences can be envisioned. Motivated by this, we address the problem of calibrating multiple cameras, with an overlapping field of view observing pedestrians in a scene walking on an uneven terrain. This problem of calibration on an uneven terrain has so far not been addressed in the vision community. We automatically estimate vertical and horizontal vanishing points by observing pedestrians in each camera and use the corresponding vanishing points to estimate the infinite homography existing between the different cameras. This homography provides constraints on intrinsic (or interior) camera parameters while also enabling us to estimate the extrinsic (or exterior) camera parameters. We test the proposed method on real as well as synthetic data, in addition to motion capture dataset and compare our results with the state of the art.     

单幅图像测量的一种新方法

本文方法表明,由一空间平面(参考平面)与其图像间的单应性矩阵(Homography)不仅 此参考平面上的距离可以测量,而且可以测量与此参考平面垂直的平面上的距离.同时,分别位 于两平面上的点间的距离也可以测量.这样就可以得到关于场景的更多的几何信息,此结果是在 前人的基础上又向前跨了一步.另外,本文提出一种新的基于平面单应性矩阵的摄像机标定方 法.模拟和真实图像试验均表明本文方法是可行的,并得到了令人满意的结果.