Multilinear Factorizations for Multi-Camera Rigid Structure from Motion Problems

Camera networks have gained increased importance in recent years. Existing approaches mostly use point correspondences between different camera views to calibrate such systems. However, it is often difficult or even impossible to establish such correspondences. But even without feature point correspondences between different camera views, if the cameras are temporally synchronized then the data from the cameras are strongly linked together by the motion correspondence: all the cameras observe the same motion. The present article therefore develops the necessary theory to use this motion correspondence for general rigid as well as planar rigid motions. Given multiple static affine cameras which observe a rigidly moving object and track feature points located on this object, what can be said about the resulting point trajectories? Are there any useful algebraic constraints hidden in the data? Is a 3D reconstruction of the scene possible even if there are no point correspondences between the different cameras? And if so, how many points are sufficient? Is there an algorithm which warrants finding the correct solution to this highly non-convex problem? This article addresses these questions and thereby introduces the concept of low-dimensional motion subspaces. The constraints provided by these motion subspaces enable an algorithm which ensures finding the correct solution to this non-convex reconstruction problem. The algorithm is based on multilinear analysis, matrix and tensor factorizations. Our new approach can handle extreme configurations, e.g. a camera in a camera network tracking only one single point. Results on synthetic as well as on real data sequences act as a proof of concept for the presented insights.     

M2Tracker: A Multi-View Approach to Segmenting and Tracking People in a Cluttered Scene

When occlusion is minimal, a single camera is generally sufficient to detect and track objects. However, when the density of objects is high, the resulting occlusion and lack of visibility suggests the use of multiple cameras and collaboration between them so that an object is detected using information available from all the cameras in the scene.In this paper, we present a system that is capable of segmenting, detecting and tracking multiple people in a cluttered scene using multiple synchronized surveillance cameras located far from each other. The system is fully automatic, and takes decisions about object detection and tracking using evidence collected from many pairs of cameras. Innovations that help us tackle the problem include a region-based stereo algorithm capable of finding 3D points inside an object knowing only the projections of the object (as a whole) in two views, a segmentation algorithm using bayesian classification and the use of occlusion analysis to combine evidence from different camera pairs.The system has been tested using different densities of people in the scene. This helps us determine the number of cameras required for a particular density of people. Experiments have also been conducted to verify and quantify the efficacy of the occlusion analysis scheme.     

点状特征柔性物体三维运动捕获方法

针对具有点状特征的柔性物体,提出了一种三维运动捕获方法.首先,该方法利用两个标定的高速摄像机拍摄柔性物体的运动视频,并对图像进行立体校正;然后,采用DOG （Difference Of Gaussian）算法获取点状特征的位置,并提取特征点极值;其次,在一定范围的窗口上搜索匹配对,匹配左右图像的特征点;再次,通过三角测量法进行三维重建;最后,利用搜索策略进行时间序列上的匹配,实现动态柔性物体的三维运动捕获,并计算空间坐标、速度、加速度参数.实验结果表明,相比于采用sift算法匹配特征点捕获柔性运动物体的方法,本方法精度更高.     

基于关键点及光流的平面物体跟踪算法

为提高复杂场景下基于关键点的平面物体跟踪算法的鲁棒性,提出一种融合光流的平面物体跟踪算法。检测目标物体与输入图像的关键点及其对应描述符,由最近邻匹配方法构建目标与图像间关键点匹配集合,通过光流法构建相邻两张图像间关键点的对应关系,将已构建的关键点匹配集合与基于光流的对应关系通过加权平均的策略进行融合,得出修正的关键点匹配集合,根据关键点匹配估计目标物体在当前图像的单应性变换矩阵,从而完成目标跟踪。在POT数据集上的实验结果表明,与SIFT、FERNS等算法相比,在校正误差阈值为5时,该算法在所有图像序列上的平均跟踪精度达到66.67%,具有较好的跟踪性能。     

Robust active stereo vision using Kullback-Leibler divergence

Active stereo vision is a method of 3D surface scanning involving the projecting and capturing of a series of light patterns where depth is derived from correspondences between the observed and projected patterns. In contrast, passive stereo vision reveals depth through correspondences between textured images from two or more cameras. By employing a projector, active stereo vision systems find correspondences between two or more cameras, without ambiguity, independent of object texture. In this paper, we present a hybrid 3D reconstruction framework that supplements projected pattern correspondence matching with texture information. The proposed scheme consists of using projected pattern data to derive initial correspondences across cameras and then using texture data to eliminate ambiguities. Pattern modulation data are then used to estimate error models from which Kullback-Leibler divergence refinement is applied to reduce misregistration errors. Using only a small number of patterns, the presented approach reduces measurement errors versus traditional structured light and phase matching methodologies while being insensitive to gamma distortion, projector flickering, and secondary reflections. Experimental results demonstrate these advantages in terms of enhanced 3D reconstruction performance in the presence of noise, deterministic distortions, and conditions of texture and depth contrast.     

Automatic high-resolution optoelectronic photogrammetric 3D surface geometry acquisition system

A fast, high-resolution, automatic, non-contact 3D surface geometry measuring system using a photogrammetric optoelectronic technique based on lateral-photoeffect diode detectors has been developed. Designed for the acquisition of surface geometries such as machined surfaces, biological surfaces, and deformed parts, the system can be used in design, manufacturing, inspection, and range finding. A laser beam is focused and scanned onto the surface of the object to be measured. Two cameras in stereo positions capture the reflected light from the surface at 10 kHz. Photogrammetric triangulation quickly transforms the pair of 2D signals created by the camera detectors into 3D coordinates of the light spot. Because only one small spot on the object is illuminated at a time, the stereo correspondence problem is solved in real time. The resolution is determined by a 12-bit A/D converter and can be improved up to 25 60025 600 by oversampling. The irregular 3D data can be regularized for use with image-based algorithms. Received: 8 October 1996 / Accepted: 3 February 1997     

Rigidity checking of 3D point correspondences under perspectiveprojection

An algorithm is described which rapidly verifies the potential rigidity of three-dimensional point correspondences from a pair of two-dimensional views under perspective projection. The output of the algorithm is a simple yes or no answer to the question “Could these corresponding points from two views be the projection of a rigid configuration?” Potential applications include 3D object recognition from a single previous view and correspondence matching for stereo or motion over widely separated views. The rigidity checking problem is different from the structure-from-motion problem because it is often the case that two views cannot provide an accurate structure-from-motion estimate due to ambiguity and ill conditioning, whereas it is still possible to give an accurate yes/no answer to the rigidity question. Rigidity checking verifies point correspondences using 3D recovery equations as a matching condition. The proposed algorithm improves upon other methods that fall under this approach because it works with as few as six corresponding points under full perspective projection, handles correspondences from widely separated views, makes full use of the disparity of the correspondences, and is integrated with a linear algorithm for 3D recovery due to Kontsevich (1993). Results are given for experiments with synthetic and real image data. A complete implementation of this algorithm is being made publicly available     

采用极线约束与圆窗口匹配的立体视觉检测

首先建立了一般形式的双目立体视觉的极线约束方程,使其包含两个独立摄像机的所有内部参数和相对参考坐标系的外部结构参数,针对传统矩形窗口匹配存在的问题,提出了采用沿极线滑动的圆形窗口实现左右图像对应点的匹配。以灰度相关系数和Hu矩不变量作为匹配测度,完成了相交圆管（工件）的立体视觉检测,在进行灰度相关计算时采用了主轴旋转校正技术。实验结果表明：圆窗口下计算的灰度相关性和Hu矩不变量的相似测度都好于矩形窗口,对获得的工件表面三维坐标进行拟合重建与实际尺寸相比误差很小,说明文中建议的极线约束下基于圆窗口的匹配技术用于立体视觉检测是有效的。     

Binocular matching constraints from motion

Structure from motion and structure from stereo are two vision cues for achieving 3D reconstruction. The two cues have complementary strengths; while 3D reconstruction is accurate but correspondence establishment is difficult in the stereo cue, the reverse is true in the motion cue. This paper addresses how to combine the two cues when a stereo pair of cameras are available to capture image data for 3D reconstruction. The work is distinct in that, in contrast with the previous ones, it is not to exploit the redundancy in the image data for boosting the reconstruction accuracy, but to make the two vision cues complementary, preserving their strengths and avoiding their weaknesses. A mechanism is introduced that allows dense motion correspondences in the two separate image streams be transferred to dense binocular correspondences across the image streams, so that 3D can be reconstructed from the latter and accurate reconstruction is possible even with short motions of the stereo rig. Both the stereo correspondences and the motion of the stereo rig are assumed to be unknown in this work. Experiments involving real image data are presented to indicate the feasibility and robustness of the approach.     

基于KCF跟踪算法的目标轨迹记录系统

为了确保跟踪算法能够实时跟踪上高速移动的目标并且记录目标的三维坐标.本系统使用了一种基于KCF（Kernelized Correlation Filters）的高速跟踪算法来保证系统能够跟踪到移动速度较快的目标.首先,使用KCF跟踪算法来跟踪目标;然后,利用ORB特征点检测来计算目标特征点从而找到多摄像机中对应的点,找到对应点之后利用多摄像机的三维重建原理计算出每一帧中目标物体的三维坐标点;最后,用多项式对每一帧运动轨迹的离散点进行拟合得到最终的运行轨迹.实验结果证明该算法能够有效跟踪目标,整个系统能够满足实际的需求.     

Extracting 3D facial animation parameters from multiview video clips

We propose an accurate and inexpensive procedure that estimates 3D facial motion parameters from mirror-reflected multiview video clips. We place two planar mirrors near a subject's cheeks and use a single camera to simultaneously capture a marker's front and side view images. We also propose a novel closed-form linear algorithm to reconstruct 3D positions from real versus mirrored point correspondences in an uncalibrated environment. Our computer simulations reveal that exploiting mirrors' various reflective properties yields a more robust, accurate, and simpler 3D position estimation approach than general-purpose stereo vision methods that use a linear approach or maximum-likelihood optimization. Our experiments show a root mean square (RMS) error of less than 2 mm in 3D space with only 20-point correspondences. For semiautomatic 3D motion tracking, we use an adaptive Kalman predictor and filter to improve stability and infer the occluded markers' position. Our approach tracks more than 50 markers on a subject's face and lips from 30-frame-per-second video clips. We've applied the facial motion parameters estimated from the proposed method to our facial animation system.     

A noniterative greedy algorithm for multiframe point correspondence

This work presents a framework for finding point correspondences in monocular image sequences over multiple frames. The general problem of multiframe point correspondence is NP-hard for three or more frames. A polynomial time algorithm for a restriction of this problem is presented and is used as the basis of the proposed greedy algorithm for the general problem. The greedy nature of the proposed algorithm allows it to be used in real-time systems for tracking and surveillance, etc. In addition, the proposed algorithm deals with the problems of occlusion, missed detections, and false positives by using a single noniterative greedy optimization scheme and, hence, reduces the complexity of the overall algorithm as compared to most existing approaches where multiple heuristics are used for the same purpose. While most greedy algorithms for point tracking do not allow the entry and exit of the points from the scene, this is not a limitation for the proposed algorithm. Experiments with real and synthetic data over a wide range of scenarios and system parameters are presented to validate the claims about the performance of the proposed algorithm.     

一种结合光流追踪的双目视觉里程计算法

针对特征点法视觉里程计频繁计算和匹配描述子导致系统实时性能变差的问题,提出一种结合光流追踪的双目视觉里程计算法;首先进行初始化,生成初始的关键帧和地图点,随后在追踪线程中使用光流追踪特征点获取匹配关系,计算并优化相机位姿;满足生成关键帧的条件后,将当前帧设置为关键帧,提取图像的ORB特征点,并使用描述子匹配获取与上一关键帧特征点的匹配关系,三角化生成新的地图点;最后在优化线程中将新的关键帧和地图点使用滑动窗口算法进行优化,剔除冗余关键帧和地图点;在KITTI数据集上的实验结果表明,所提出的算法轨迹误差与双目模式下的ORB-SLAM3算法处于同一水平,同时实时性能有大幅度提高,追踪每一帧图像的平均时间从52ms降至16ms,在保证高精度的情况下运行速度大大提高,具有较高的实用价值.     

Joint radiometric calibration and feature tracking system with an application to stereo

To capture the full brightness range of natural scenes, cameras automatically adjust the exposure value which causes the brightness of scene points to change from frame to frame. Given such a video sequence, we introduce a system for tracking features and estimating the radiometric response function of the camera and the exposure difference between frames simultaneously. We model the global and nonlinear process that is responsible for the changes in image brightness rather than adapting to the changes locally and linearly which makes our tracking more robust to the change in brightness. We apply our system to perform structure-from-motion and stereo to reconstruct a texture-mapped 3D surface from a video taken in a high dynamic range environment.     

A Maximum Likelihood Stereo Algorithm

A stereo algorithm is presented that optimizes a maximum likelihood cost function. The maximum likelihood cost function assumes that corresponding features in the left and right images are normally distributed about a common true value and consists of a weighted squared error term if two features are matched or a (fixed) cost if a feature is determined to be occluded. The stereo algorithm finds the set of correspondences that maximize the cost function subject to ordering and uniqueness constraints. The stereo algorithm is independent of the matching primitives. However, for the experiments described in this paper, matching is performed on the $cf4$individual pixel intensities.$cf3$ Contrary to popular belief, the pixel-based stereo appears to be robust for a variety of images. It also has the advantages of (i) providing adensedisparity map, (ii) requiringnofeature extraction, and (iii)avoidingthe adaptive windowing problem of area-based correlation methods. Because feature extraction and windowing are unnecessary, a very fast implementation is possible. Experimental results reveal that good stereo correspondences can be found using only ordering and uniqueness constraints, i.e., withoutlocalsmoothness constraints. However, it is shown that the original maximum likelihood stereo algorithm exhibits multiple global minima. The dynamic programming algorithm is guaranteed to find one, but not necessarily the same one for each epipolar scanline, causing erroneous correspondences which are visible as small local differences between neighboring scanlines. Traditionally, regularization, which modifies the original cost function, has been applied to the problem of multiple global minima. We developed several variants of the algorithm that avoid classical regularization while imposing several global cohesiveness constraints. We believe this is a novel approach that has the advantage of guaranteeing that solutions minimize the original cost function and preserve discontinuities. The constraints are based on minimizing the total number of horizontal and/or vertical discontinuities along and/or between adjacent epipolar lines, and local smoothing is avoided. Experiments reveal that minimizing the sum of the horizontal and vertical discontinuities provides the most accurate results. A high percentage of correct matches and very little smearing of depth discontinuities are obtained. An alternative to imposing cohesiveness constraints to reduce the correspondence ambiguities is to use more than two cameras. We therefore extend the two camera maximum likelihood toNcameras. TheN-camera stereo algorithm determines the “best” set of correspondences between a given pair of cameras, referred to as the principal cameras. Knowledge of the relative positions of the cameras allows the 3D point hypothesized by an assumed correspondence of two features in the principal pair to be projected onto the image plane of the remainingN− 2 cameras. TheseN− 2 points are then used to verify proposed matches. Not only does the algorithm explicitly model occlusion between features of the principal pair, but the possibility of occlusions in theN− 2 additional views is also modeled. Previous work did not model this occlusion process, the benefits and importance of which are experimentally verified. Like other multiframe stereo algorithms, the computational and memory costs of this approach increase linearly with each additional view. Experimental results are shown for two outdoor scenes. It is clearly demonstrated that the number of correspondence errors is significantly reduced as the number of views/cameras is increased.     

结构上下文:一种新的物体类别描述符

局部描述符(如SIFT)方法能够将图像中关键点的局部表观信息作为图像的特征,具有旋转不变性、尺度变换不变性、仿射不变性等性质,被广泛应用于物体分类、物体识别、图像匹配等领域。但是,它存在一个重要缺陷:只能描述物体的局部特征,忽略了整个物体的构造,而这在表示物体时是非常重要的。设计了一个新的"结构上下文"局部描述符,通过当前关键点和其他关键点间的空间拓扑结构关系描述各个关键点的特征。实验证明这种描述符在描述相同物体种类时特别有效。     

3D Slit Scanning with Planar Constraints★

We present a planarity constraint and a novel three‐dimensional (3D) point reconstruction algorithm for a multiview laser range slit scanner. The constraint is based on the fact that all observed points on a projected laser line lie on the same plane of laser light in 3D. The parameters of the plane of laser light linearly parametrize a homography between a pair of images of the laser points. This homography can be recovered from point correspondences derived from epipolar geometry. The use of the planar constraint reduces outliers in the reconstruction and allows for the reconstruction of points seen in only one view. We derive an optimal reconstruction of points subject to the planar constraint and compare the accuracy to the suboptimal approach in prior work. We also construct a catadioptric stereo rig with high quality optical components to remove error due to camera synchronization and non‐uniform laser projection. The reconstruction results are compared to prior work that uses inexpensive optics and two cameras.