Robust multi-view L2 triangulation via optimal inlier selection and 3D structure refinement

This paper presents a new robust approach for multi-view L₂ triangulation based on optimal inlier selection and 3D structure refinement. The proposed method starts with estimating the scale of noise in image measurements, which affects both the quantity and the accuracy of reconstructed 3D points but is overlooked or ignored in existing triangulation pipelines. A new residual-consensus scheme within which the uncertainty of epipolar transfer is analytically characterized by deriving its closed-form covariance is developed to robustly estimate the noise scale. Different from existing robust triangulation pipelines, the issue of outliers is addressed by directly searching for the optimal 3D points that are within either the theoretical correct error bounds calculated by second-order cone programming (SOCP) or the efficiently calculated approximate ranges. In particular, both the inlier selection and 3D structure refinement are realized in an optimal fashion using Differential Evolution (DE) optimization which allows flexibility in the design of the objective function. To validate the performance of the proposed method, extensive experiments using both synthetic data and real image sequences were carried out. Comparing with state-of-the-art robust triangulation strategies, the proposed method can consistently identify more reliable inliers and hence, reconstruct more unambiguous 3D points with higher accuracy than existing methods.     

Recursive-Batch Estimation of Motion and Structure from Monocular Image Sequences

This paper addresses the issue of optimal motion and structure estimation from monocular image sequences of a rigid scene. The new method has the following characteristics: (1) the dimension of the search space in the nonlinear optimization is drastically reduced by exploiting the relationship between structure and motion parameters; (2) the degree of reliability of the observations and estimates is effectively taken into account; (3) the proposed formulation allows arbitrary interframe motion; (4) the information about the structure of the scene, acquired from previous images, is systematically integrated into the new estimations; (5) the integration of multiple views using this method gives a large 2.5D visual map, much larger than that covered by any single view. It is shown also that the scale factor associated with any two consecutive images in a monocular sequence is determined by the scale factor of the first two images. Our simulation results and experiments with long image sequences of real world scenes indicate that the optimization method developed in this paper not only greatly reduces the computational complexity but also substantially improves the motion and structure estimates over those produced by the linear algorithms.     

A loop-consistency measure for dense correspondences in multi-view video

Many applications in computer vision and computer graphics require dense correspondences between images of multi-view video streams. Most state-of-the-art algorithms estimate correspondences by considering pairs of images. However, in multi-view videos, several images capture nearly the same scene. In this article we show that this redundancy can be exploited to estimate more robust and consistent correspondence fields. We use the multi-video data structure to establish a confidence measure based on the consistency of the correspondences in a loop of three images. This confidence measure can be applied after flow estimation is terminated to find the pixels for which the estimate is reliable. However, including the measure directly into the estimation process yields dense and highly accurate correspondence fields. Additionally, application of the loop consistency confidence measure allows us to include sparse feature matches directly into the dense optical flow estimation. With the confidence measure, spurious matches can be successfully suppressed during optical flow estimation while correct matches contribute to increase the accuracy of the flow.     

小场景交互式稠密三维重建系统

近年来,随着 GPU 技术的深入发展和并行算法的日益成熟,使得实时三维重建成 为可能。文中实现了一种针对小场景的交互式稠密三维重建系统,此系统借助先进的移动跟踪 技术,可以准确地估计相机的即时位置。提出了一种改进的多视深度生成算法,在 GPU 加速下 能够实时计算场景的深度。改进算法中的亚像素级的半全局匹配代价累积提高了多视立体匹配 的精度,并结合全局优化的方法计算出了准确的场景深度信息。深度图被转换为距离场,使用 全局优化的直方图压缩融合算法和并行的原始对偶算法实现了深度的实时融合。实验结果证明 了重建系统的可行性和重建算法的正确性。     

Structure from Motion: Beyond the Epipolar Constraint

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images.The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the least varying estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.     

3D modeling of multiple-object scenes from sets of images

This paper proposes a new approach for multi-object 3D scene modeling. Scenes with multiple objects are characterized by object occlusions under several views, complex illumination conditions due to multiple reflections and shadows, as well as a variety of object shapes and surface properties. These factors raise huge challenges when attempting to model real 3D multi-object scene by using existing approaches which are designed mainly for single object modeling. The proposed method relies on the initialization provided by a rough 3D model of the scene estimated from the given set of multi-view images. The contributions described in this paper consists of two new methods for identifying and correcting errors in the reconstructed 3D scene. The first approach corrects the location of 3D patches from the scene after detecting the disparity between pairs of their projections into images. The second approach is called shape-from-contours and identifies discrepancies between projections of 3D objects and their corresponding contours, segmented from images. Both unsupervised and supervised segmentations are used to define the contours of objects.     

Fast variational multi-view segmentation through backprojection of spatial constraints

We present a variational segmentation method which exploits color, edge and spatial information between an arbitrary number of views. In contrast to purely image based information like color and gradient, spatial consistency is a new cue for segmentation, which originates from the field of 3D reconstruction. We show that this cue can be easily integrated in a variational formulation and allows pixel-accurate segmentation, even for objects which are hard to segment. The use of inherently parallel algorithms and the implementation on modern GPUs allows us to apply this method to semi-supervised and completely automatic settings. On publicly available datasets we show that our method is faster and more accurate than the state of the art. The successful applications within a catadioptric measurement system and multi-view background subtraction shows its practical relevance.     

3D Archive System for Traditional Performing Arts

We developed a 3D archive system for Japanese traditional performing arts. The system generates sequences of 3D actor models of the performances from multi-view video by using a graph-cuts algorithm and stores them with CG background models and related information. The system can show a scene from any viewpoint as follows; the 3D actor model is integrated with the background model and the integrated model is projected to a viewpoint that the user indicates with a viewpoint controller.     

Compression of Probabilistic Volumetric Models using multi-resolution scene flow

This paper presents a novel method to estimate dense scene flow using volumetric and probabilistic 3-d models. The method first reconstructs 3-d models at each time step using images synchronously captured from multiple views. Then, the 3-d motion between two consecutive 3-d models is estimated using a formulation that is the analog of Horn and Schunck's optical flow method. This particular choice of 3-d model representation allows estimating highly dense scene flow results, tracking of surfaces undergoing topological change and reliably recovering large motion displacements. The benefits of the method and the accuracy of 3-d flow results are demonstrated on recent multi-view datasets. The second goal of this work is to compress and reconstruct 3-d scenes at various time points using the estimated flow. A new method of scene warping is proposed that involves partitioning the optical flow field in regions of coherent motion which are subsequently parametrized by affine transformations. The compression objective of this work is achieved by the low storage requirements of the affine parameters that describe the optical flow field and the efficient reconstruction method through warping.     

基于多视角的高精度三维人脸重建

提出了一种多阶段优化的方法来解决基于多视角图片在未知姿态、表情以及光照条件下的高精度三维人脸重建问题.首先,通过重新渲染合成的方法将参数化模型拟合到输入的多视角图片,然后在纹理域上求解一个光流问题来获取不同视角之间的对应关系.通过对应关系可以恢复出人脸的点云,并利用基于明暗恢复几何的方法来恢复人脸细节.在真实数据以及合成数据下的实验结果表明,文中方法能够恢复出带有几何细节的高精度的三维人脸模型,并且提高了现有方法的重建精度.     

People Watching: Human Actions as a Cue for Single View Geometry

We present an approach which exploits the coupling between human actions and scene geometry to use human pose as a cue for single-view 3D scene understanding. Our method builds upon recent advances in still-image pose estimation to extract functional and geometric constraints on the scene. These constraints are then used to improve single-view 3D scene understanding approaches. The proposed method is validated on monocular time-lapse sequences from YouTube and still images of indoor scenes gathered from the Internet. We demonstrate that observing people performing different actions can significantly improve estimates of 3D scene geometry.     

Non-rigid 3D shape tracking from multiview video

We present a fast and efficient non-rigid shape tracking method for modeling dynamic 3D objects from multiview video. Starting from an initial mesh representation, the shape of a dynamic object is tracked over time, both in geometry and topology, based on multiview silhouette and 3D scene flow information. The mesh representation of each frame is obtained by deforming the mesh representation of the previous frame towards the optimal surface defined by the time-varying multiview silhouette information with the aid of 3D scene flow vectors. The whole time-varying shape is then represented as a mesh sequence which can efficiently be encoded in terms of restructuring and topological operations, and small-scale vertex displacements along with the initial model. The proposed method has the ability to deal with dynamic objects that may undergo non-rigid transformations and topological changes. The time-varying mesh representations of such non-rigid shapes, which are not necessarily of fixed connectivity, can successfully be tracked thanks to restructuring and topological operations employed in our deformation scheme. We demonstrate the performance of the proposed method both on real and synthetic sequences.     

On 3-D scene flow and structure recovery from multiview image sequences

Two novel systems computing dense three-dimensional (3-D) scene flow and structure from multiview image sequences are described in this paper. We do not assume rigidity of the scene motion, thus allowing for nonrigid motion in the scene. The first system, integrated model-based system (IMS), assumes that each small local image region is undergoing 3-D affine motion. Non-linear motion model fitting based on both optical flow constraints and stereo constraints is then carried out on each local region in order to simultaneously estimate 3-D motion correspondences and structure. The second system is based on extended gradient-based system (EGS), a natural extension of two-dimensional (2-D) optical flow computation. In this method, a new hierarchical rule-based stereo matching algorithm is first developed to estimate the initial disparity map. Different available constraints under a multiview camera setup are further investigated and utilized in the proposed motion estimation. We use image segmentation information to adopt and maintain the motion and depth discontinuities. Within the framework for EGS, we present two different formulations for 3-D scene flow and structure computation. One formulation assumes that initial disparity map is accurate, while the other does not. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3-D motion and structure recovery schemes. Empirical comparison between IMS and EGS is also reported.     

Combining 3D flow fields with silhouette-based human motion capture for immersive video

In recent years, the convergence of computer vision and computer graphics has put forth a new field of research that focuses on the reconstruction of real-world scenes from video streams. To make immersive 3D video reality, the whole pipeline spanning from scene acquisition over 3D video reconstruction to real-time rendering needs to be researched. In this paper, we describe latest advancements of our system to record, reconstruct and render free-viewpoint videos of human actors. We apply a silhouette-based non-intrusive motion capture algorithm making use of a 3D human body model to estimate the actor’s parameters of motion from multi-view video streams. A renderer plays back the acquired motion sequence in real-time from any arbitrary perspective. Photo-realistic physical appearance of the moving actor is obtained by generating time-varying multi-view textures from video. This work shows how the motion capture sub-system can be enhanced by incorporating texture information from the input video streams into the tracking process. 3D motion fields are reconstructed from optical flow that are used in combination with silhouette matching to estimate pose parameters. We demonstrate that a high visual quality can be achieved with the proposed approach and validate the enhancements caused by the the motion field step.     

结合改进密集模块深度估计网络和多视几何的视觉里程计

以多视图几何原理为基础,有效结合卷积神经网络进行图像深度估计和匹配筛选,构造无监督单目视觉里程计方法.针对主流深度估计网络易丢失图像浅层特征的问题,构造一种基于改进密集模块的深度估计网络,有效地聚合浅层特征,提升图像深度估计精度.里程计利用深度估计网络精确预测单目图像深度,利用光流网络获得双向光流,通过前后光流一致性原则筛选高质量匹配.利用多视图几何原理和优化方式求解获得初始位姿和计算深度,并通过特定的尺度对齐原则得到全局尺度一致的6自由度位姿.同时,为了提高网络对场景细节和弱纹理区域的学习能力,将基于特征图合成的特征度量损失结合到网络损失函数中.在KITTI Odometry数据集上进行实验验证,不同阈值下的深度估计取得了85.9%、95.8%、97.2%的准确率.在09和10序列上进行里程计评估,绝对轨迹误差在0.007 m.实验结果验证了所提出方法的有效性和准确性,表明其在深度估计和视觉里程计任务上的性能优于现有方法.     

A Variational Framework for Structure from Motion in Omnidirectional Image Sequences

We address the problem of depth and ego-motion estimation from omnidirectional images. We propose a correspondence-free structure-from-motion problem for sequences of images mapped on the 2-sphere. A novel graph-based variational framework is first proposed for depth estimation between pairs of images. The estimation is cast as a TV-L1 optimization problem that is solved by a fast graph-based algorithm. The ego-motion is then estimated directly from the depth information without explicit computation of the optical flow. Both problems are finally addressed together in an iterative algorithm that alternates between depth and ego-motion estimation for fast computation of 3D information from motion in image sequences. Experimental results demonstrate the effective performance of the proposed algorithm for 3D reconstruction from synthetic and natural omnidirectional images.     

Adaptive partial graph learning and fusion for incomplete multi-view clustering

Most of existing multi-view clustering methods assume that different feature views of data are fully observed. However, it is common that only portions of data features can be obtained in many practical applications. The presence of incomplete feature views hinders the performance of the conventional multi-view clustering methods to a large extent. Recently proposed incomplete multi-view clustering methods often focus on directly learning a common representation or a consensus affinity similarity graph from available feature views while ignore the valuable information hidden in the missing views. In this study, we present a novel incomplete multi-view clustering method via adaptive partial graph learning and fusion (APGLF), which can capture the local data structure of both within-view and cross-view. Specifically, we use the available data of each view to learn a corresponding view-specific partial graph, in which the within-view local structure can be well preserved. Then we design a cross-view graph fusion term to learn a consensus complete graph for different views, which can take advantage of the complementary information hidden in the view-specific partial graphs learned from incomplete views. In addition, a rank constraint is imposed on the graph Laplacian matrix of the fused graph to better recover the optimal cluster structure of original data. Therefore, APGLF integrates within-view partial graph learning, cross-view partial graph fusion and cluster structure recovering into a unified framework. Experiments on five incomplete multi-view data sets are conducted to validate the efficacy of APGLF when compared with eight state-of-the-art methods.     

面向沉浸式流场可视化的多视图数据管理方法

由于沉浸式环境下的三维交互方式对二维界面操作不够友好,使得依赖于二维列表界面的流场数据管理任务变得复杂且低效。为了实现沉浸式虚拟环境下对流场数据高效的组织和管理,增强用户对流场空间信息的理解,提出一种基于多视图结合交互的沉浸式流场可视化数据块管理方法。该方法构建了一个三维小视图用于提供场景概览,并通过“主视图交互+小视图辅助“”小视图交互+主视图反馈”等多种多视图组合交互方式完成对多块流场数据的管理交互操作。最后构建了一个基于手势的沉浸式流场可视化系统,定义多项交互任务,从学习时长、完成时间和用户反馈几个方面对比了多视图方法和传统交互方法差异。实验结果表明,相比于传统交互方法,多视图方法可以显著提高数据管理任务的效率。     

Integrated 3-D analysis and analysis-guided synthesis of flightimage sequences

This paper is concerned with three-dimensional (3D) analysis, and analysis-guided syntheses, of images showing 3-D motion of an observer relative to a scene. There are two objectives of the paper. First, it presents an approach to recovering 3D motion and structure parameters from multiple cues present in a monocular image sequence, such as point features, optical flow, regions, lines, texture gradient, and vanishing line. Second, it introduces the notion that the cues that contribute the most to 3-D interpretation are also the ones that would yield the most realistic synthesis, thus suggesting an approach to analysis guided 3-D representation. For concreteness, the paper focuses on flight image sequences of a planar, textured surface. The integration of information in these diverse cues is carried out using optimization. For reliable estimation, a sequential batch method is used to compute motion and structure. Synthesis is done by using (i) image attributes extracted from the image sequence, and (ii) simple, artificial image attributes which are not present in the original images. For display, real and/or artificial attributes are shown as a monocular or a binocular sequence. Performance evaluation is done through experiments with one synthetic sequence, and two real image sequences digitized from a commercially available video tape and a laserdisc. The attribute based representation of these sequences compressed their sizes by 502 and 367. The visualization sequence appears very similar to the original sequence in informal, monocular as well as stereo viewing on a workstation monitor     

基于智能的配电网电力大数据三维场景可视化分析

配电网电力大数据的三维场景重构是实现数据优化挖掘的关键,提出基于人工智能的配电网电力大数据三维场景可视化分析方法。建立配电网电力大数据三维场景的网格分布结构模型,并进行配电网电力大数据三维场景实时数据监测,根据监测结果进行配电网电力大数据的统计特征分析,对配电网电力大数据三维场景实时数据采用信息融合和模糊层析性分析方法进行信息融合和自适应调度,提取配电网电力大数据的三维可视化分布特征量,采用视觉特征重构技术,实现对配电网电力大数据三维场景可视化重构,在人工智能算法控制下提高电力大数据三维场景可视化重构的精度。仿真结果表明,采用该方法进行配电网电力大数据三维场景可视化重构的精度较高,提高了配电网电力大数据挖掘的效能。