Image-Based Modeling by Joint Segmentation

The paper first traces the image-based modeling back to feature tracking and factorization that have been developed in the group led by Kanade since the eighties. Both feature tracking and factorization have inspired and motivated many important algorithms in structure from motion, 3D reconstruction and modeling. We then revisit the recent quasi-dense approach to structure from motion. The key advantage of the quasi-dense approach is that it not only delivers the structure from motion in a robust manner for practical modeling purposes, but also it provides a cloud of sufficiently dense 3D points that allows the objects to be explicitly modeled. To structure the available 3D points and registered 2D image information, we argue that a joint segmentation of both 3D and 2D is the fundamental stage for the subsequent modeling. We finally propose a probabilistic framework for the joint segmentation. The optimal solution to such a joint segmentation is still generally intractable, but approximate solutions are developed in this paper. These methods are implemented and validated on real data set.     

基于网格变形的从图像重建三维人脸

从图像重建高质量三维人脸一直是计算机视觉和图形学的一个重要研究问题.不同于传统的基于立体匹配的窄基线多视几何和数据驱动的人脸形变方法,提出一种结合网格变形技术和立体视觉原理的、从图像重建高质量三维人脸模型方法.给定从不同视角拍摄的几幅人脸图像,基于健壮图像特征获得可靠的相机外部参数和稀疏三维点;在此基础上,提出一种结合几何细节保持和图像一致性约束的三维人脸变形算法重建三维人脸,通过对人脸模板的网格变形,使得变形人脸在多幅图像中的可见投影具有一致性的图像颜色强度.基于模板的人脸变形可以有效地解决三维模型成像中的遮挡问题,采用健壮估计法消除噪声、离群点和光照对目标函数收敛性的影响,对目标函数的多次非线性优化求解进一步改进了人脸重建的质量.采用合成人脸图像和真实人脸图像重建三维人脸的实验结果表明,文中算法可以从几幅宽基线图像重建高质量的三维人脸模型.     

Implicit Surface-Based Geometric Fusion

This paper introduces a general purpose algorithm for reliable integration of sets of surface measurements into a single 3D model. The new algorithm constructs a single continuous implicit surface representation which is the zero-set of a scalar field function. An explicit object model is obtained using any implicit surface polygonization algorithm. Object models are reconstructed from both multiple view conventional 2.5D range images and hand-held sensor range data. To our knowledge this is the first geometric fusion algorithm capable of reconstructing 3D object models from noisy hand-held sensor range data.This approach has several important advantages over existing techniques. The implicit surface representation allows reconstruction of unknown objects of arbitrary topology and geometry. A continuous implicit surface representation enables reliable reconstruction of complex geometry. Correct integration of overlapping surface measurements in the presence of noise is achieved using geometric constraints based on measurement uncertainty. The use of measurement uncertainty ensures that the algorithm is robust to significant levels of measurement noise. Previous implicit surface-based approaches use discrete representations resulting in unreliable reconstruction for regions of high curvature or thin surface sections. Direct representation of the implicit surface boundary ensures correct reconstruction of arbitrary topology object surfaces. Fusion of overlapping measurements is performed using operations in 3D space only. This avoids the local 2D projection required for many previous methods which results in limitations on the object surface geometry that is reliably reconstructed. All previous geometric fusion algorithms developed for conventional range sensor data are based on the 2.5D image structure preventing their use for hand-held sensor data. Performance evaluation of the new integration algorithm against existing techniques demonstrates improved reconstruction of complex geometry.     

Match propagation for image-based modeling and rendering

This paper presents a quasi-dense matching algorithm between images based on the match propagation principle. The algorithm starts from a set of sparse seed matches, then propagates to the neighboring pixels by the best-first strategy, and produces a quasi-dense disparity map. The quasi-dense matching aims at broad modeling and visualization applications which rely heavily on matching information. Our algorithm is robust to initial sparse match outliers due to the best-first strategy. It is efficient in time and space as it is only output sensitive. It handles half-occluded areas because of the simultaneous enforcement of newly introduced discrete 2D gradient disparity limit and the uniqueness constraint. The properties of the algorithm are discussed and empirically demonstrated. The quality of quasi-dense matching are validated through intensive real examples.     

Combinatorial manifold mesh reconstruction and optimization from unorganized points with arbitrary topology

In this paper, a novel approach is proposed to reliably reconstruct the geometric shape of a physically existing object based on unorganized point cloud sampled from its boundary surface. The proposed approach is composed of two steps. In the first step, triangle mesh structure is reconstructed as a continuous manifold surface by imposing explicit relationship among the discrete data points. For efficient reconstruction, a growing procedure is employed to build the 2-manifold directly without intermediate 3D representation. Local and global topological operations with ensured completeness and soundness are defined to incrementally construct the 2-manifold with arbitrary topology. In addition, a novel criterion is proposed to control the growing process for ensured geometric integrity and automatic boundary detection with a non-metric threshold. The reconstructed manifold surface captures the object topology with the built-in combinatorial structure and approximates the object geometry to the first order. In the second step, new methods are proposed to efficiently obtain reliable curvature estimation for both the object surface and the reconstructed mesh surface. The combinatorial structure of the triangle mesh is then optimized by changing its local topology to minimize the curvature difference between the two surfaces. The optimized triangle mesh achieves second order approximation to the object geometry and can serve as a basis for many applications including virtual reality, computer vision, and reverse engineering.     

基于序列图像的鲁棒三维重建方法

提出了基于序列图像的鲁棒三维重建方法。首先利用两幅图像的最优参数估计,然后添加新图像并采用稀疏调整,减少图像坐标测量值的最小几何误差。通过对三维结构和摄像机参数进行全局优化处理,以提高重建的鲁棒性。实验结果表明,该方法提高了重建的精度和鲁棒性,并真实地再现了物体的三维模型。     

一种用于三维重建的彩色Sift准稠密匹配算法

针对复杂光照条件下Sift算法对彩色图像匹配能力较差,基于Kubelka-Munk理论,提出了一种适用于未标定图像的准稠密立体匹配算法,有助于更精确地进行三维重建。该算法首先求出彩色图像各个像素的颜色不变量,提取彩色特征点并通过构造彩色Sift特征描述子进行初匹配,采用RANSAC鲁棒算法消除误匹配生成种子点;然后依据视差约束提出一种基于视差梯度均值自适应窗口方法,根据视差梯度均值调整搜索范围;最后采用最优先原则进行区域增长。实验证明,该算法能获得比较满意的匹配效果,是一种有效的用于三维重建的准稠密匹配算法。     

An adaptive normal estimation method for scanned point clouds with sharp features

Normal estimation is an essential task for scanned point clouds in various CAD/CAM applications. Many existing methods are unable to reliably estimate normals for points around sharp features since the neighborhood employed for the normal estimation would enclose points belonging to different surface patches across the sharp feature. To address this challenging issue, a robust normal estimation method is developed in order to effectively establish a proper neighborhood for each point in the scanned point cloud. In particular, for a point near sharp features, an anisotropic neighborhood is formed to only enclose neighboring points located on the same surface patch as the point. Neighboring points on the other surface patches are discarded. The developed method has been demonstrated to be robust towards noise and outliers in the scanned point cloud and capable of dealing with sparse point clouds. Some parameters are involved in the developed method. An automatic procedure is devised to adaptively evaluate the values of these parameters according to the varying local geometry. Numerous case studies using both synthetic and measured point cloud data have been carried out to compare the reliability and robustness of the proposed method against various existing methods.     

Silhouette‐Aware Warping for Image‐Based Rendering

Image‐based rendering (IBR) techniques allow capture and display of 3D environments using photographs. Modern IBR pipelines reconstruct proxy geometry using multi‐view stereo, reproject the photographs onto the proxy and blend them to create novel views. The success of these methods depends on accurate 3D proxies, which are difficult to obtain for complex objects such as trees and cars. Large number of input images do not improve reconstruction proportionally; surface extraction is challenging even from dense range scans for scenes containing such objects. Our approach does not depend on dense accurate geometric reconstruction; instead we compensate for sparse 3D information by variational image warping. In particular, we formulate silhouette‐aware warps that preserve salient depth discontinuities. This improves the rendering of difficult foreground objects, even when deviating from view interpolation. We use a semi‐automatic step to identify depth discontinuities and extract a sparse set of depth constraints used to guide the warp. Our framework is lightweight and results in good quality IBR for previously challenging environments.     

Multiresolution Shape Deformations for Meshes with Dynamic Vertex Connectivity

Multiresolution shape representation is a very effective way to decompose surface geometry into several levels of detail. Geometric modeling with such representations enables flexible modifications of the global shape while preserving the detail information. Many schemes for modeling with multiresolution decompositions based on splines, polygonal meshes and subdivision surfaces have been proposed recently. In this paper we modify the classical concept of multiresolution representation by no longer requiring a global hierarchical structure that links the different levels of detail. Instead we represent the detail information implicitly by the geometric difference between independent meshes. The detail function is evaluated by shooting rays in normal direction from one surface to the other without assuming a consistent tesselation. In the context of multiresolution shape deformation, we propose a dynamic mesh representation which adapts the connectivity during the modification in order to maintain a prescribed mesh quality. Combining the two techniques leads to an efficient mechanism which enables extreme deformations of the global shape while preventing the mesh from degenerating. During the deformation, the detail is reconstructed in a natural and robust way. The key to the intuitive detail preservation is a transformation map which associates points on the original and the modified geometry with minimum distortion. We show several examples which demonstrate the effectiveness and robustness of our approach including the editing of multiresolution models and models with texture.     

基于尺度不变Harris特征的准稠密匹配算法

准稠密匹配是多视图三维重建的重要技术,其性能对重建结果至关重要。针对常用的Sift算法提取的种子点进行准稠密匹配正确率较低、重建效果不佳的问题,提出了一种基于尺度不变Harris角点特征的准稠密匹配算法。该算法首先在图像多尺度空间构造尺度不变Harris特征,并采用余弦距离测度对不同视图进行双向匹配;然后根据稀疏匹配获取种子点,采用最优最先匹配扩散策略进行准稠密扩散;最后采用局部非极大值抑制策略对匹配结果进行重采样。实验表明,本文算法提取的种子点既能够体现场景结构信息,又具有尺度不变特性,用于准稠密匹配能够提高匹配的效果和精度,是一种有效的用于三维重建的准稠密匹配算法。     

基于压缩感知的SAR图像鲁棒编码传输

目的 尽管传统的联合信源信道编码方案可以获得高效的压缩性能,但当信道恶化超过信道编码的纠错能力时会导致解码端重构性能的急剧下降;为此利用压缩感知的民主性提出一种鲁棒的SAR图像编码传输方案,且采用了一系列方法提高该方案的率失真性能。方法 考虑到SAR图像丰富的边缘信息,采用具有更强方向表示能力的方向提升小波变换(DLWT)对SAR图像进行稀疏表示,且为消除压缩感知中恢复非稀疏信号时存在的混叠效应,采用了稀疏滤波方法保证大系数的精确恢复,在解码端采用了高效的Bayesian重建算法获得图像的高性能重建。结果 在同等码率下,与传统的联合信源信道编码方案CCSDS-RS相比,本文方案可以实现更加鲁棒的编码传输,当丢包率达到0.05时,本文方案DSFB-CS获得的重建性能明显要高于CCSDS-RS;与基于Bayesian重建算法TSW-CS的传统方案相比,本文方案可提高峰值信噪比(PSNR)3.9 dB。结论 本文方案DSFB-CS 实现了SAR图像的鲁棒传输,随着丢包率的上升,DSFB-CS获得的重建性能缓慢下降,保证了面对不稳定信道时,解码端可以获得相对稳定的重构图像。     

A robust hybrid image-based modeling system

This paper presents a new robust image-based modeling system for creating high-quality 3D models of complex objects from a sequence of unconstrained photographs. The images can be acquired by a video camera or hand-held digital camera without the need of camera calibration. In contrast to previous methods, we integrate correspondence-based and silhouette-based approaches, which significantly enhances the reconstruction of objects with few visual features (e.g., uni-colored objects) and improves surface smoothness. Our solution uses a mesh segmentation and charting approach in order to create a low-distortion mesh parameterization suitable for objects of arbitrary genus. A high-quality texture is produced by first parameterizing the reconstructed objects using a segmentation and charting approach, projecting suitable sections of input images onto the model, and combining them using a graph-cut technique. Holes in the texture due to surface patches without projecting input images are filled using a novel exemplar-based inpainting method which exploits appearance space attributes to improve patch search, and blends patches using Poisson-guided interpolation. We analyzed the effect of different algorithm parameters, and compared our system with a laser scanning-based reconstruction and existing commercial systems. Our results indicate that our system is robust, superior to other image-based modeling techniques, and can achieve a reconstruction quality visually not discernible from that of a laser scanner.     

3D human pose estimation from image using couple sparse coding

Recent studies have demonstrated that high-level semantics in data can be captured using sparse representation. In this paper, we propose an approach to human body pose estimation in static images based on sparse representation. Given a visual input, the objective is to estimate 3D human body pose using feature space information and geometrical information of the pose space. On the assumption that each data point and its neighbors are likely to reside on a locally linear patch of the underlying manifold, our method learns the sparse representation of the new input using both feature and pose space information and then estimates the corresponding 3D pose by a linear combination of the bases of the pose dictionary. Two strategies for dictionary construction are presented: (i) constructing the dictionary by randomly selecting the frames of a sequence and (ii) selecting specific frames of a sequence as dictionary atoms. We analyzed the effect of each strategy on the accuracy of pose estimation. Extensive experiments on datasets of various human activities show that our proposed method outperforms state-of-the-art methods.     

下视SAR数据3维表面重建

目的 合成孔径雷达(SAR)因成像方法、几何角度等原因使得采集到的数据具有稀疏性及残缺性,如果直接用其进行建模,不能真实地还原物体。针对下视SAR数据的特点,提出一种在建模过程中能够自动修补稀疏及残缺数据的重建方法。方法 首先引入大津法对3维SAR数据进行预处理,然后将2维图像分割方法中的Chan-Vese模型推广应用到下视SAR数据的表面重建中,在初始表面及轮廓指示函数的求取过程中引入距离函数和内积函数。结果 将本文方法与等值面抽取法的重建结果进行比较,本文方法在重建的过程中能够自动修补空洞,重建出的模型表面更加光滑,能更加真实地反映原物体的特征。结论 可以将本文方法推广应用到稀疏及残缺SAR数据的建模中。     

A computational model of bounded developable surfaces with application to image‐based three‐dimensional reconstruction

Developable surfaces have been extensively studied in computer graphics because they are involved in a large body of applications. This type of surfaces has also been used in computer vision and document processing in the context of three‐dimensional (3D) reconstruction for book digitization and augmented reality. Indeed, the shape of a smoothly deformed piece of paper can be very well modeled by a developable surface. Most of the existing developable surface parameterizations do not handle boundaries or are driven by overly large parameter sets. These two characteristics become issues in the context of developable surface reconstruction from real observations. Our main contribution is a generative model of bounded developable surfaces that solves these two issues. Our model is governed by intuitive parameters whose number depends on the actual deformation and including the “flat shape boundary”. A vast majority of the existing image‐based paper 3D reconstruction methods either require a tightly controlled environment or restricts the set of possible deformations. We propose an algorithm for reconstructing our model's parameters from a general smooth 3D surface interpolating a sparse cloud of 3D points. The latter is assumed to be reconstructed from images of a static piece of paper or any other developable surface. Our 3D reconstruction method is well adapted to the use of keypoint matches over multiple images. In this context, the initial 3D point cloud is reconstructed by structure‐from‐motion for which mature and reliable algorithms now exist and the thin‐plate spline is used as a general smooth surface model. After initialization, our model's parameters are refined with model‐based bundle adjustment. We experimentally validated our model and 3D reconstruction algorithm for shape capture and augmented reality on seven real datasets. The first six datasets consist of multiple images or videos and a sparse set of 3D points obtained by structure‐from‐motion. The last dataset is a dense 3D point cloud acquired by structured light. Our implementation has been made publicly available on the authors' web home pages. Copyright © 2012 John Wiley & Sons, Ltd.     

Manifold surface reconstruction of an environment from sparse Structure-from-Motion data

The majority of methods for the automatic surface reconstruction of an environment from an image sequence have two steps: Structure-from-Motion and dense stereo. From the computational standpoint, it would be interesting to avoid dense stereo and to generate a surface directly from the sparse cloud of 3D points and their visibility information provided by Structure-from-Motion. The previous attempts to solve this problem are currently very limited: the surface is non-manifold or has zero genus, the experiments are done on small scenes or objects using a few dozens of images. Our solution does not have these limitations. Furthermore, we experiment with hand-held or helmet-held catadioptric cameras moving in a city and generate 3D models such that the camera trajectory can be longer than one kilometer.     

Modeling Complex Unfoliaged Trees from a Sparse Set of Images

We present a novel image‐based technique for modeling complex unfoliaged trees. Existing tree modeling tools either require capturing a large number of views for dense 3D reconstruction or rely on user inputs and botanic rules to synthesize natural‐looking tree geometry. In this paper, we focus on faithfully recovering real instead of realistically‐looking tree geometry from a sparse set of images. Our solution directly integrates 2D/3D tree topology as shape priors into the modeling process. For each input view, we first estimate a 2D skeleton graph from its matte image and then find a 2D skeleton tree from the graph by imposing tree topology. We develop a simple but effective technique for computing the optimal 3D skeleton tree most consistent with the 2D skeletons. For each edge in the 3D skeleton tree, we further apply volumetric reconstruction to recover its corresponding curved branch. Finally, we use piecewise cylinders to approximate each branch from the volumetric results. We demonstrate our framework on a variety of trees to illustrate the robustness and usefulness of our technique.     

Tomographic Reconstruction of 3D Objects Using Marked Point Process Framework

For reconstructing sparse volumes of 3D objects from projection images taken from different viewing directions, several volumetric reconstruction techniques are available. Most popular volume reconstruction methods are algebraic algorithms (e.g. the multiplicative algebraic reconstruction technique, MART). These methods which belong to voxel-oriented class allow volume to be reconstructed by computing each voxel intensity. A new class of tomographic reconstruction methods, called “object-oriented” approach, has recently emerged and was used in the Tomographic Particle Image Velocimetry technique (Tomo-PIV). In this paper, we propose an object-oriented approach, called Iterative Object Detection—Object Volume Reconstruction based on Marked Point Process (IOD-OVRMPP), to reconstruct the volume of 3D objects from projection images of 2D objects. Our approach allows the problem to be solved in a parsimonious way by minimizing an energy function based on a least squares criterion. Each object belonging to 2D or 3D space is identified by its continuous position and a set of features (marks). In order to optimize the population of objects, we use a simulated annealing algorithm which provides a “Maximum A Posteriori” estimation. To test our approach, we apply it to the field of Tomo-PIV where the volume reconstruction process is one of the most important steps in the analysis of volumetric flow. Finally, using synthetic data, we show that the proposed approach is able to reconstruct densely seeded flows.     

Realistic surface geometry reconstruction using a hand-held RGB-D camera

In this paper, we have proposed a novel approach for the reconstruction of real object/scene with realistic surface geometry using a hand-held, low-cost, RGB-D camera. To achieve accurate reconstruction, the most important issues to consider are the quality of the geometry information provided and the global alignment method between frames. In our approach, new surface geometry refinement is used to recover finer scale surface geometry from depth data by utilizing high-quality RGB images. In addition, a weighted multi-scale iterative closest point method is exploited to align each scan to the global model accurately. We show the effectiveness of the proposed surface geometry refinement method by comparing it with other depth refinement methods. We also show both the qualitative and quantitative results of reconstructed models by comparing it with other reconstruction methods.