Putting the User in the Loop for Image-Based Modeling

We refer to the task of recovering the 3D structure of an object or a scene using 2D images as image-based modeling. In this paper, we formulate the task of recovering the 3D structure as a discrete optimization problem solved via energy minimization. In this standard framework of a Markov random field (MRF) defined over the image we present algorithms that allow the user to intuitively interact with the algorithm. We introduce an algorithm where the user guides the process of image-based modeling to find and model the object of interest by manually interacting with the nodes of the graph. We develop end user applications using this algorithm that allow object of interest 3D modeling on a mobile device and 3D printing of the object of interest. We also propose an alternate active learning algorithm that guides the user input. An initial attempt is made at reconstructing the scene without supervision. Given the reconstruction, an active learning algorithm uses intuitive cues to quantify the uncertainty of the algorithm and suggest regions, querying the user to provide support for the uncertain regions via simple scribbles. These constraints are used to update the unary and the pairwise energies that, when solved, lead to better reconstructions. We show through machine experiments and a user study that the proposed approach intelligently queries the users for constraints, and users achieve better reconstructions of the scene faster, especially for scenes with textureless surfaces lacking strong textural or structural cues that algorithms typically require.     

A Sketch-Based User Interface for Reconstructing Architectural Drawings

We present a framework for interactive sketching that allows users to create three‐dimensional (3D) architectural models quickly and easily from a source drawing. The sketching process has four steps. (1) The user calibrates a viewing camera by specifying the origin and vanishing points of the drawing. (2) The user outlines surface polygons in the drawing. (3) A 3D reconstruction algorithm uses perceptual constraints to determine the closest visual fit for the polygon. (4) The user can then adjust aesthetic controls to produce several stylistic effects in the scene: a smooth transition between day and night rendering, a horizon knockout effect and entourage figures. The major advantage of our approach lies in the combination of perception‐based techniques, which allow us to minimize unnecessary interactions, and a hinging‐angle scheme, which shows significant improvement in numerical stability over previous optimization‐based 3D reconstruction algorithms. We also demonstrate how our reconstruction algorithm can be extended to work with perspective images, a feature unavailable in previous approaches.     

An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery

Modeling the energy performance of existing buildings enables quick identification and reporting of potential areas for building retrofit. However, current modeling practices of using energy simulation tools do not model the energy performance of buildings at their element level. As a result, potential retrofit candidates caused by construction defects and degradations are not represented. Furthermore, due to manual modeling and calibration processes, their application is often time-consuming. Current application of 2D thermography for building diagnostics is also facing several challenges due to a large number of unordered and non-geo-tagged images. To address these limitations, this paper presents a new computer vision-based method for automated 3D energy performance modeling of existing buildings using thermal and digital imagery captured by a single thermal camera. First, using a new image-based 3D reconstruction pipeline which consists of Graphic Processing Unit (GPU)-based Structure-from-Motion (SfM) and Multi-View Stereo (MVS) algorithms, the geometrical conditions of an existing building is reconstructed in 3D. Next, a 3D thermal point cloud model of the building is generated by using a new 3D thermal modeling algorithm. This algorithm involves a one-time thermal camera calibration, deriving the relative transformation by forming the Epipolar geometry between thermal and digital images, and the MVS algorithm for dense reconstruction. By automatically superimposing the 3D building and thermal point cloud models, 3D spatio-thermal models are formed, which enable the users to visualize, query, and analyze temperatures at the level of 3D points. The underlying algorithms for generating and visualizing the 3D spatio-thermal models and the 3D-registered digital and thermal images are presented in detail. The proposed method is validated for several interior and exterior locations of a typical residential building and an instructional facility. The experimental results show that inexpensive digital and thermal imagery can be converted into ubiquitous reporters of the actual energy performance of existing buildings. The proposed method expedites the modeling process and has the potential to be used as a rapid and robust building diagnostic tool.     

An Intuitive Interface for Interactive High Quality Image‐Based Modeling

We present the design of an interactive image‐based modeling tool that enables a user to quickly generate detailed 3D models with texture from a set of calibrated input images. Our main contribution is an intuitive user interface that is entirely based on simple 2D painting operations and does not require any technical expertise by the user or difficult pre‐processing of the input images. One central component of our tool is a GPU‐based multi‐view stereo reconstruction scheme, which is implemented by an incremental algorithm, that runs in the background during user interaction so that the user does not notice any significant response delay.     

A Linear Image Reconstruction Framework Based on Sobolev Type Inner Products

Exploration of information content of features that are present in images has led to the development of several reconstruction algorithms. These algorithms aim for a reconstruction from the features that is visually close to the image from which the features are extracted. Degrees of freedom that are not fixed by the constraints are disambiguated with the help of a so-called prior (i.e. a user defined model). We propose a linear reconstruction framework that generalizes a previously proposed scheme. The algorithm greatly reduces the complexity of the reconstruction process compared to non-linear methods. As an example we propose a specific prior and apply it to the reconstruction from singular points. The reconstruction is visually more attractive and has a smaller 핃₂-error than the reconstructions obtained by previously proposed linear methods. Bart Jansen, Frans Kanters and Remco Duits are joint main authors of this article.     

A robust hybrid image-based modeling system

This paper presents a new robust image-based modeling system for creating high-quality 3D models of complex objects from a sequence of unconstrained photographs. The images can be acquired by a video camera or hand-held digital camera without the need of camera calibration. In contrast to previous methods, we integrate correspondence-based and silhouette-based approaches, which significantly enhances the reconstruction of objects with few visual features (e.g., uni-colored objects) and improves surface smoothness. Our solution uses a mesh segmentation and charting approach in order to create a low-distortion mesh parameterization suitable for objects of arbitrary genus. A high-quality texture is produced by first parameterizing the reconstructed objects using a segmentation and charting approach, projecting suitable sections of input images onto the model, and combining them using a graph-cut technique. Holes in the texture due to surface patches without projecting input images are filled using a novel exemplar-based inpainting method which exploits appearance space attributes to improve patch search, and blends patches using Poisson-guided interpolation. We analyzed the effect of different algorithm parameters, and compared our system with a laser scanning-based reconstruction and existing commercial systems. Our results indicate that our system is robust, superior to other image-based modeling techniques, and can achieve a reconstruction quality visually not discernible from that of a laser scanner.     

3D Video Billboard Clouds

3D video billboard clouds reconstruct and represent a dynamic three-dimensional scene using displacement-mapped billboards. They consist of geometric proxy planes augmented with detailed displacement maps and combine the generality of geometry-based 3D video with the regularization properties of image-based 3D video. 3D video billboards are an image-based representation placed in the disparity space of the acquisition cameras and thus provide a regular sampling of the scene with a uniform error model. We propose a general geometry filtering framework which generates time-coherent models and removes reconstruction and quantization noise as well as calibration errors. This replaces the complex and time-consuming sub-pixel matching process in stereo reconstruction with a bilateral filter. Rendering is performed using a GPU-accelerated algorithm which generates consistent view-dependent geometry and textures for each individual frame. In addition, we present a semi-automatic approach for modeling dynamic three-dimensional scenes with a set of multiple 3D video billboards clouds.     

Fast joint estimation of silhouettes and dense 3D geometry from multiple images

We propose a probabilistic formulation of joint silhouette extraction and 3D reconstruction given a series of calibrated 2D images. Instead of segmenting each image separately in order to construct a 3D surface consistent with the estimated silhouettes, we compute the most probable 3D shape that gives rise to the observed color information. The probabilistic framework, based on Bayesian inference, enables robust 3D reconstruction by optimally taking into account the contribution of all views. We solve the arising maximum a posteriori shape inference in a globally optimal manner by convex relaxation techniques in a spatially continuous representation. For an interactively provided user input in the form of scribbles specifying foreground and background regions, we build corresponding color distributions as multivariate Gaussians and find a volume occupancy that best fits to this data in a variational sense. Compared to classical methods for silhouette-based multiview reconstruction, the proposed approach does not depend on initialization and enjoys significant resilience to violations of the model assumptions due to background clutter, specular reflections, and camera sensor perturbations. In experiments on several real-world data sets, we show that exploiting a silhouette coherency criterion in a multiview setting allows for dramatic improvements of silhouette quality over independent 2D segmentations without any significant increase of computational efforts. This results in more accurate visual hull estimation, needed by a multitude of image-based modeling approaches. We made use of recent advances in parallel computing with a GPU implementation of the proposed method generating reconstructions on volume grids of more than 20 million voxels in up to 4.41 seconds.     

海量数据的曲面分层重建算法

从二维图像序列进行表面重建的问题由来已久.传统的重建方法通常是先重建或先等值面抽取,再简化数据量.随着处理数据量的增长,传统算法的中间过程会因为存储空间的限制不能进行下去.如何利用有限的存储空间对大数据量进行处理,从而完成曲面的重建曾是要研究的问题.针对大数据量的已分割的医学切片图像,利用逐层重建、即时简化的基本思想,给出一个易于操作实现、数据量可控制的算法.这样可以在硬件条件不太高的计算机(如内存不太大的个人微机)上实现大数据量的医学图像表面重建.     

Multi-object reconstruction from dynamic scenes: An object-centered approach

In this paper, we present a new framework for three-dimensional (3D) reconstruction of multiple rigid objects from dynamic scenes. Conventional 3D reconstruction from multiple views is applicable to static scenes, in which the configuration of objects is fixed while the images are taken. In our framework, we aim to reconstruct the 3D models of multiple objects in a more general setting where the configuration of the objects varies among views. We solve this problem by object-centered decomposition of the dynamic scenes using unsupervised co-recognition approach. Unlike conventional motion segmentation algorithms that require small motion assumption between consecutive views, co-recognition method provides reliable accurate correspondences of a same object among unordered and wide-baseline views. In order to segment each object region, we benefit from the 3D sparse points obtained from the structure-from-motion. These points are reliable and serve as automatic seed points for a seeded-segmentation algorithm. Experiments on various real challenging image sequences demonstrate the effectiveness of our approach, especially in the presence of abrupt independent motions of objects.     

Photo Hull regularized stereo

A regularization-based approach to 3D reconstruction from multiple images is proposed. As one of the most widely used multiple-view 3D reconstruction algorithms, Space Carving can produce a Photo Hull of a scene, which is at best a coarse volumetric model. The two-view stereo algorithm, on the other hand, can generate a more accurate reconstruction of the surfaces, provided that a given surface is visible to both views. The proposed method is essentially a data fusion approach to 3D reconstruction, combining the above two algorithms by means of regularization. The process is divided into two steps: (1) computing the Photo Hull from multiple calibrated images and (2) selecting two of the images as input and solving the two-view stereo problem by global optimization, using the Photo Hull as the regularizer. Our dynamic programming implementation of this regularization-based stereo approach potentially provides an efficient and robust way of reconstructing 3D surfaces. The results of an implementation of this theory is presented on real data sets and compared with peer algorithms.     

Superresolution Reflectance Fields: Synthesizing images for intermediate light directions

Captured reflectance fields tend to provide a relatively coarse sampling of the incident light directions. As a result, sharp illumination features, such as highlights or shadow boundaries, are poorly reconstructed during relighting; highlights are disconnected, and shadows show banding artefacts. In this paper, we propose a novel interpolation technique for 4D reflectance fields that reconstructs plausible images even for non-observed light directions. Given a sparsely sampled reflectance field, we can effectively synthesize images as they would have been obtained from denser sampling. The processing pipeline consists of three steps: (1) segmentation of regions where intermediate lighting cannot be obtained by blending, (2) appropriate flow algorithms for highlights and shadows, plus (3) a final reconstruction technique that uses image-based priors to faithfully correct errors that might be introduced by the segmentation or flow step. The algorithm reliably reproduces scenes that contain specular highlights, interreflections, shadows or caustics.     

基于弱监督学习的三维人脸形状与纹理重建

三维人脸相较于二维人脸包含了更多特征信息, 可应用于如人脸识别、影视娱乐、医疗美容等更多实际应用场景, 因此三维人脸重建技术一直是计算机视觉领域的研究热点. 由于真实三维人脸数据较难获取, 很多基于深度学习的重建算法首先利用传统重建方法为大量二维人脸图像构建三维标签, 作为训练数据, 这些数据可能并不精准, 从而导致算法的重建精度受到影响. 为此, 本文提出一种基于multi-level损失函数的弱监督学习模型, 结合传统三维人脸形变模型3DMM与深度学习方法, 直接从大量无三维标签的二维人脸图像中学习三维人脸特征信息, 从而实现基于单张二维人脸图像的三维人脸重建算法. 此外, 为解决二维人脸图像中常存在遮挡或大姿态情况而影响人脸纹理重建的问题, 本文使用基于CelebAMask-HQ数据集的人脸解析分割算法对图像进行预处理去除遮挡区域. 实验结果表明, 基于本文方法的三维人脸重建质量与重建精度均实现了一定的提升.     

Camera calibration based on arbitrary parallelograms

Existing algorithms for camera calibration and metric reconstruction are not appropriate for image sets containing geometrically transformed images for which we cannot apply the camera constraints such as square or zero-skewed pixels. In this paper, we propose a framework to use scene constraints in the form of camera constraints. Our approach is based on image warping using images of parallelograms. We show that the warped image using parallelograms constrains the camera both intrinsically and extrinsically. Image warping converts the calibration problems of transformed images into the calibration problem with highly constrained cameras. In addition, it is possible to determine affine projection matrices from the images without explicit projective reconstruction. We introduce camera motion constraints of the warped image and a new parameterization of an infinite homography using the warping matrix. Combining the calibration and the affine reconstruction results in the fully metric reconstruction of scenes with geometrically transformed images. The feasibility of the proposed algorithm is tested with synthetic and real data. Finally, examples of metric reconstructions are shown from the geometrically transformed images obtained from the Internet.     

3D reconstruction from 2D images with hierarchical continuous simplices

This paper presents an effective framework for the reconstruction of volumetric data from a sequence of 2D images. The 2D images are first aligned to generate an initial 3D volume, followed by the creation of a tetrahedral domain using the Carver algorithm. The resulting tetrahedralization preserves both the geometry and topology of the original dataset. Then a solid model is reconstructed using simplex splines with fitting and faring procedures. The reconstructed heterogenous volumetric model can be quantitatively analyzed and easily visualized. Our experiments demonstrated that our approach can achieve high accuracy in the data reconstruction. The novel techniques and algorithms proposed in this paper can be applied to reconstruct a heterogeneous solid model with complex geometry and topology from other visual data.     

Image-Based Modeling by Joint Segmentation

The paper first traces the image-based modeling back to feature tracking and factorization that have been developed in the group led by Kanade since the eighties. Both feature tracking and factorization have inspired and motivated many important algorithms in structure from motion, 3D reconstruction and modeling. We then revisit the recent quasi-dense approach to structure from motion. The key advantage of the quasi-dense approach is that it not only delivers the structure from motion in a robust manner for practical modeling purposes, but also it provides a cloud of sufficiently dense 3D points that allows the objects to be explicitly modeled. To structure the available 3D points and registered 2D image information, we argue that a joint segmentation of both 3D and 2D is the fundamental stage for the subsequent modeling. We finally propose a probabilistic framework for the joint segmentation. The optimal solution to such a joint segmentation is still generally intractable, but approximate solutions are developed in this paper. These methods are implemented and validated on real data set.     

Image-based interactive exploration of real-world environments

Interactive scene walkthroughs have long been an important computer graphics application area. More recently, researchers have developed techniques for constructing photorealistic 3D architectural models from real-world images. We present an image-based rendering system that brings us a step closer to a compelling sense of being there. Whereas many previous systems have used still photography and 3D scene modeling, we avoid explicit 3D reconstruction because it tends to be brittle. Our system is not the first to propose interactive video-based tours. We believe, however, that our system is the first to deliver fully interactive, photorealistic image-based tours on a personal computer at or above broadcast video resolutions and frame rates. Moreover, to our knowledge, no other tour provides the same rich set of interactions or visually complex environments.     

Images for accelerating architectural walkthroughs

We describe our image-based solutions to produce a system for the interactive walkthrough of large architectural models. Our overall approach uses a specialization of impostors introduced by Maciel and Shirley (1995). With this method, portions of a general 3D model can be replaced with representations that take less time to render, namely 2D textures. Solving the general problem of deciding where to place textures to improve rendering performance proves difficult. The cells and portals framework lets us formulate a set of concrete and efficient algorithms for replacing geometry with images     

Preliminary coarse image registration by using straight lines found on them for constructing super resolution mosaics and 3D scene recovery

An algorithm of coarse image registration of a 3D scene taken from different camera perspectives is proposed. The algorithm uses information on geometrical parameters of straight lines found on the images and on distribution of color and/or brightness around these lines. Colors are taken into account by using the fuzzy logic technique. The result of the algorithm operation is a planar projective transformation (planar homography) matching approximately the images. In order to use the technique in algorithms of 3D scene reconstruction, an estimate of size of the window used for searching correspondent points after the coarse image registration is obtained.