3D Reconstruction of Indoor Scenes via Image Registration

With the development of computer vision technologies, 3D reconstruction has become a hotspot. At present, 3D reconstruction relies heavily on expensive equipment and has poor real-time performance. In this paper, we aim at solving the problem of 3D reconstruction of an indoor scene with large vertical span. In this paper, we propose a novel approach for 3D reconstruction of indoor scenes with only a Kinect. Firstly, this method uses a Kinect sensor to get color images and depth images of an indoor scene. Secondly, the combination of scale-invariant feature transform and random sample consensus algorithm is used to determine the transformation matrix of adjacent frames, which can be seen as the initial value of iterative closest point (ICP). Thirdly, we establish the relative coordinate relation between pair-wise frames which are the initial point cloud data by using ICP. Finally, we achieve the 3D visual reconstruction model of indoor scene by the top-down image registration of point cloud data. This approach not only mitigates the sensor perspective restriction and achieves the indoor scene reconstruction of large vertical span, but also develops the fast algorithm of indoor scene reconstruction with large amount of cloud data. The experimental results show that the proposed algorithm has better accuracy, better reconstruction effect, and less running time for point cloud registration. In addition, the proposed method has great potential applied to 3D simultaneous location and mapping.     

遥感图像辅助的折反射全向图三维重构

提出了一种针对室外场景的单幅折反射全向图三维重构方法,能够自动重构出全向图中360°视野内景物的三维模型,并实现自由漫游。基于全向图与遥感图匹配把全向图分为水平地面、垂直建筑物立面和垂直背景景物面三类区域,得到全向图场景的基本结构;在此基础上利用折反射光路投射模型计算出全向图中每个像素点的三维几何位置,从而实现了折反射全向图的重构。实验证明该方法具有采集简单、视野大、处理过程全自动化、能够重构非平面场景等特点。     

Indoor Scene Reconstruction from Sets of Noisy Range Images

This paper describes a system for building 3D models of indoor scenes from sets of noisy laser range images. It addresses several important aspects of this problem, namely, preprocessing, which includes image segmentation and planar model fitting; view registration, which is the method of determining the rigid transformation that describes the relative pose of the camera platform between views; and reconstruction, which is the subsequent integration or fusion of separate range images into a single 3D model. Our proposed strategy is to use a statistical sensor model. We thus account for noise properties of the data at each stage in the reconstruction process, which produces reliable results even in the presence of significant measurement noise. We give an empirical analysis of a plane-based registration method and present results using real range data that demonstrate the performance of the entire reconstruction system.     

大规模室外图像3维重建技术研究进展

基于图像的3维重建旨在从一组2维多视角图像中精确地恢复真实场景的几何形状,是计算机视觉和摄影测量中基础且活跃的研究课题,具有重要的理论研究意义和应用价值,在智慧城市、虚拟旅游、数字遗产保护、数字地图和导航等领域有着广泛应用。随着图像采集系统（智能手机、消费级数码相机和民用无人机等）的普及和互联网的高速发展,通过搜索引擎可以获取大量关于某个室外场景的互联网图像。利用这些图像进行高效鲁棒准确的3维重建,为用户提供真实感知和沉浸式体验已经成为研究热点,引发了学术界和产业界的广泛关注,涌现了多种方法。深度学习的出现为大规模室外图像的3维重建提供了新的契机。首先阐述大规模室外图像3维重建的基本串行过程,包括图像检索、图像特征点匹配、运动恢复结构和多视图立体。然后从传统方法和基于深度学习的方法两个角度,分别系统全面地回顾大规模室外图像3维重建技术在各重建子过程中的发展和应用,总结各子过程中适用于大规模室外场景的数据集和评价指标。最后介绍现有主流的开源和商业3维重建系统以及国内相关产业的发展现状。     

基于深度学习的立体影像视差估计方法综述

三维重建技术常用于自动驾驶、机器人、无人机和增强现实等领域。视差估计是三维重建的关键步骤,随着数据集的增加、硬件和网络模型的发展,深度学习视差估计模型被广泛使用并取得良好效果。然而,这些方法常用室外场景的物体,很少使用在室内场景的数据集中。回顾了双目视差估计的深度学习方法,选用5种深度学习网络：PSMNet(pyramid stereo matching network)、GA-Net(guided aggregation network)、LEAStereo(hierarchical neural architecture search for deep stereo matching)、DeepPruner(learning efficient stereo matching via differentiable patchmatch)、BGNet(bilateral grid learning for stereo matching networks),将其运用在一套真实世界的街景数据集（KITTI2015）和两套室内场景数据集（Middlebury2014、Instereo2K...     

Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications

Current state-of-the-art image-based scene reconstruction techniques are capable of generating high-fidelity 3D models when used under controlled capture conditions. However, they are often inadequate when used in more challenging environments such as sports scenes with moving cameras. Algorithms must be able to cope with relatively large calibration and segmentation errors as well as input images separated by a wide-baseline and possibly captured at different resolutions. In this paper, we propose a technique which, under these challenging conditions, is able to efficiently compute a high-quality scene representation via graph-cut optimisation of an energy function combining multiple image cues with strong priors. Robustness is achieved by jointly optimising scene segmentation and multiple view reconstruction in a view-dependent manner with respect to each input camera. Joint optimisation prevents propagation of errors from segmentation to reconstruction as is often the case with sequential approaches. View-dependent processing increases tolerance to errors in through-the-lens calibration compared to global approaches. We evaluate our technique in the case of challenging outdoor sports scenes captured with manually operated broadcast cameras as well as several indoor scenes with natural background. A comprehensive experimental evaluation including qualitative and quantitative results demonstrates the accuracy of the technique for high quality segmentation and reconstruction and its suitability for free-viewpoint video under these difficult conditions.     

灰关联分析在视频运动目标检测中的应用

由于运动图像和背景具有极大相似性,通过背景图像和运动图像之间关联程度的大小能够检测出运动目标,故提出一种基于灰关联分析的运动目标检测方法.在室内和室外不同光照场景下,通过固定摄像机捕获的视频图像序列中的运动车体和人体进行检测;选取适当的比较图像序列,对该序列和含有运动目标的视频图像作灰关联分析,以清楚、完整地提取出运动目标.该方法对背景的要求很低,对噪声的抑制能力强,可以在一定程度上抑制阴影的影响.     

结合视觉惯性模组的室内三维布局鲁棒重建方法

针对使用传统单目相机的全自动三维重建方法结果精确度差和整体结构理解缺失等问题,提出一种结合视觉惯性里程计和由运动到结构的全自动室内三维布局重建系统.首先利用视觉里程计获得关键帧图像序列和对应空间位置姿态,并利用运动恢复结构算法计算精确相机位姿;然后利用多图视立体几何算法生成高质量稠密点云;最后基于曼哈顿世界假设,针对典型的现代建筑室内场景,设计一种基于规则的自底向上的布局重建方法,得到最终房间外轮廓布局.使用浙江大学CAD&CG实验室场景现场扫描数据集和人工合成的稠密点云数据集作为实验数据,在Ubuntu 16.04和PCL 1.9环境下进行实验.结果表明,文中方法对三维点云噪声容忍度高,能够有效地重建出室内场景的三维外轮廓布局.     

基于支持向量机多分类器的室内外场景感知算法

针对普适室内外场景持续感知面临的低功耗、复杂动态环境、异构使用模式带来的挑战,提出了一种轻量级的基于支持向量机多分类器的高精度、低功耗室内外场景检测算法.该算法使用智能手机集成的各种传感器(可见光传感器、磁传感器、加速度传感器、陀螺仪传感器和气压传感器),在挖掘分析各种传感器在室内外场景的不同特征,以及人们在室内外场景的行为差异基础上,根据时间和气象条件设计多个支持向量机分类器,对复杂室内外场景进行识别.实验结果表明,基于支持向量机多分类器的室内外场景检测算法具有较好的普适性,可获得95%以上的室内外判定准确率,平均功耗小于5 mW.     

The IFC-based path planning for 3D indoor spaces

Path planning is a fundamental problem, especially for various AEC applications, such as architectural design, indoor and outdoor navigation, and emergency evacuation. However, the conventional approaches mainly operate path planning on 2D drawings or building layouts by simply considering geometric information, while losing abundant semantic information of building components. To address this issue, this paper introduces a new method to cope with path planning for 3D indoor space through an IFC (Industry Foundation Classes) file as input. As a major data exchange standard for Building Information Modeling (BIM), the IFC standard is capable of restoring both geometric information and rich semantic information of building components to support lifecycle data sharing. The method consists of three main steps: (1) extracting both geometric and semantic information of building components defined within the IFC file, (2) discretizing and mapping the extracted information into a planar grid, (3) and finally finding the shortest path based on the mapping for path planning using Fast Marching Method. The paper aims to process different kinds of building components and their corresponding properties to obtain rich semantic information that can enhance applications of path planning. In addition, the IFC-based distributed data sharing and management is implemented for path planning. The paper also presents some experiments to demonstrate the accuracy, efficiency and adaptability of the method. Video demonstration is available from http://cgcad.thss.tsinghua.edu.cn/liuyushen/ifcpath/.     

一种快速的人工场景图像消失点检测方法

人工场景中包含了大量的空间平行线以及垂直边,这些空间平行线映射到图像中相交产生的交点即消失点。消失点检测对摄像机标定、三维场景重建等都有着重要的意义。传统的消失点检测算法往往基于二维霍夫参数空间,复杂度高、效率低。因此,提出一种新的方法,先检测图像中较长的边界线,并将检测到的线段进行筛选、分组;然后利用消失点与焦距之间的制约关系,确定三向消失点的位置以及焦距的大小。该方法将传统的二维霍夫参数空间转换为二级一维霍夫参数空间。实验表明,这种方法运算复杂度低、运行时间短。在室外场景图像中,鲁棒性好,且保持较高的准确率。     

Shape-indifferent stereo disparity based on disparity gradient estimation

A new algorithm capable of estimating disparity gradients to produce accurate dense disparities is proposed. Such a disparity gradient plays a critical role in acquiring accurate disparities for scenes with many different object shapes. The target is a road traffic scene because it contains various objects, including the road surface, vehicles, pedestrians, sidewalks, and walls. In this paper, we adopt several methods, such as initial matching cost computation, scanline optimization, left/right consistency check, and cost aggregation. However, disparity accuracy is slightly improved by the simple organization of such methods. Disparity quality decisively relies on the application of disparity gradients. Accordingly, in the proposed algorithm, cost aggregation is performed along the direction of the estimated disparity gradient in a disparity space image. This approach improves disparity quality significantly. However, this cost aggregation is time consuming. To reduce the time required, we designed a new 2D integral cost technique. The robustness of the proposed algorithm is demonstrated through the disparity maps obtained from standard images on the Web, indoor images, and outdoor images of various road traffic scenes.     

Interactive Design and Preview of Colored Snapshots of Indoor Scenes

This paper presents an interactive system for quickly designing and previewing colored snapshots of indoor scenes. Different from high-quality 3D indoor scene rendering, which often takes several minutes to render a moderately complicated scene under a specific color theme with high-performance computing devices, our system aims at improving the effectiveness of color theme design of indoor scenes and employs an image colorization approach to efficiently obtain high-resolution snapshots with editable colors. Given several pre-rendered, multi-layer, gray images of the same indoor scene snapshot, our system is designed to colorize and merge them into a single colored snapshot. Our system also assists users in assigning colors to certain objects/components and infers more harmonious colors for the unassigned objects based on pre-collected priors to guide the colorization. The quickly generated snapshots of indoor scenes provide previews of interior design schemes with different color themes, making it easy to determine the personalized design of indoor scenes. To demonstrate the usability and effectiveness of this system, we present a series of experimental results on indoor scenes of different types, and compare our method with a state-of-the-art method for indoor scene material and color suggestion and offline/online rendering software packages.     

Indoor space art design based on immersive VR and embedded system

Indoor space systems are mainly used in outdoor environments, but indoor navigation systems are still under development. This article introduces indoor spatial analysis based on virtual reality to people who help in indoor environments. Work is done with the mechanisms and methods used for storage, dynamic reconstruction, and navigation in 3D virtual space using databases. The characteristics of the room are shown in the database table. The proposed Virtual Reality Environment Modeling (VREM) method is used to create a 3D image to analyze the room space. The client browser requests the virtual world through the server that provides the URL. The selected virtual world and server virtual format is returned to the client. The new 3D simulation embedded system is used to act as an agent for the behavior of simulation embedded system and other objects on the Internet and their associated syntactic structures. This method is first applied to the forward transform 3D to enhance the virtual and reconstructed images of the discrete cosine transform. Effectively view large virtual worlds and easily modify them. The size, shape, or other aspects of the virtual world's elements can be changed by changing the data in the database. Finally, virtual block matching algorithms are applied to analyze virtual indoor components or objects to provide better results than existing methods.     

Dual-view holographic AR display based on Bragg mismatched reconstruction of the holographic optical element

A method for dual-view holographic display based on Bragg mismatched reconstruction of holographic optical element (HOE) is proposed. Under the Bragg mismatched condition, the reconstructed images are guided into two separated viewing zones to realize dual-view holographic display. Meanwhile, the viewing angle of each perspective is increased to 11.2°, which is almost 2.5 times as large as the traditional holographic display system. The design process of HOE is simple only by interference of plane reference wave and converging spherical signal wave, which has high practicability. Furthermore, the HOE can mix the virtual 3D image with real-world scenes, which could implement augmented reality (AR) display. Experiments validate that the proposed system can achieve dual-view holographic AR three-dimensional (3D) display with accommodation effect.     

基于迁移学习的移动机器人单帧图像坡度检测算法

针对未知环境下移动机器人平稳上坡控制对坡度感知精度的要求,本文提出了一种基于迁移学习的移动机器人单帧图像坡度检测算法.利用室内图像标准数据集训练深度卷积神经场-全连接超像素池化网络(deep convolutional neural field-fully connected superpixel pooling ne...     

基于并联卷积与残差网络的图像超分辨率重建

现有的图像超分辨率重建算法可以改善图像整体视觉效果或者提升重建图像的客观评价值,然而对图像感知效果和客观评价值的均衡提升效果不佳,且重建图像缺乏高频信息,导致纹理模糊。针对上述问题,提出了一种基于并联卷积与残差网络的图像超分辨率重建算法。首先,以并联结构为整体框架,在并联结构上采用不同卷积组合来丰富特征信息,并加入跳跃连接来进一步丰富特征信息并融合输出,从而提取更多的高频信息。其次,引入自适应残差网络以补充信息并优化网络性能。最后,采用感知损失来提升恢复后图像的整体质量。实验结果表明,相较于超分辨率卷积神经网络（SRCNN）、深度超分辨率重建网络（VDSR）和超分辨率生成对抗网络（SRGAN）等算法,所提算法在重建图像上有更好的表现,其放大效果图的细节纹理更清晰。在客观评价上,所提算法在4倍重建时的峰值信噪比（PSNR）和结构相似性（SSIM）相较于SRGAN分别平均提升了0.25 dB和0.019。     

融合感知损失的单幅雾霾图像深度估计

针对雾霾情况下室内外图像深度难以估计的问题,提出了融合感知损失函数的单幅雾霾图像深度估计方法.首先采用双尺度网络模型对雾霾图像进行粗提取,再结合底层特征进行局部细化;然后在上采样阶段使用多卷积核上采样方法,得到雾霾图像的预测深度图;最后将像素级损失函数与感知损失函数结合构造新的复合损失函数,对网络进行训练.在室内NYU...     

Calibrated,Registered Images of an Extended Urban Area

We describe a dataset of several thousand calibrated, time-stamped, geo-referenced, high dynamic range color images, acquired under uncontrolled, variable illumination conditions in an outdoor region spanning several hundred meters. The image data is grouped into several regions which have little mutual inter-visibility. For each group, the calibration data is globally consistent on average to roughly five centimeters and 0 1°, or about four pixels of epipolar registration. All image, feature and calibration data is available for interactive inspection and downloading at http://city.lcs.mit.edu/data.Calibrated imagery is of fundamental interest in a variety of applications. We have made this data available in the belief that researchers in computer graphics, computer vision, photogrammetry and digital cartography will find it of value as a test set for their own image registration algorithms, as a calibrated image set for applications such as image-based rendering, metric 3D reconstruction, and appearance recovery, and as input for existing GIS applications.     

The separation of reflected and transparent layers from real-world image sequence

This paper presents an optimal method for the separation of reflected and transparent layers from real-world scene images. Whereas past research has been applied to indoor environments and static cameras, our technique can be used for outdoor scenes and motion cameras. The method is based on spatio-temporal analysis, especially using epipolar plane images (EPI). The edge and color information of EPI has been used to segment the areas on EPIs efficiently and separate the reflected and transparent layers. This method can be used for refining building textures by removing reflections from captured images for the purpose of city modeling.