Data Processing Algorithms for Generating Textured 3D Building Facade Meshes from Laser Scans and Camera Images

In this paper, we develop a set of data processing algorithms for generating textured facade meshes of cities from a series of vertical 2D surface scans and camera images, obtained by a laser scanner and digital camera while driving on public roads under normal traffic conditions. These processing steps are needed to cope with imperfections and non-idealities inherent in laser scanning systems such as occlusions and reflections from glass surfaces. The data is divided into easy-to-handle quasi-linear segments corresponding to approximately straight driving direction and sequential topological order of vertical laser scans; each segment is then transformed into a depth image. Dominant building structures are detected in the depth images, and points are classified into foreground and background layers. Large holes in the background layer, caused by occlusion from foreground layer objects, are filled in by planar or horizontal interpolation. The depth image is further processed by removing isolated points and filling remaining small holes. The foreground objects also leave holes in the texture of building facades, which are filled by horizontal and vertical interpolation in low frequency regions, or by a copy-paste method otherwise. We apply the above steps to a large set of data of downtown Berkeley with several million 3D points, in order to obtain texture-mapped 3D models.     

Modeling and segmentation of floating foreground and background in videos

We propose an on-line algorithm to segment foreground from background in videos captured by a moving camera. In our algorithm, temporal model propagation and spatial model composition are combined to generate foreground and background models, and likelihood maps are computed based on the models. After that, an energy minimization technique is applied to the likelihood maps for segmentation. In the temporal step, block-wise models are transferred from the previous frame using motion information, and pixel-wise foreground/background likelihoods and labels in the current frame are estimated using the models. In the spatial step, another block-wise foreground/background models are constructed based on the models and labels given by the temporal step, and the corresponding per-pixel likelihoods are also generated. A graph-cut algorithm performs segmentation based on the foreground/background likelihood maps, and the segmentation result is employed to update the motion of each segment in a block; the temporal model propagation and the spatial model composition step are re-evaluated based on the updated motions, by which the iterative procedure is implemented. We tested our framework with various challenging videos involving large camera and object motions, significant background changes and clutters.     

一种将羽毛球比赛的2D视频转换到3D视频的算法

文中提出一种羽毛球比赛的2D视频转换到3D视频的算法。在这类视频中,前景是最受关注的部分,准确地从背景中提取出前景对象是获取深度图的关键。文中采用一种改进的图割算法来获取前景,并根据场景结构构建背景深度模型,获取背景深度图;在背景深度图的基础上,根据前景与镜头之间的距离关系为前景对象进行深度赋值,从而得到前景深度图。然后,融合背景深度图和前景深度图,得到完整的深度图。最后,通过基于深度图像的虚拟视点绘制技术DIBR来获取用于3D显示的立体图像对。实验结果表明,最终生成的立体图像对具有较好的3D效果。     

图切割支持的融合颜色和梯度特征的实时背景减除方法

通过融合图像的颜色和梯度特征,实现了一种实时背景减除方法.首先融合颜色和梯度特征建立新的能量函数;然后基于图切割算法最小化能量函数,并对前景/背景进行分割;最后使用光流验证前景区域的真实性,并更新背景模型.对不同场景的实验结果表明：该方法可以实时地检测出视频序列中的运动物体,结果准确、有效.     

基于Kinect深度数据的视频艺术化处理

利用深度摄像机Kinect for XBOX360提取视频的深度信息来实现视频前景和背景的分离,并分别对视频前景和背景进行多风格的艺术渲染,从而获取更好的视频风格化定制效果。首先,系统利用Kinect深度数据实现视频前景的提取;然后在光流场指导下,利用基于纹理传输的方法对视频前景和背景进行不同风格的艺术化渲染;最后,将风格化后的前景视频和背景视频进行融合,从而得到最终的风格化艺术视频。另外,由于采用纹理传输的方式实现对视频的艺术化处理,因此用户可以选择不同的纹理样本来实现自定义的多风格艺术渲染。经过实验测试,前景和背景视频融合后生成的风格化视频取得了较好的艺术效果,从而证明了该系统具有较好的视频前景提取能力和视频风格化渲染能力。     

A probabilistic approach for foreground and shadow segmentation in monocular image sequences

This paper presents a novel method of foreground and shadow segmentation in monocular indoor image sequences. The models of background, edge information, and shadow are set up and adaptively updated. A Bayesian network is proposed to describe the relationships among the segmentation label, background, intensity, and edge information. A maximum a posteriori—Markov random field estimation is used to boost the spatial connectivity of segmented regions.     

基于前景增强与背景抑制的显著性物体检测

显著性物体检测的关键在于准确地突出前景区域,多数传统方法在处理复杂背景图像时效果不理想。针对上述问题,提出了一种基于前景增强与背景抑制的显著性物体检测方法。首先,利用简单线性迭代聚类(SLIC)将图像进行分割得到多个超像素区域,通过区域间的对比和边界信息分别获得图像的显著区域与背景种子,并通过计算得到基于区域间对比和基于背景的两幅显著图。然后,在两幅图像中运用Seam Carving和Graph based的图像分割法区分显著与非显著区域,进而得到前景增强与背景抑制模板。最终,融合两幅显著图与模板得到最终的显著图。在公开数据集MSRA 1000上对算法进行验证,结果表明,所提算法与7种主流算法相比具有更好的查准率和查全率。     

联合聚焦度和传播机制的光场图像显著性检测

目的 图像显著性检测方法对前景与背景颜色、纹理相似或背景杂乱的场景,存在背景难抑制、检测对象不完整、边缘模糊以及方块效应等问题。光场图像具有重聚焦能力,能提供聚焦度线索,有效区分图像前景和背景区域,从而提高显著性检测的精度。因此,提出一种基于聚焦度和传播机制的光场图像显著性检测方法。方法 使用高斯滤波器对焦堆栈图像的聚焦度信息进行衡量,确定前景图像和背景图像。利用背景图像的聚焦度信息和空间位置构建前/背景概率函数,并引导光场图像特征进行显著性检测,以提高显著图的准确率。另外,充分利用邻近超像素的空间一致性,采用基于K近邻法（K-nearest neighbor,K-NN）的图模型显著性传播机制进一步优化显著图,均匀地突出整个显著区域,从而得到更加精确的显著图。结果 在光场图像基准数据集上进行显著性检测实验,对比3种主流的传统光场图像显著性检测方法及两种深度学习方法,本文方法生成的显著图可以有效抑制背景区域,均匀地突出整个显著对象,边缘也更加清晰,更符合人眼视觉感知。查准率达到85.16%,高于对比方法,F度量（F-measure）和平均绝对误差（mean absolute error,MAE）分别为72.79%和13.49%,优于传统的光场图像显著性检测方法。结论 本文基于聚焦度和传播机制提出的光场图像显著性模型,在前/背景相似或杂乱背景的场景中可以均匀地突出显著区域,更好地抑制背景区域。     

一种图象前背景分离的方法及实现

本文提出一种有效的图像的前背景分离算法及其实现。本文的基本思想是利用Meanshift算法对图象进行预分割,然后利用图论的观点对图象进行分割,最后利用matting算法对处理结果的局部进行优化,得到最终结果。实验结果表明,这种方法在仅需要少量用户输入情况之下,能够得到较好的分割效果。     

基于邻域相关性和帧间连续性的前景目标分割

提出了一种新的运动目标分割算法。首先利用像素的颜色、空间的和帧间的特性信息结合贝叶斯判别定理对视频图像进行粗分割,得到一个前景目标的二值图,由于该类方法基于像素间彼此独立的假设,导致分割出的前景目标不完整存在很多空洞。其次,基于前景目标局部邻域空间的一致性假设,计算该邻域内像素间的互相关系数;同时,基于背景的帧间连续性和前景的不连续性,计算像素帧间的互相关系数。最后,依据像素的互相关系数在该邻域内进行二次判决,以填补粗分割中前景目标内部的空洞。实验表明,在复杂背景交通视频中该分割算法具有较强的鲁棒性,并能获得更完整准确的前景目标。     

Using texture to complement color in image matting

Current image matting methods based on color sampling use color to distinguish between foreground and background pixels. However, they fail when the corresponding color distributions overlap. Other methods that define correlation between neighboring pixels based on color aim to propagate the opacity parameter α from known pixels to unknown pixels. However, strong edges of textured regions may block the propagation of α. In this paper, a new matting strategy is proposed that delivers an accurate matte by considering texture as a feature that can complement color even if the foreground and background color distributions overlap and the image is a complex one with highly textured regions. The texture feature is extracted in such a way as to increase distinction between foreground and background regions. An objective function containing color and texture components is optimized to find the best foreground and background pair among a set of candidate pairs. The effectiveness of proposed method is compared quantitatively as well as qualitatively with other matting methods by evaluating their results on a benchmark dataset and a set of complex images. The evaluations show that the proposed method presented the best among state of the art matting methods.     

Automatic inpainting by removing fence-like structures in RGBD images

Recent inpainting techniques usually require human interactions which are labor intensive and dependent on the user experiences. In this paper, we introduce an automatic inpainting technique to remove undesired fence-like structures from images. Specifically, the proposed technique works on the RGBD images which have recently become cheaper and easier to obtain using the Microsoft Kinect. The basic idea is to segment and remove the undesired fence-like structures by using both depth and color information, and then adapt an existing inpainting algorithm to fill the holes resulting from the structure removal. We found that it is difficult to achieve a satisfactory segmentation of such structures by only using the depth channel. In this paper, we use the depth information to help identify a set of foreground and background strokes, with which we apply a graph-cut algorithm on the color channels to obtain a more accurate segmentation for inpainting. We demonstrate the effectiveness of the proposed technique by experiments on a set of Kinect images.     

Mesh saliency detection via double absorbing Markov chain in feature space

We propose a mesh saliency detection approach using absorbing Markov chain. Unlike most of the existing methods based on some center-surround operator, our method employs feature variance to obtain insignificant regions and considers both background and foreground cues. Firstly, we partition an input mesh into a set of segments using Ncuts algorithm and then each segment is over segmented into patches based on Zernike coefficients. Afterwards, some background patches are selected by computing feature variance within the segments. Secondly, the absorbed time of each node is calculated via absorbing Markov chain with the background patches as absorbing nodes, which gives a preliminary saliency measure. Thirdly, a refined saliency result is generated in a similar way but with foreground nodes extracted from the preliminary saliency map as absorbing nodes, which inhibits the background and efficiently enhances salient foreground regions. Finally, a Laplacian-based smoothing procedure is utilized to spread the patch saliency to each vertex. Experimental results demonstrate that our scheme performs competitively against the state-of-the-art approaches.     

Video chroma keying via global sampling and trimap propagation

Chroma keying is a widely used video editing technique, which finely separates the foreground objects from the background. Two major concerns are involved in chroma keying problems: alpha estimation and foreground color restoration. The alpha values reveal the opacity property of the foreground objects. The foreground color restoration removes the background color influence to the foreground appearance especially at transparent regions and objects’ boundaries. In this paper, the color range of the solid background is well analyzed to automatically separate foreground from background. Global sampling is utilized to robustly and reliably estimate the foreground color at boundaries and transparent regions. Furthermore, we propose to propagate the geometric shape of foreground boundaries between adjacent frames by using optical flow and thin plate splines interpolation. The trimap, which is an initial foreground/background/unknown segmentation of each frame can be automatically updated for each video frame by using our proposed propagation method. Compared to previous methods, our proposed matting method estimates high-quality alpha matte and reliable foreground color with least user interference.     

Target tracking based on foreground probability

According to the mean shift tracking algorithm, weights are used to reduce the background interference. However, weights also weaken the representation of the target at the same time. In order to reduce of weakness for model representation brought by the weight, weighted fusion which is composed of target model, candidate model and the probability that the pixel belonging to foreground is proposed to enhance the difference between foreground and background. The purpose is to resist the affection brought by background pixels. Firstly, weak classifiers composed of color and texture features are deduced by Bayesian and update the weak classifiers by changing the parameters of the Gauss distribution. Projection vector to distinguish the foreground and background is found through iteration. Then the projection vector obtained by foreground probability map and weight in mean shift is fused. The projection vector that strengthens the difference between foreground and background is updated to adapt to the changes of illumination or background. Finally, the target center position, scale and rotation angle are determined to achieve the target tracking by the moment features based on the improved weight.     

基于动态阈值的核密度估计前景检测算法

为解决煤层气开采(CBM)现场中抽水机往复运动和风吹草动等动态环境对前景检测的干扰及核密度估计(KDE)目标检测法实时性差的问题,提出了一种改进核密度估计前景检测算法。该方法先用背景差分法(BS)融合三帧差算法将图像分割成动态背景区与非动态背景区,对于动态背景区再用核密度算法分割前景。分割前景时提出了一种新的动态阈值求取方法,综合了相邻样本绝对差均值和样本方差来确定窗宽,并用定时更新与实时更新相结合的策略更新第二背景模型,在替换样本时用随机抽取策略代替先进先出(FIFO)方式。仿真结果表明,改进核密度估计算法与核密度估计法和背景差分核密度估计(BS-KDE)法相比,平均每帧图像算法耗时分别降低了94.18%和15.38%,识别的运动目标也更为完整。实验结果表明所提算法在煤层气开采场景中能准确检测到前景,并基本满足标清视频监控实时性要求。     

基于图像抠图技术的多聚焦图像融合方法

针对多聚焦图像融合容易出现信息丢失、块效应明显等问题,提出了一种新的基于图像抠图技术的多聚焦图像融合算法。首先,通过聚焦检测获得源图像的聚焦信息,并根据所有源图像的聚焦信息生成融合图像的三分图,即前景、背景和未知区域;然后,利用图像抠图技术,根据三分图获得每一幅源图像的精确聚焦区域;最后,将这些聚焦区域结合起来构成融合图像的前景和背景,并根据抠图算法得到的确定前景、背景对未知区域进行最优融合,增强融合图像前景、背景与未知区域相邻像素之间的联系,实现图像融合。实验结果表明,与传统算法相比,所提算法在客观评价方面能获得更高的互信息量（MI）和边缘保持度,在主观评价方面能有效抑制块明显效应,得到更优的视觉效果。该算法可以应用到目标识别、计算机视觉等领域,以期得到更优的融合效果。     

Hybrid learning of Bayesian multinets for binary classification

We propose a scoring criterion, named mixture-based factorized conditional log-likelihood (mfCLL), which allows for efficient hybrid learning of mixtures of Bayesian networks in binary classification tasks. The learning procedure is decoupled in foreground and background learning, being the foreground the single concept of interest that we want to distinguish from a highly complex background. The overall procedure is hybrid as the foreground is discriminatively learned, whereas the background is generatively learned. The learning algorithm is shown to run in polynomial time for network structures such as trees and consistent κ-graphs. To gauge the performance of the mfCLL scoring criterion, we carry out a comparison with state-of-the-art classifiers. Results obtained with a large suite of benchmark datasets show that mfCLL-trained classifiers are a competitive alternative and should be taken into consideration.     

Efficient Depth Propagation for Constructing a Layered Depth Image from a Single Image

In this paper, we propose an interactive technique for constructing a 3D scene via sparse user inputs. We represent a 3D scene in the form of a Layered Depth Image (LDI) which is composed of a foreground layer and a background layer, and each layer has a corresponding texture and depth map. Given user‐specified sparse depth inputs, depth maps are computed based on superpixels using interpolation with geodesic‐distance weighting and an optimization framework. This computation is done immediately, which allows the user to edit the LDI interactively. Additionally, our technique automatically estimates depth and texture in occluded regions using the depth discontinuity. In our interface, the user paints strokes on the 3D model directly. The drawn strokes serve as 3D handles with which the user can pull out or push the 3D surface easily and intuitively with real‐time feedback. We show our technique enables efficient modeling of LDI that produce sufficient 3D effects.