Detecting and extracting natural snow from videos

In this paper, we propose a new algorithm to detect and extract natural snow from video. We detect the snow particle from images or videos by a series of filters, and each of these filters can recognize the features of snow efficiently. We label snow in videos and extract the alpha value of the snow particles by alpha matting. Our method can be applied to many fields such as background reconstruction. Experimental results show that our method is effective.     

A PatchMatch‐based Approach for Matte Propagation in Videos

Despite considerable advances in natural image matting over the last decades, video matting still remains a difficult problem. The main challenges faced by existing methods are the large amount of user input required, and temporal inconsistencies in mattes between pairs of adjacent frames. We present a temporally‐coherent matte‐propagation method for videos based on PatchMatch and edge‐aware filtering. Given an input video and trimaps for a few frames, including the first and last, our approach generates alpha mattes for all frames of the video sequence. We also present a user scribble‐based interface for video matting that takes advantage of the efficiency of our method to interactively refine the matte results. We demonstrate the effectiveness of our approach by using it to generate temporally‐coherent mattes for several natural video sequences. We perform quantitative comparisons against the state‐of‐the‐art sparse‐input video matting techniques and show that our method produces significantly better results according to three different metrics. We also perform qualitative comparisons against the state‐of‐the‐art dense‐input video matting techniques and show that our approach produces similar quality results while requiring only about 7% of the amount of user input required by such techniques. These results show that our method is both effective and user‐friendly, outperforming state‐of‐the‐art solutions.     

数字抠图技术综述

数字抠图是一种把图像的前景部分从背景中分离出来的技术,它通过用户指定图像中少量部分前景和背景区域,并根据这些提示按照一定的判定规则自动、准确地分离出所有的前景物体.回顾了该技术的历史发展过程,包括光学场景抽取、蓝屏抠图和图像合成技术,重点介绍了数字蓝屏抠图、自然图像抠图以及数字抠图技术的扩展和应用,如环境抠图、阴影抠图、视频抠图等,并对这几种技术的局限性以及发展方向做了简要的说明.     

一种鲁棒视频抠图算法*

基于鲁棒抠图算法提出了一种鲁棒视频抠图算法。新算法给出了一种基于少量手工标记的Strokes生成Trimap的方法,为视频帧半自动生成较为粗糙的Trimap划分,同时结合标记传递策略大大减少视频抠图过程中的手工标记操作。理论分析和实验结果证明鲁棒视频抠图算法能够获得较优的视频抠图结果。     

A Progressive Tri‐level Segmentation Approach for Topology‐Change‐Aware Video Matting

Previous video matting approaches mostly adopt the “binary segmentation + matting” strategy, i.e., first segment each frame into foreground and background regions, then extract the fine details of the foreground boundary using matting techniques. This framework has several limitations due to the fact that binary segmentation is employed. In this paper, we propose a new supervised video matting approach. Instead of applying binary segmentation, we explicitly model segmentation uncertainty in a novel tri‐level segmentation procedure. The segmentation is done progressively, enabling us to handle difficult cases such as large topology changes, which are challenging to previous approaches. The tri‐level segmentation results can be naturally fed into matting techniques to generate the final alpha mattes. Experimental results show that our system can generate high quality results with less user inputs than the state‐of‐theart methods.     

Exploring Defocus Matting: Nonparametric Acceleration, Super-Resolution, and Off-Center Matting

Defocus matting is a fully automatic and passive method for pulling mattes from video captured with coaxial cameras that have different depths of field and planes of focus. Nonparametric sampling can accelerate the video-matting process from minutes to seconds per frame. In addition, a super-resolution technique efficiently bridges the gap between mattes from high-resolution video cameras and those from low-resolution cameras. Off-center matting pulls mattes for an external high-resolution camera that doesn't share the same center of projection as the low-resolution cameras used to capture the defocus matting data. In this article, we address these limitations and extend defocus matting in several important ways     

Temporally coherent and spatially accurate video matting

Image and video matting are still challenging problems in areas with low foreground‐background contrast. Video matting also has the challenge of ensuring temporally coherent mattes because the human visual system is highly sensitive to temporal jitter and flickering. On the other hand, video provides the opportunity to use information from other frames to improve the matte accuracy on a given frame. In this paper, we present a new video matting approach that improves the temporal coherence while maintaining high spatial accuracy in the computed mattes. We build sample sets of temporal and local samples that cover all the color distributions of the object and background over all previous frames. This helps guarantee spatial accuracy and temporal coherence by ensuring that proper samples are found even when distantly located in space or time. An explicit energy term encourages temporal consistency in the mattes derived from the selected samples. In addition, we use localized texture features to improve spatial accuracy in low contrast regions where color distributions overlap. The proposed method results in better spatial accuracy and temporal coherence than existing video matting methods.     

Automatic Real-Time Video Matting Using Time-of-Flight Camera and Multichannel Poisson Equations

This paper presents an automatic real-time video matting system. The proposed system consists of two novel components. In order to automatically generate trimaps for live videos, we advocate a Time-of-Flight (TOF) camera-based approach to video bilayer segmentation. Our algorithm combines color and depth cues in a probabilistic fusion framework. The scene depth information returned by the TOF camera is less sensitive to environment changes, which makes our method robust to illumination variation, dynamic background and camera motion. For the second step, we perform alpha matting based on the segmentation result. Our matting algorithm uses a set of novel Poisson equations that are derived for handling multichannel color vectors, as well as the depth information captured. Real-time processing speed is achieved through optimizing the algorithm for parallel processing on graphics hardware. We demonstrate the effectiveness of our matting system on an extensive set of experimental results.     

一种基于时空一致性的视频抠取方法

提出了一种交互式的时空一致性视频抠取方法,能够有效地从视频序列中提取出移动的前景目标.只需在视频体上简单地勾画时空的前景和背景线条,然后构造一个拉普拉斯方程驱动线条在三维的视频体上进行扩散,得到一个粗糙的抠取结果.随后,采用一种新颖的保持时空一致性的抠取技术,利用局部的统计信息和邻域信息,通过少数的迭代即可收敛到全局最优的抠取结果.最后,最优化一个新的全局代价函数在整个三维体上重建前景颜色,如实地保留了抠图结果的时空连贯性.算法的每一步计算都可转化为线性方程组进行求解,因此对于千万像素级的视频数据,也能快速得到高质量的抠取结果.通过对复杂的视频序列进行测试,展示了高质量的抠图结果和算法的高效性.     

针对视频序列分类的快速抠像技术

针对视频序列分类的问题提出了一种快速抠像技术.根据视频序列间的相关度进行关键帧的区分,得到关键帧、序列间变化细微的非关键帧、序列间变化较大的非关键帧;对于关键帧,采用闭合式的抠像方法来进行处理,获得透明度值、前景像素值和背景像素值;对于变化细微的非关键帧,提出了一种基于帧间连续性的透明度值估计和优化方法;对于变化较大的非关键帧,提出了一种基于特征流传递的机制来传递关键帧的有效信息.实验结果表明,最终在获得与传统方法相比可接受的抠像效果条件下,这种快速抠像技术缩短了处理时间.     

基于模糊估计融合显著性检测的自动抠图算法

提出了一种新的自动抠图算法框架。首先,估计输入图像各个区域的模糊程度;其次,对图像进行显著性的计算;然后融合模糊度和显著性信息,大致分割出前景和背景,从而自动生成标注前景、背景、未知区域的三色图;最后,采用基于采样的抠图算法就可以准确地完成前景目标物体的自动抠图。该算法无须人工辅助或附加信息。在标准数据集和实拍图像上的实验结果证明了该算法的准确性和实用性,可广泛应用于图像和视频的编辑合成。     

融合多线索信息的数字图像抠图方法研究

针对自然图像抠图方法中存在对先验知识过度依赖和交互输入繁琐的问题,为了扩展自然图像抠图方法的使用范围,提升自然图像抠图方法的自动化程度,提出一种融合多线索信息的数字图像抠图方法。利用原始自然图像所对应的深度信息和视觉显著度信息进行感兴趣区域粗分割;利用形态学的膨胀与腐蚀算法对感兴趣区域的分割结果进行粗分割区域膨胀和粗分割区域腐蚀操作,从而得到抠图过程所需的三分元素图;利用彩色纹理图像和三分元素图,并结合使用相似性传递抠图方法获得精细的前景目标抠图结果。实验结果表明,该方法不仅能够得到较为理想的抠图效果,而且大大提升了自然图像抠图方法的自动化程度。     

A novel compact yet rich key frame creation method for compressed video summarization

Video summarization has great potential to enable rapid browsing and efficient video indexing in many applications. In this study, we propose a novel compact yet rich key frame creation method for compressed video summarization. First, we directly extract DC coefficients of I frame from a compressed video stream, and DC-based mutual information is computed to segment the long video into shots. Then, we select shots with static background and moving object according to the intensity and range of motion vector in the video stream. Detecting moving object outliers in each selected shot, the optimal object set is then selected by importance ranking and solving an optimum programming problem. Finally, we conduct an improved KNN matting approach on the optimal object outliers to automatically and seamlessly splice these outliers to the final key frame as video summarization. Previous video summarization methods typically select one or more frames from the original video as the video summarization. However, these existing key frame representation approaches for video summarization eliminate the time axis and lose the dynamic aspect of the video scene. The proposed video summarization preserves both compactness and considerably richer information than previous video summaries. Experimental results indicate that the proposed key frame representation not only includes abundant semantics but also is natural, which satisfies user preferences.     

复杂环境下基于信息融合的视频实时抠像算法

针对复杂场景下传统的视频抠像算法对目标物体与背景纹理相似或边界不清晰的图像分割困难的问题,提出了一种基于视觉传感器和激光雷达信息相融合的视频实时抠像算法。该算法从原始激光雷达点云数据中获取感兴趣区域深度信息,并作为先验知识融合到改进的谱抠图算法,创建感兴趣区域深度抠图拉普拉斯矩阵,通过聚类算法最优迭代得出抠像结果,并运用导向滤波器对抠像结果进行后处理。实验证明,对比于融合深度信息的传统算法和没有融合其他信息的算法,该算法降低了欠分割率、提高了运行效率,抠像目标的边缘信息也更加饱满、清晰、平滑。     

A Markov Random Field Model-Based Approach to Natural Image Matting

This paper proposes a Markov Random Field (MRF) model-based approach to natural image matting with complex scenes. After the trimap for matting is given manually, the unknown region is roughly segmented into several joint sub-regions. In each sub-region, we partition the colors of neighboring background or foreground pixels into several clusters in RGB color space and assign matting label to each unknown pixel. All the labels are modelled as an MRF and the matting problem is then formulated as a maximum a posteriori (MAP) estimation problem. Simulated annealing is used to find the optimal MAP estimation. The better results can be obtained under the same user-interactions when images are complex. Results of natural image matting experiments performed on complex images using this approach are shown and compared in this paper.     

自然图像抠图技术综述

着重对近几年自然图像抠图研究新进展进行综述,针对近年来基于启发式优化的抠图研究以及基于深度学习的抠图研究新成果对抠图研究工作进行重新分类和梳理,分析不同类型自然图像抠图研究主要的方法并整理其中的关键问题,介绍常用的自然图像抠图评价指标。最后通过实例分析总结了当前自然图像抠图研究面临的挑战,并对未来的研究前景进行了探讨。     

基于MRF的复杂图像抠图

所谓复杂图像抠图就是从复杂图像中抠取出目标物体的一种图像处理算法。为了取得更好的抠图效果,提出了一种基于马尔可夫随机场的自然图像抠图方法。该方法首先手工把图像分成3个区域:前景区域、背景区域和未知区域;然后,再将未知区域用手工粗略地划分成几个相交的小区域;接着在每一个小区域内,以其中的未知区域的像素点为节点,定义抠图标号,同时在这些节点上面建立MRF抠图模型,并把这些标号赋给这些节点,这样抠图问题被定义为在这个MRF模型和它的Gibbs分布上MAP估计问题;继而再计算出每个小区域的掩像;最后把这些掩像合并,即得到输入图像最终的掩像。和其他算法相比,对复杂图像的抠图问题,该方法可以取得更好的抠图效果。     

Adaptive propagation matting based on transparency of image

Image matting is an essential technique in many image and video editing applications. Although many matting methods have been proposed, it is still a challenge for most to obtain satisfactory matting results in the transparent foreground region of an image. To solve this problem, this paper proposes a novel matting algorithm, i.e. adaptive transparency-based propagation matting (ATPM) algorithm. ATPM algorithm considers image matting from a new slant. We pay attention to the transparencies of the input images and creatively assign them into three categories (highly transparent, strongly transparent and little transparent) according to the transparencies of the foreground objects in the images. Our matting model can make relevant adjustment in terms of the transparency types of the input images. Moreover, many current matting methods do not perform well when the foreground and background regions have similar color distributions. Our method adds texture as an additional feature to effectively discriminate the foreground and background regions. Experimental results on the benchmark dataset show that our method gets high-quality matting results for images of three transparency types, especially provides more accurate results for highly transparent images comparing with the state-of-the-art methods.     

A closed-form solution to natural image matting

Interactive digital matting, the process of extracting a foreground object from an image based on limited user input, is an important task in image and video editing. From a computer vision perspective, this task is extremely challenging because it is massively ill-posed -- at each pixel we must estimate the foreground and the background colors, as well as the foreground opacity ("alpha matte") from a single color measurement. Current approaches either restrict the estimation to a small part of the image, estimating foreground and background colors based on nearby pixels where they are known, or perform iterative nonlinear estimation by alternating foreground and background color estimation with alpha estimation.In this paper we present a closed-form solution to natural image matting. We derive a cost function from local smoothness assumptions on foreground and background colors, and show that in the resulting expression it is possible to analytically eliminate the foreground and background colors to obtain a quadratic cost function in alpha. This allows us to find the globally optimal alpha matte by solving a sparse linear system of equations. Furthermore, the closed-form formula allows us to predict the properties of the solution by analyzing the eigenvectors of a sparse matrix, closely related to matrices used in spectral image segmentation algorithms. We show that high quality mattes for natural images may be obtained from a small amount of user input.     

局部自适应输入控制的随机游走抠图

针对传统性抠图算法中,非完全正确用户标注及不精确超像素分割造成的信息误扩散,以随机游走算法为基础,提出带软性约束的抠图算法。通过对扩展Dirichlet问题的推导,指出带软约束的随机游走与部分自吸收随机游走概率的关联性。以吸收概率为指导,在传统相似扩散所构建的图模型上,根据局部窗口内特征矩阵的秩与方差设计了输入控制矩阵,使得信息扩散的过程能够跟随图像的局部特征进行自适应扩散。最后将软约束随机游走应用到单帧双层抠图及视频抠图中。实验表明,所提算法具有信息远距传播能力和良好的容错性能,尤其在用户标注不够充分的情况下能够取得更加优良的抠图结果。