复杂运动摄像机拍摄视频的背景修复技术

视频背景修复问题正受到越来越多的关注,对于复杂运动的摄像机所拍摄的视频而言,该问题具有更高的难度。针对此问题,提出一种由运动场引导的优化算法,填补由于去掉前景物体所留下的视频体空洞。首先,为了估计视频空洞部分的运动场,构建全局目标方程并利用分层次迭代的方法求得其最优解;修复问题继而被抽象为马尔可夫随机场问题。将运动场作为引导,最优地从已知区域选择可用的像素修复视频的背景。最后,改进传统的光照迁移方法,提出一种亮度调整策略,消除修复区域光照不连续的现象。该算法在多种不同类型的视频上均取得良好的效果。与现有算法相比,该算法能更好地保证时空连续性,并能修复由复杂运动的摄像机所拍摄的、含有复杂背景的视频。     

基于时空注意模型的视频分割算法

针对已有视频分割算法对复杂动态背景下所出现的误分割问题,提出通过显著性映射构造时空注意特征,并采用分层条件随机场进行视频分割,提高分割准确率。算法首先根据视觉注意理论提取时域和空域特征,并建立加权混合模型。其次,采用该混合模型计算运动目标的显著性映射概率分布,有效地提取出运动目标区域。最后,在显著性映射概率分布基础上,采用高斯混合模型建立前景和背景的能量函数,构造分层条件随机场模型对这些特征能量函数进行分割建模,精确地提取出运动对象目标。实验结果表明,该算法即使对复杂动态背景下的视频也能够得到稳定的分割效果,有效地去除摄像机运动等所导致的误分割问题。     

基于保边滤波的显著目标快速分割方法

在视频中自动发掘目标并对其进行精确分割是一个非常有挑战性的计算机视觉问题。本文提出了一种基于保边滤波的显著目标快速分割方法。首先,通过融合外观特征与运动特征,将视频中的显著目标发掘转为能量函数最小化问题进行求解。其次,为了更精确地进行分割目标,融合外观的高斯混合外观模型(Gaussian mixture mode,GMM)、位置先验以及时空平滑约束构建马尔科夫随机场(Markov random field,MRF)模型,并使用图割算法进行求解。本文提出的基于保边滤波的显著目标快速分割方法,在牺牲较少的精度下,极大地提高了分割效率。最后在两个数据集上进行了对比实验,实验结果表明,本文算法的分割精度超过了其他5种目标分割方法,且加速算法在损失少量精度的情况下提高了2倍分割效率。     

基于多分辨率马尔可夫随机场运动目标分割

提出一种基于三维时空小波变换和马尔可夫随机场(Markov Random Field)模型的多分辨率运动目标分割算法.该算法利用三维时空小波变换对图像序列进行分解得到多分辨率的图像序列,并在此基础上建立多分辨率的马尔可夫随机场模型,构造相应的能量函数.通过条件迭代模型优化算法(Iterated Conditional Modes)求解能量函数的最优解,得出标记场,提取出运动目标.实验结果证明,该算法能够很好地消除了单一分辨率的MRF运动检测结果中"空洞"现象,对运动目标分割具有很好的分割效果.     

采用多组单应约束和马尔可夫随机场的运动目标检测算法

针对现有动态背景下目标检测算法的局限性,提出一种基于多组单应约束和马尔可夫随机场的运动目标检测算法.该算法以视频序列多帧跟踪的运动轨迹为基础,通过轨迹分离和像素标记2个阶段实现运动目标的检测:在轨迹分离阶段,利用多组单应约束对视频序列的背景运动进行建模,并基于该约束通过累积确认的策略实现背景轨迹和前景轨迹的准确分离;在像素标记阶段,以超像素为节点建立时空马尔可夫随机场模型,将轨迹分离信息以及超像素的时空邻域关系统一建模在马尔可夫随机场的能量函数中,并通过最小化能量函数得到每个像素的前背景标记结果.与现有基于运动轨迹的方法相比,文中算法不需要仿射摄像机模型的假设,有效地解决了运动轨迹等长带来的轨迹点区域缺失问题,并可同时处理静态背景和动态背景2种类型的视频;在多个公开数据集的测试结果表明,该算法在轨迹分离准确性、轨迹点密度以及像素标记准确率等方面均优于现有方法.     

基于多分辨率马尔可夫随机场运动目标分割

提出一种基于三维时空小波变换和马尔可夫随机场(MarkovRandomField)模型的多分辨率运动目标分割算法。该算法利用三维时空小波变换对图像序列进行分解得到多分辨率的图像序列,并在此基础上建立多分辨率的马尔可夫随机场模型,构造相应的能量函数。通过条件迭代模型优化算法(IteratedConditionalModes)求解能量函数的最优解,得出标记场,提取出运动目标。实验结果证明,该算法能够很好地消除了单一分辨率的MRF运动检测结果中"空洞"现象,对运动目标分割具有很好的分割效果。     

基于SLIC和马尔科夫随机场的H.264视频对象分割

为了提高H.264压缩域视频对象分割时的鲁棒性和准确性,提出了一种基于简单线性迭代聚类(SLIC)和图割优化的马尔科夫随机场(MRF)运动对象分割算法.算法直接利用从摄像机产生的H.264压缩码流中提取的运动矢量.首先对运动矢量场进行预处理,然后构建基于改进的SLIC分割的马尔科夫模型能量函数,最后利用图割法求解能量函数进而分割出运动对象.在公开的数据集上进行实验表明,与近年来经典压缩域视频对象分割算法相比,上述算法在复杂背景下可以有效提高分割的准确率和F度量,运算速度平均提高约1.85倍.与先进的像素域分割方法相比,运算速度提高了5倍,算法适用于实时性要求较高的视频监控场合,可有效减少数据存储和处理的内存需求.     

基于马尔可夫随机场与活动轮廓的运动目标分割

针对视频序列图像中的运动目标分割，提出了将马尔可夫随机场模型和活动轮廓模型相结合的运动目标分割算法。该算法首先利用马尔可夫随机场模型的运动检测算法，得到运动目标的初始模板。在此基础上提取出活动轮廓模型的初始轮廓点，然后构造活动轮廓模型的能量函数。用改进的贪婪算法求得能量函数最小值，提取出运动目标的精确轮廓，从而得到具有精确边缘的运动目标。实验结果表明该算法能有效地分割和提取出视频序列中的运动目标。     

面向智能视频监控中的运动目标检测方法研究

运动目标检测是实现智能视频监控的基础,针对当前运动目标检测方法在复杂场景中适应性差的问题,提出了一种结合时空马尔可夫随机场模型和高斯混合模型的运动目标检测方法。在训练时空马尔可夫随机场模型时,采用高斯混合模型的参数更新算法计算邻域图像分割区域的均值和方差,并通过时空邻域标记场设置势函数。通过与传统目标检测方法的仿真比较,验证了该方法的优越性。结果表明,与传统的目标检测方法相比,该方法在复杂场景下具有更高的检测精度,能够更清晰地分割前景中的运动目标。     

一种基于小波变换的马尔可夫随机场的视频对象分割

马尔可夫随机场（Markov Random Field,MRF）理论已经被广泛地应用于视频图像的分割。提出一种基于小波变换的马尔可夫随机场模型的视频对象分割算法。该算法利用小波变换将图像序列分解到小波域,并在此基础上建立马尔可夫随机场模型,构造相应的能量函数。通过迭代求解能量函数的最优解,得出标记场,提取出运动对象。仿真结果表明,该算法能够有效地抑制噪声,提高构成对象边界像素的数量,快速有效地提取出视频对象。     

Space-time completion of video

This paper presents a new framework for the completion of missing information based on local structures. It poses the task of completion as a global optimization problem with a well-defined objective function and derives a new algorithm to optimize it. Missing values are constrained to form coherent structures with respect to reference examples. We apply this method to space-time completion of large space-time "holes" in video sequences of complex dynamic scenes. The missing portions are filled in by sampling spatio-temporal patches from the available parts of the video, while enforcing global spatio-temporal consistency between all patches in and around the hole. The consistent completion of static scene parts simultaneously with dynamic behaviors leads to realistic looking video sequences and images. Space-time video completion is useful for a variety of tasks, including, but not limited to: 1) sophisticated video removal (of undesired static or dynamic objects) by completing the appropriate static or dynamic background information. 2) Correction of missing/corrupted video frames in old movies. 3) Modifying a visual story by replacing unwanted elements. 4) Creation of video textures by extending smaller ones. 5) Creation of complete field-of-view stabilized video. 6) As images are one-frame videos, we apply the method to this special case as well     

3D Motion Completion in Crowded Scenes

Crowded motions refer to multiple objects moving around and interacting such as crowds, pedestrians and etc. We capture crowded scenes using a depth scanner at video frame rates. Thus, our input is a set of depth frames which sample the scene over time. Processing such data is challenging as it is highly unorganized, with large spatio‐temporal holes due to many occlusions. As no correspondence is given, locally tracking 3D points across frames is hard due to noise and missing regions. Furthermore global segmentation and motion completion in presence of large occlusions is ambiguous and hard to predict. Our algorithm utilizes Gestalt principles of common fate and good continuity to compute motion tracking and completion respectively. Our technique does not assume any pre‐given markers or motion template priors. Our key‐idea is to reduce the motion completion problem to a 1D curve fitting and matching problem which can be solved efficiently using a global optimization scheme. We demonstrate our segmentation and completion method on a variety of synthetic and real world crowded scanned scenes.     

Video retargeting combining warping and summarizing optimization

We construct a unified interactive video retargeting system for video summarization, completion, and reshuffling. Our system combines the advantages of both video warping and summarizing processing. We first warp the video to present initial editing results, then refine the results using patch-based summarizing optimization, which mainly eliminates possible distortion produced in the warping step. We develop a Mean Value Coordinate (MVC) warping method due to its simplicity and efficiency used in the initialization. For refining processing, the summarization optimization is built on a 3D bidirectional similarity measure between the original and edited video, to preserve the coherence and completeness of the final editing result. We further improve the quality of summarization by applying a color histogram matching during the optimization, and accelerate the summarization optimization by using a constrained 3D Patch-Match algorithm. Experiment results show that the proposed video retargeting system effectively supports video summarization, completion, and reshuffling while avoiding issues like texture broken, video jittering, and detail losing.     

面向三维视频的虚拟视点合成技术研究进展

基于深度图像的虚拟视点合成技术是三维视频信息处理、虚拟现实和计算机图形学领域的新兴交叉技术。介绍了三维视频虚拟视点绘制（或合成）技术的原理;分别就虚拟视点合成过程产生的重叠、伪影、空洞三种问题的产生机理进行分析总结;分类综述了空洞填补中基于背景重建的方法、基于图像补全的方法和基于相关图像的修复方法,并列举了提升合成图像质量的方法。总结并展望了深度学习等技术是虚拟视点合成技术的未来研究方向。     

面向行人移除的街景图像全局优化时空修补方法

针对街景图像中往往包含大量行人等隐私对象的问题,以移除图像中的行人为例,提出一种全局优化的时空图像修补方法.首先利用运动获得结构算法建立参考图像与目标图像之间的对应关系,该过程不依赖场景的简化假设,使得该图像修补方法适合各类复杂场景;然后对待修补区域建立马尔科夫随机场,通过合理设计标号集和能耗函数,把时域和空域修补结合到同一优化过程中,并自动判断何时选择何种修补方式,使修补结果尽量符合实际场景同时又具有较好的视觉一致性.大量实验结果表明,该方法对各种复杂场景的街景图像都能够得到较好的修补效果.     

3D convolutional networks with multi-layer-pooling selection fusion for video classification

C3D has been widely used for video representation and understanding. However, it is performed on spatio-temporal contexts in a global view, which often weakens its capacity of learning local representation. To alleviate this problem, a concise and novel multi-layer feature fusion network with the cooperation of local and global views is introduced. For the current network, the global view branch is used to learn the core video semantics, while the local view branch is used to capture the contextual local semantics. Unlike traditional C3D model, the global view branch can only provide the big view branch with the most activated video features from a broader 3D receptive field. Via adding such shallow-view contexts, the local view branch can learn more robust and discriminative spatio-temporal representations for video classification. Thus we propose 3D convolutional networks with multi-layer-pooling selection fusion for video classification, the integrated deep global feature is combined with the information originated from shallow layer of local feature extraction networks, through the space-time pyramid pooling, adaptive pooling and attention pooling three different pooling units, different time–space feature information is obtained, and finally cascaded and used for classification. Experiments on the UCF-101 and HMDB-51 datasets achieve correct classification rate 95.0% and 72.2% respectively. The results show that the proposed 3D convolutional networks with multi-layer-pooling selection fusion has better classification performance.

     

Superparsing

This paper presents a simple and effective nonparametric approach to the problem of image parsing, or labeling image regions (in our case, superpixels produced by bottom-up segmentation) with their categories. This approach is based on lazy learning, and it can easily scale to datasets with tens of thousands of images and hundreds of labels. Given a test image, it first performs global scene-level matching against the training set, followed by superpixel-level matching and efficient Markov random field (MRF) optimization for incorporating neighborhood context. Our MRF setup can also compute a simultaneous labeling of image regions into semantic classes (e.g., tree, building, car) and geometric classes (sky, vertical, ground). Our system outperforms the state-of-the-art nonparametric method based on SIFT Flow on a dataset of 2,688 images and 33 labels. In addition, we report per-pixel rates on a larger dataset of 45,676 images and 232 labels. To our knowledge, this is the first complete evaluation of image parsing on a dataset of this size, and it establishes a new benchmark for the problem. Finally, we present an extension of our method to video sequences and report results on a video dataset with frames densely labeled at 1 Hz.     

A lattice-based MRF model for dynamic near-regular texture tracking

A near-regular texture (NRT) is a geometric and photometric deformation from its regular origin - a congruent wallpaper pattern formed by 2D translations of a single tile. A dynamic NRT is an NRT under motion. Although NRTs are pervasive in man-made and natural environments, effective computational algorithms for NRTs are few. This paper addresses specific computational challenges in modeling and tracking dynamic NRTs, including ambiguous correspondences, occlusions, and drastic illumination and appearance variations. We propose a lattice-based Markov-random-field (MRF) model for dynamic NRTs in a 3D spatiotemporal space. Our model consists of a global lattice structure that characterizes the topological constraint among multiple textons and an image observation model that handles local geometry and appearance variations. Based on the proposed MRF model, we develop a tracking algorithm that utilizes belief propagation and particle filtering to effectively handle the special challenges of the dynamic NRT tracking without any assumption on the motion types or lighting conditions. We provide quantitative evaluations of the proposed method against existing tracking algorithms and demonstrate its applications in video editing     

A unified method for appearance and geometry completion of point set surfaces

This paper presents a novel approach for appearance and geometry completion over point-sampled geometry. Based on the result of surface clustering and a given texture sample, we define a global texture energy function on the point set surface for direct texture synthesis. The color texture completion is performed by minimizing a constrained global energy using the existing surface texture on the surface as the input texture sample. We convert the problem of context-based geometry completion into a task of texture completion on the surface. The geometric detail is then peeled and converted into a piece of signed gray-scale texture on the base surface of the point set surface. We fill the holes on the base surface by smoothed extrapolation and the geometric details over these patches are reconstructed by a process of gray-scale texture completion. Experiments show that our method is flexible, efficient and easy to implement. It provides a practical texture synthesis and geometry completion tool for 3D point set surfaces.     

快速结构化图像修补

图像修补的目的是对图像中缺失的区域进行修复,或是将图像中的物体抠去并进行背景填充,以取得融合到难以用肉眼分辨的效果。在图像修补的过程中,较大的结构信息是修补的难点。为此提出了一种快速结构化的图像修补算法,该方法将图像修补分为结构修补与纹理填充两个部分,即在用户指定待修补区域与结构曲线之后,首先定义全局最优化能量函数,并用动态规划与置信度传播的算法将其最小化来完成结构修补;然后对剩余的待修补区域通过按行扫描来进行纹理填充,其中对于边界处的点是使用基于样本的修补算法,而对于待修补区域内部的点,则使用快速的加权Ashikhmin-WL算法,扫描完成后输出修补后的图像;最后实现了一个快速结构化图像修补系统,并给出一些实验结果,从实验结果中可以看到,该方法的修补流程与算法是有实际应用价值的。