Video inpainting under constrained camera motion.

A framework for inpainting missing parts of a video sequence recorded with a moving or stationary camera is presented in this work. The region to be inpainted is general: it may be still or moving, in the background or in the foreground, it may occlude one object and be occluded by some other object. The algorithm consists of a simple preprocessing stage and two steps of video inpainting. In the preprocessing stage, we roughly segment each frame into foreground and background. We use this segmentation to build three image mosaics that help to produce time consistent results and also improve the performance of the algorithm by reducing the search space. In the first video inpainting step, we reconstruct moving objects in the foreground that are "occluded" by the region to be inpainted. To this end, we fill the gap as much as possible by copying information from the moving foreground in other frames, using a priority-based scheme. In the second step, we inpaint the remaining hole with the background. To accomplish this, we first align the frames and directly copy when possible. The remaining pixels are filled in by extending spatial texture synthesis techniques to the spatiotemporal domain. The proposed framework has several advantages over state-of-the-art algorithms that deal with similar types of data and constraints. It permits some camera motion, is simple to implement, fast, does not require statistical models of background nor foreground, works well in the presence of rich and cluttered backgrounds, and the results show that there is no visible blurring or motion artifacts. A number of real examples taken with a consumer hand-held camera are shown supporting these findings.     

A comprehensive survey and mathematical insights towards video summarization

Video Summarization is a technique to reduce the original raw video into a short video summary. Video summarization automates the task of acquiring key frames/segments from the video and combining them to generate a video summary. This paper provides a framework for summarization based on different criteria and also compares different literature work related to video summarization. The framework deals with formulating model for video summarization based on different criteria. Based on target audience/ viewership, number of videos, type of output intended, type of video summary and summarization factor; a model generating video summarization framework is proposed. The paper examines significant research works in the area of video summarization to present a comprehensive review against the framework. Different techniques, perspectives and modalities are considered to preserve the diversity of survey. This paper examines important mathematical formulations to provide meaningful insights for video summarization model creation.     

Hybrid error concealment method combining exemplar-based image inpainting and spatial interpolation

This paper proposes an efficient error concealment method for the reconstruction of pixels that are lost in video communication. The proposed method is developed by combining exemplar-based image inpainting for patch reconstruction and spatial interpolation for pixel reconstruction using adaptive threshold by local complexity. By exemplar-based image inpainting, regions with regular structures are reconstructed. For complex regions with irregular structures, just one pixel is reconstructed using the proposed spatial interpolation method. The proposed spatial interpolation method performs reconstruction by selecting adaptively directional interpolation or neighbor interpolation based on gradient information. Simulation results show that the proposed hybrid method performs reconstruction with significantly improved subjective quality compared with the previous spatial error concealment and image inpainting methods. The proposed method also gives substantial improvements of PSNR compared with the previous methods.     

基于三通道多小波紧标架的图像曲率修复模型

为解决正交小波域图像修复方法的现存小波 系数不能提供丢 失小波系数的充足信息问题,将曲率修复(CDD)模型扩展到非正交小波域,提出一种新的基 于三通道多小波 紧标架的图像CDD模型。进一步考虑到CDD模型的等照度线是按照直线连接的问 题,提出了一种新的CCD模型。新模型的修复在两个方向上进行,在法向上按照改进的曲率 函数进行修复, 在切向上实现输运机制。新模型结合了多小波紧标架分解技术和改进的CDD模型的 优势,利用标架域中 小波系数之间具有的冗余性对缺损的信息进行弥补,对现有CDD模型进行改进。给出了 有效的split Bregman 仿真算法,并采用不同的图像进行了仿真。实验结果表明,新模型对大面积缺损修复及噪声 抑制都具有良好 的修复效果,即使在大量小波系数丢失的情况下,也能保持图像的边缘结构等几何特征,大 大的改善修复质量。     

基于Navier-Stokes方程的视频去隔行方法

随着技术的发展,常需将已有的隔行视频转换成高质量的逐行视频。本文提出一种基于Navier-Stokes（N-S）方程的视频去隔行方法。将基于N-S方程的图像修复模型应用到视频空间,由理想三维流体方程,建立视频的局部约束条件;然后,根据其各向异性特征,导出单个像素点的修复方程;最后,通过对隔行视频间隔区域的逐点修复,实现视频去隔行。实验表明,本文方法能够较好的抑制运动虚像和边沿锯齿。      

An efficient framework for image/video inpainting

Image inpainting has been widely applied to many applications, such as restoring corrupted old photos, erasing video logos, concealing errors in a digital video processing system, and so on. However, traditional geometric inpainting methods suffer low efficiency. To tackle this problem, this paper addresses an efficient transform based framework for geometric methods. Given an image, we firstly decompose it, then separately perform restoration process and finally employ Laplacian diffusion function to hold local texture coherence. Experimental results show that the proposed method not only speeds up and enhances the performances of geometric methods, but also obtains a better restoration results compared with the traditional texture and hybrid methods.     

Decomposing style,content, and motion for videos

In this paper, we present the first video decomposition framework, named SyCoMo, that factorizes a video into style, content, and motion. Such a fine-grained decomposition enables flexible video editing, and for the first time allows for tripartite video synthesis. SyCoMo is a unified and domain-agnostic learning framework which can process videos of various object categories without domain-specific design or supervision. Different from other motion decomposition work, SyCoMo derives motion from style-free content by isolating style from content in the first place. Content is organized into subchannels, each of which corresponds to an atomic motion. This design naturally forms an information bottleneck which facilitates a clean decomposition. Experiments show that SyCoMo decomposes videos of various categories into interpretable content subchannels and meaningful motion patterns. Ablation studies also show that deriving motion from style-free content makes the keypoints or landmarks of the object more accurate. We demonstrate the photorealistic quality of the novel tripartite video synthesis in addition to three bipartite synthesis tasks named as style, content, and motion transfer.     

基于样本和线性结构信息的大范围图像修复算法

基于纹理的图像修复算法对于修复破损区域比较大的图像效果较好,但该算法对于含有结构信息的图像修复效果很差.通过新的优先项的计算、平均值补偿及增加惩罚项提高传统的基于样本的图像修复算法的修复效果,结合图像中常出现的直线和曲线结构特征,提出了基于样本和结构信息的大范围图像修复算法.实验表明,该算法易于实现,修复结果能达到令人满意的效果,具有较高的实用价值.     

Deinterlacing using variational methods

We present a variational framework for deinterlacing that was originally used for inpainting and subsequently redeveloped for deinterlacing. From the framework, we derive a motion adaptive (MA) deinterlacer and a motion compensated (MC) deinterlacer and test them together with a selection of known deinterlacers. To illustrate the need for MC deinterlacing, the problem of details in motion (DIM) is introduced. It cannot be solved by MA deinterlacers or any simpler deinterlacers but only by MC deinterlacers. The major problem in MC deinterlacing is computing reliable optical flow [motion estimation (ME)] in interlaced video. We discuss a number of strategies for computing optical flows on interlaced video hoping to shed some light on this problem. We produce results on challenging real world video data with our variational MC deinterlacer that in most cases are indistinguishable from the ground truth.     

基于深度学习的数字图像修复算法最新进展

数字图像修复是一项利用计算机技术还原破损图像的缺失信息,从而实现自动修复破损图像的技术,其广泛应用于文物修复、图像去雾、电影特效生成等方面。近年来深度学习的发展为图像修复提供了新的思路,即将估计缺失信息的问题转为有条件的图像生成问题。基于深度学习的图像修复研究已成为底层计算机视觉问题的研究热点之一。本文对深度学习在数字图像修复领域的最新进展进行总结归纳,并详细阐述卷积模式和网络结构优化的研究进展,最后对未来的研究方向进行展望。      

基于优选特征轨迹的全分辨率视频稳定

该文提出一种基于优选特征轨迹的视频稳定算法。首先,采用改进的Harris角点检测算子提取特征点,通过K-Means聚类算法剔除前景特征点。然后,利用帧间特征点的空间运动一致性减少错误匹配和时间运动相似性实现长时间跟踪,从而获取有效特征轨迹。最后,建立同时包含特征轨迹平滑度与视频质量退化程度的目标函数计算视频序列的几何变换集以平滑特征轨迹获取稳定视频。针对图像扭曲产生的空白区,由当前帧定义区与参考帧的光流作引导来腐蚀,并通过图像拼接填充仍属于空白区的像素。经仿真验证,该文方法稳定的视频,空白区面积仅为Matsushita方法的33%左右,对动态复杂场景和多个大运动前景均具有较高的有效性并可生成内容完整的视频,既提高了视频的视觉效果,又减轻了费时的边界修复任务。     

Multi-scale gradient attention guidance and adaptive style fusion for image inpainting

Image inpainting aims to fill in the missing regions of damaged images with plausible content. Existing inpainting methods tend to produce ambiguous artifacts and implausible structures. To address the above issues, our method aims to fully utilize the information of known regions to provide style and structural guidance for missing regions. Specifically, the Adaptive Style Fusion (ASF) module reduces artifacts by transferring visual style features from known regions to missing regions. The Gradient Attention Guidance (GAG) module generates accurate structures by aggregating semantic information along gradient boundary regions. In addition, the Multi-scale Attentional Feature Extraction (MAFE) module extracts global contextual information and enhances the representation of image features. The sufficient experimental results on the three datasets demonstrate that our proposed method has superior performance in terms of visual plausibility and structural consistency compared to state-of-the-art inpainting methods.     

Inpainting of binary images using the Cahn-Hilliard equation.

Image inpainting is the filling in of missing or damaged regions of images using information from surrounding areas. We outline here the use of a model for binary inpainting based on the Cahn-Hilliard equation, which allows for fast, efficient inpainting of degraded text, as well as super-resolution of high contrast images.     

Surveillance video synopsis in the compressed domain for fast video browsing

The traditional pixel-domain based video analysis methods have taken dominated places for long. However, due to the rapidly increasing volume and resolution of surveillance video, the desirable fast and scalable browsing encounters significant challenges in terms of efficiency and flexibility. Under this circumstance, operating surveillance video in compressed domain has aroused great concern in academy and industry. In order to perform the intelligent video analysis task on the premise of preserving accuracy and controlling complexity, this paper presents a compressed-domain approach for massive surveillance video synopsis generation, labeling and browsing. The main work and achievements include: (1) a compressed-domain scheme is established to condense the compressed surveillance video and record the synopsis results; (2) a background modeling method via the Motion Vector based Local Binary Pattern (MVLBP) is introduced to extract moving objects in an efficient way; (3) an object flags based synopsis labeling method is proposed to represent the object regions as well as their display modes in a flexible way. Experimental results show that the video analysis system based on this framework can provide not only efficient synopsis generation but also flexible scalable or playback browsing.     

Deep multi-query video retrieval

Video retrieval methods have been developed for a single query. Multi-query video retrieval problem has not been investigated yet. In this study, an efficient and fast multi-query video retrieval framework is developed. Query videos are assumed to be related to more than one semantic. The framework supports an arbitrary number of video queries. The method is built upon using binary video hash codes. As a result, it is fast and requires a lower storage space. Database and query hash codes are generated by a deep hashing method that not only generates hash codes but also predicts query labels when they are chosen outside the database. The retrieval is based on the Pareto front multi-objective optimization method. Re-ranking performed on the retrieved videos by using non-binary deep features increases the retrieval accuracy considerably. Simulations carried out on two multi-label video databases show that the proposed method is efficient and fast in terms of retrieval accuracy and time.     

基于视频修复的运动目标删除篡改行为的检测算法

将运动对象从视频中删除是视频篡改的一种常见 形式,针对删除视频运动对象这一篡改操作, 提出了基于视频修复痕迹的检测方法。运动对象删除后需采用数字视频修复技术填补由于移 除操作产生的 黑洞,使得篡改后的视频遗留有修复痕迹;通过深入分析篡改视频中遗留的修复痕迹,对篡 改后未压缩视 频采用对称帧差法检测运动对象删除区域;对压缩后的篡改视频从运动光流场的角度,由视 频帧光流方向 的不一致性进行检测。实验结果表明,本文方法不依赖于原始视频,计算复杂度低,能够有 效检测运动对象删除操作,并在空时域上对篡改区域进行定位。     

AN HMM BASED ANALYSIS FRAMEWORK FOR SEMANTIC VIDEO EVENTS

Semantic video analysis plays an important role in the field of machine intelligence and pattern recognition. In this paper, based on the Hidden Markov Model （HMM）, a semantic recognition framework on compressed videos is proposed to analyze the video events according to six low-level features. After the detailed analysis of video events, the pattern of global motion and five features in foreground-the principal parts of videos, are employed as the observations of the Hidden Markov Model to classify events in videos. The applications of the proposed framework in some video event detections demonstrate the promising success of the proposed framework on semantic video analysis.     

Face completion with Hybrid Dilated Convolution

Image completion is a challenging task which aims to fill the missing or masked regions in images with plausibly synthesized contents. In this paper, we focus on face image inpainting tasks, aiming at reconstructing missing or damaged regions of an incomplete face image given the context information. We specially design the U-Net architecture to tackle the problem. The proposed U-Net based method combines Hybrid Dilated Convolution (HDC) and spectral normalization to fill in missing regions of any shape with sharp structures and fine-detailed textures. We perform both qualitative and quantitative evaluation on two challenging face datasets. Experimental results demonstrate that our method outperforms previous learning-based inpainting methods. The proposed method can generate realistic and semantically plausible images.