Motion segmentation and depth ordering using an occlusion detector

We present a novel method for motion segmentation and depth ordering from a video sequence in general motion. We first compute motion segmentation based on differential properties of the spatio-temporal domain, and scale-space integration. Given a motion boundary, we describe two algorithms to determine depth ordering from two- and three- frame sequences. An remarkable characteristic of our method is its ability compute depth ordering from only two frames. The segmentation and depth ordering algorithms are shown to give good results on 6 real sequences taken in general motion. We use synthetic data to show robustness to high levels of noise and illumination changes; we also include cases where no intensity edge exists at the location of the motion boundary, or when no parametric motion model can describe the data. Finally, we describe human experiments showing that people, like our algorithm, can compute depth ordering from only two frames, even when the boundary between the layers is not visible in a single frame.     

Pixel-wise video stabilization

In this paper, we present a novel video stabilization method with a pixel-wise motion model. In order to avoid distortion introduced by traditional feature points based motion models, we focus on constructing a more accurate model to capture the motion in videos. By taking advantage of dense optical flow, we can obtain the dense motion field between adjacent frames and set up a pixel-wise motion model which is accurate enough. Our method first estimates dense motion field between adjacent frames. A PatchMatch based dense motion field estimation algorithm is proposed. This algorithm is specially designed for similar video frames rather than arbitrary images to reach higher speed and better performance. Then, a simple and fast smoothing algorithm is performed to make the jittered motion stabilized. After that, we warp input frames using a weighted average algorithm to construct the output frames. Some pixels in output frames may be still empty after the warping step, so in the last step, these empty pixels are filled using a patch based image completion algorithm. We test our method on many challenging videos and demonstrate the accuracy of our model and the effectiveness of our method.     

Multi-body Factorization with Uncertainty: Revisiting Motion Consistency

Dynamic analysis of video sequences often relies on the segmentation of the sequence into regions of consistent motions. Approaching this problem requires a definition of which motions are regarded as consistent. Common approaches to motion segmentation usually group together points or image regions that have the same motion between successive frames (where the same motion can be 2D, 3D, or non-rigid). In this paper we define a new type of motion consistency, which is based on temporal consistency of behaviors across multiple frames in the video sequence. Our definition of consistent “temporal behavior” is expressed in terms of multi-frame linear subspace constraints. This definition applies to 2D, 3D, and some non-rigid motions without requiring prior model selection. We further show that our definition of motion consistency extends to data with directional uncertainty, thus leading to a dense segmentation of the entire image. Such segmentation is obtained by applying the new motion consistency constraints directly to covariance-weighted image brightness measurements. This is done without requiring prior correspondence estimation nor feature tracking.     

使用PSNR周期特性检测视频帧率上转伪造

帧率上转（FRUC）是最常用的一种视频编辑技术,它在原始视频帧间周期性地插入新的帧,以便增加视频的帧率,这种技术经常用于两段不同帧率的视频拼接伪造中。为了减少视觉痕迹,高级的FRUC方法通常采用运动补偿的插值方式,这也带来了针对这种插值伪造检测的挑战。在本文,我们提出一种新的简单但有效的方法,可正确检测出这种伪造,并能估计出视频的原始帧率。该方法利用了FRUC算法生成的插值帧与相邻原始帧构成的视频序列再次插值重建得到的帧对在PSNR上的周期性差异。测试序列的实验结果表明本文方法检测准确率高,其中对有损压缩视频序列的测试结果进一步证实了该方法的实际使用价值。     

A motion location based video watermarking scheme using ICA to extract dynamic frames

In this paper, we propose a novel video watermarking scheme based on motion location. In the proposed scheme, independent component analysis is used to extract a dynamic frame from two successive frames of original video, and the motion is located by using the variance of 8 × 8 block in the extracted dynamic frame. Then according to the located motion, we choose a corresponding region in the former frame of the two successive frames, where watermark is embedded by using the quantization index modulation algorithm. The procedure above is repeated until each frame of the video (excluding the last one) is watermarked. The simulations show that the proposed scheme has a good performance to resist Gaussian noising, MPEG2 compression, frame dropping, frame cropping, etc. This work was originally presented in the Fifth International Symposium on Neural Networks.     

Video object tracking based on improved gradient vector flow snake and intra-frame centroids tracking method

Accurately tracking the video object in video sequence is a crucial stage for video object processing which has wide applications in different fields. In this paper, a novel video object tracking algorithm based on the improved gradient vector flow (GVF) snake model and intra-frame centroids tracking algorithm is proposed. Unlike traditional gradient vector flow snake, the improved gradient vector flow snake adopts anisotropic diffusion and a four directions edge operator to solve the blurry boundary and edge shifting problem. Then the improved gradient vector flow snake is employed to extract the object contour in each frame of the video sequence. To set the initial contour of the gradient vector flow snake automatically, we design an intra-frame centroids tracking algorithm. Splitting the original video sequence into segments, for each segment, the initial contours of first two frames are set by change detection based on t-distribution significance test. Then, utilizing the redundancy between the consecutive frames, the subsequent frames’ initial contours are obtained by intra-frame motion vectors. Experimental results with several test video sequences indicate the validity and accuracy of the video object tracking.     

基于主元分析法的行为识别

通过研究,建立了一个基于主元分析的识别办体行为的系统,其方法是通过在Ｈ、Ｓ、Ｉ颜色空间对皮肤颜色建立高期模型,结合运动限制和区域连续性,系统地分割并跟踪人脸和双手,然后,在ＰＣＡ框架下,表示脸和手的运动参数曲线,并和范例进行匹配,这种通过对行为在时空域变化的建模方法,能在行为主体和成象条件有变化的情况下识别行为,以太极拳式谡列,来验证方法和系统的效果,实验结果证明了此方法误识率低,有一定的鲁棒性,     

基于形态学的高斯模型和八邻域帧差法混合运动目标检测算法

针对视频中运动目标的提取问题,提出一种基于形态学的高斯模型和八邻域帧差法相融合的提取算法。该算法首先将视频中某些帧转化为灰度图,建立以混合高斯分布为基础的统计模型,并结合八邻域帧差法提取出运动目标的大致轮廓,然后利用自适应更新的高斯模型算法进行精确的减除,最后再进行形态学处理,从而使检测出的运动目标更加清晰完整。实验结果表明,该算法对含有低速运动物体、阴影较多的视频提取效果较好,具有很好的鲁棒性。     

Background subtraction using finite mixtures of asymmetric Gaussian distributions and shadow detection

Foreground segmentation of moving regions in image sequences is a fundamental step in many vision systems including automated video surveillance, human-machine interface, and optical motion capture. Many models have been introduced to deal with the problems of modeling the background and detecting the moving objects in the scene. One of the successful solutions to these problems is the use of the well-known adaptive Gaussian mixture model. However, this method suffers from some drawbacks. Modeling the background using the Gaussian mixture implies the assumption that the background and foreground distributions are Gaussians which is not always the case for most environments. In addition, it is unable to distinguish between moving shadows and moving objects. In this paper, we try to overcome these problem using a mixture of asymmetric Gaussians to enhance the robustness and flexibility of mixture modeling, and a shadow detection scheme to remove unwanted shadows from the scene. Furthermore, we apply this method to real image sequences of both indoor and outdoor scenes. The results of comparing our method to different state of the art background subtraction methods show the efficiency of our model for real-time segmentation.     

Continuous frame motion sensitive self-supervised collaborative network for video representation learning

Motion, as a feature of video that changes in temporal sequences, is crucial to visual understanding. The powerful video representation and extraction models are typically able to focus attention on motion features in challenging dynamic environments to complete more complex video understanding tasks. However, previous approaches discriminate mainly based on similar features in the spatial or temporal domain, ignoring the interdependence of consecutive video frames. In this paper, we propose the motion sensitive self-supervised collaborative network, a video representation learning framework that exploits a pretext task to assist feature comparison and strengthen the spatiotemporal discrimination power of the model. Specifically, we first propose the motion-aware module, which extracts consecutive motion features from the spatial regions by frame difference. The global–local contrastive module is then introduced, with context and enhanced video snippets being defined as appropriate positive samples for a broader feature similarity comparison. Finally, we introduce the snippet operation prediction module, which further assists contrastive learning to obtain more reliable global semantics by sensing changes in continuous frame features. Experimental results demonstrate that our work can effectively extract robust motion features and achieve competitive performance compared with other state-of-the-art self-supervised methods on downstream action recognition and video retrieval tasks.     

基于运动区域定位的视频水印算法

提出一种基于运动区域定位的视频水印算法.算法采用独立分量分析(ICA)算法,从原始视频的相邻两帧中提取包含这两帧相对运动信息的运动分量帧.根据提取的运动分量帧,定位原始视频帧中相对运动最剧烈的区域,此区域对应至原始视频相邻两帧中的前帧,即为嵌入水印的运动区域.在嵌入水印时,采用基于Watson视觉模型的量化索引调制(QIM)算法,以保证算法的鲁棒性.实验结果表明,本算法在保持视频良好视觉质量的同时,对高斯白噪声、MPEG-2压缩、帧删除及帧剪裁具有较好的鲁棒性.     

混合高斯模型和帧间差分相融合的自适应背景模型

提出了运动目标检测中背景动态建模的一种方法。该方法是在Stauffer等人提出的自适应混合高斯背景模型基础上,为每个像素构建混合高斯背景模型,通过融入帧间差分把每帧中的图像区分为背景区域、背景显露区域和运动物体区域。相对于背景区域,背景显露区中的像素点将以大的更新率更新背景模型,使得长时间停滞物体由背景变成运动前景时,被遮挡的背景显露区被快速恢复。与Stauffer等人提出的方法不同的是,物体运动区不再构建新的高斯分布加入到混合高斯分布模型中,减弱了慢速运动物体对背景的影响。实验结果表明,在有诸多不确定性因素的序列视频中构建的背景有较好的自适应性,能迅速响应实际场景的变化。     

Video frame interpolation via optical flow estimation with image inpainting

As we all know, video frame rate determines the quality of the video. The higher the frame rate, the smoother the movements in the picture, the clearer the information expressed, and the better the viewing experience for people. Video interpolation aims to increase the video frame rate by generating a new frame image using the relevant information between two consecutive frames, which is essential in the field of computer vision. The traditional motion compensation interpolation method will cause holes and overlaps in the reconstructed frame, and is easily affected by the quality of optical flow. Therefore, this paper proposes a video frame interpolation method via optical flow estimation with image inpainting. First, the optical flow between the input frames is estimated via combined local and global-total variation (CLG-TV) optical flow estimation model. Then, the intermediate frames are synthesized under the guidance of the optical flow. Finally, the nonlocal self-similarity between the video frames is used to solve the optimization problem, to fix the pixel loss area in the interpolated frame. Quantitative and qualitative experimental results show that this method can effectively improve the quality of optical flow estimation, generate realistic and smooth video frames, and effectively increase the video frame rate.     

融合非局部神经网络的行为检测模型

针对在视频行为检测中卷积神经网络(CNN)对时域信息理解能力不足的问题,提出了一种融合非局部神经网络的行为检测模型.模型采用一种双分支的CNN结构,分别提取视频的空间特征和运动特征.将视频单帧和视频连续帧序列作为网络输入,空间网络对视频当前帧进行2D CNN特征提取,时空网络采用融合非局部模块的3D CNN来捕获视频帧...     

自适应混合高斯建模的高效运动目标检测

目的 如何使快速性与完整性达到平衡是运动目标检测的关键问题。现有的满足快速性的算法容易受到光照的影响，对动态环境的适应能力较弱，获取的目标信息不完整，导致空洞问题的产生。而具有较高完整性的算法复杂度高，运算速度慢，实时性差。为此，本文提出基于自适应混合高斯建模的3帧差分算法。方法 利用3帧差分运算简单、可扩展性强、抗干扰能力好的特性，对视频图像进行目标轮廓的提取。针对3帧差分运算导致目标内部信息提取不完整的问题，采用学习率自适应调整的混合高斯背景差分，在模型创建之初，通过较快的模型更新速率，增加背景模型的迭代次数，消除物体运动造成的"鬼影"。在背景模型中的干扰信息消除之后，以目标像素及相邻8像素在当前帧与背景模型中的差异度为依据调整学习率，实现背景模型的自适应修正，增加目标图像的完整性；同时，通过删除冗余的高斯分布，降低算法复杂度。为进一步确保目标边缘的完整及连续，采用边缘对比差分算法，使参与运算的帧数依据目标的运动速度自适应选取，以降低背景点的误判率，使边缘信息尽可能地连续、完整。结果 本文算法获取的目标信息完整，且边缘平滑。在提升检测率的同时保证较高的准确率，达到了95.23%，所获目标的完整度提高了28.95%；与传统混合高斯算法相比，时间消耗降低了29.18%，基本达到实时性要求。与基于混合高斯建模的背景差分法（BD-GMM）和基于边缘对比的3帧差分法（TFD-EC）相比，本文算法明显占优。结论 实验结果表明，本文算法可以有效抑制动态环境的干扰，降低算法复杂度，既保证实时性，又具有较好的完整性，可广泛应用于智能视频监控、军事应用、工业检测、航空航天等领域。     

一种内容完整的视频稳定算法

设计了一种基于可靠特征集合匹配的内容完整的视频稳定算法。为了避免运动前景上的特征点参与运动估计,由经典的KLT(Kanade-Lucas-Tomasi)算法提取特征点,而后基于特征有效性判定规则对特征点集合进行有效性验证以提高特征点的可靠性。利用通过验证的特征点对全局运动进行估计,得到精确的运动参数并据此对视频图像进行运动补偿。对于运动补偿造成的无定义区,首先计算当前帧的定义区与相邻帧的光流,以此为向导腐蚀无定义区;利用拼接的方法,填充仍为无定义区的像素。实验结果表明该算法对于前景物体运动具有较好的鲁棒性并能够生成内容完整的稳定视频序列。     

A new multi-level statistical model for variable bit rate MPEG sources over ATM networks and its performance study

In this paper, we propose a new traffic model for MPEG-coded video sequences. The proposed modeling scheme uses scene-based traffic characteristics and considers the correlations between frames of consecutive group of pictures (GOPs). Using a simple scene detection algorithm, scene changes are modeled by a state transition matrix and the number of GOPs of a scene state is modeled by a geometric distribution. Frames of a scene are modeled by the mean I, P, and B frame sizes of each state. For more accurate traffic modeling, the residual bits that represent the difference between the original frame size and the mean frame size of each frame type are compensated by autoregressive processes. The modeling results show that our scene-based model can capture the statistical traffic characteristics of the original video sequences well and estimate the queueing performance with good approximation quality.     

Video Summarization Using R-Sequences

In this paper, we propose a new method of temporal summarization of digital video. First, we address the problem of extracting a fixed number of representative frames to summarize a given digital video. To solve it, we have devised an algorithm called content-based adaptive clustering (CBAC). In our algorithm, shot boundary detection is not needed. Video frames are treated as points in the multi-dimensional feature space corresponding to a low-level feature such as color, motion, shape and texture. The changes of their distances are compared globally for extraction of representative frames. Second, we address how to use the representative frames to comprise representative sequences (R - Sequence) which can be used for temporal summarization of video. A video player based on our devised algorithm is developed which has functions of content-based browsing and content-based video summary. Experiments are also shown in the paper.     

Visual tracking using the Earth Mover's Distance between Gaussian mixtures and Kalman filtering

In this paper, we demonstrate how the differential Earth Mover's Distance (EMD) may be used for visual tracking in synergy with Gaussian mixtures models (GMM). According to our model, motion between adjacent frames results in variations of the mixing proportions of the Gaussian components representing the object to be tracked. These variations are computed in closed form by minimizing the differential EMD between Gaussian mixtures, yielding a very fast algorithm with high accuracy, without recurring to the EM algorithm in each frame. Moreover, we also propose a framework to handle occlusions, where the prediction for the object's location is forwarded to an adaptive Kalman filter whose parameters are estimated on line by the motion model already observed. Experimental results show significant improvement in tracking performance in the presence of occlusion.