Nonlinear color space and spatiotemporal MRF for hierarchical segmentation of face features in video

This paper deals with the low-level joint processing of color and motion for robust face analysis within a feature-based approach. To gain robustness and contrast under unsupervised viewing conditions, a nonlinear color transform relevant for hue segmentation is derived from a logarithmic model. A hierarchical segmentation scheme is based on Markov random field modeling, that combines hue and motion detection within a spatiotemporal neighborhood. Relevant face regions are segmented without parameter tuning. The accuracy of the label fields enables not only face detection and tracking but also geometrical measurements on facial feature edges, such as lips or eyes. Results are shown both on typical test sequences and on various sequences acquired from micro- or mobile cameras. The efficiency of the method makes it suitable for real-time applications aiming at audiovisual communication in unsupervised environments.     

Multi-view video based multiple objects segmentation using graph cut and spatiotemporal projections

In this paper, we present an automatic algorithm to segment multiple objects from multi-view video. The Initial Interested Objects (IIOs) are automatically extracted in the key view of the initial frame based on the saliency model. Multiple objects segmentation is decomposed into several sub-segmentation problems, and solved by minimizing the energy function using binary label graph cut. In the proposed novel energy function, the color and depth cues are integrated with the data term, which is then modified with background penalty with occlusion reasoning. In the smoothness term, foreground contrast enhancement is developed to strengthen the moving objects boundary, and at the same time attenuates the background contrast. To segment the multi-view video, the coarse predictions of the other views and the successive frame are projected by pixel-based disparity and motion compensation, respectively, which exploits the inherent spatiotemporal consistency. Uncertain band along the object boundary is shaped based on activity measure and refined with graph cut, resulting in a more accurate Interested Objects (IOs) layer across all views of the frames. The experiments are implemented on a couple of multi-view videos with real and complex scenes. Excellent subjective results have shown the robustness and efficiency of the proposed algorithm.     

Finding spatio-temporal salient paths for video objects discovery

Many videos capture and follow salient objects in a scene. Detecting such salient objects is thus of great interests to video analytics and search. However, the discovery of salient objects in an unsupervised way is a challenging problem as there is no prior knowledge of the salient objects provided. Different from existing salient object detection methods, we propose to detect and track salient object by finding a spatio-temporal path which has the largest accumulated saliency density in the video. Inspired by the observation that salient video objects usually appear in consecutive frames, we leverage the motion coherence of videos into the path discovery and make the salient object detection more robust. Without any prior knowledge of the salient objects, our method can detect salient objects of various shapes and sizes, and is able to handle noisy saliency maps and moving cameras. Experimental results on two public datasets validate the effectiveness of the proposed method in both qualitative and quantitative terms. Comparisons with the state-of-the-art methods further demonstrate the superiority of our method on salient object detection in videos.     

Spatio-temporal segmentation for video surveillance

The automatic extraction of moving objects and construction of site models are key problems in video surveillance systems. A novel approach to segmenting moving objects from static scenes as well as acquiring background images automatically is presented. Experimental results on real video sequences demonstrate the robustness and accuracy of the algorithm     

Modified intelligent scissors and video decomposing for video object segmentation

A semiautomatic video object segmentation is proposed. The initial object contour is obtained by modified intelligent scissors. Video decomposing is performed to avoid errors accumulating during object tracking. Snake-based bidirectional tracking is utilised to interpolate the VOPs of successive frames. Experimental results show the effectiveness of the method.     

Fast pixel-matching for video object segmentation

Video object segmentation, aiming to segment the foreground objects given the annotation of the first frame, has been attracting increasing attentions. Many state-of-the-art approaches have achieved great performance by relying on online model updating or mask-propagation techniques. However, most online models require high computational cost due to model fine-tuning during inference. Most mask-propagation based models are faster but with relatively low performance due to failure to adapt to object appearance variation. In this paper, we are aiming to design a new model to make a good balance between speed and performance. We propose a model, called NPMCA-net, which directly localizes foreground objects based on mask-propagation and non-local technique by matching pixels in reference and target frames. Since we bring in information of both first and previous frames, our network is robust to large object appearance variation, and can better adapt to occlusions. Extensive experiments show that our approach can achieve a new state-of-the-art performance with a fast speed at the same time (86.5% IoU on DAVIS-2016 and 72.2% IoU on DAVIS-2017, with speed of 0.11s per frame) under the same level comparison. Source code is available at https://github.com/siyueyu/NPMCA-net.     

Spatiotemporal segmentation for compact video representation

In this paper, a novel hierarchical object-oriented video segmentation and representation algorithm is proposed. The local variance contrast and the frame difference contrast are jointly exploited for structural spatiotemporal video segmentation because these two visual features can indicate the spatial homogeneity of the grey levels and the temporal coherence of the motion fields efficiently, where the two-dimensional (2D) spatiotemporal entropic technique is further selected for generating the 2D thresholding vectors adaptively according to the variations of the video components. After the region growing and edge simplification procedures, the accurate boundaries among the different video components are further exploited by an intra-block edge extraction procedure. Moreover, the relationships of the video components among frames are exploited by a temporal tracking procedure. This proposed object-oriented spatiotemporal video segmentation algorithm may be useful for MPEG-4 system generating the video object plane (VOP) automatically.     

CNN spatiotemporal features and fusion for surveillance video forgery detection

Surveillance cameras are widely used to provide protection and security; also their videos are used as strong evidences in the courts. Through the availability of video editing tools, it has become easy to distort these evidences. Sometimes, to hide the traces of forgery, some post-processing operations are performed after editing. Hence, the authenticity and integrity of surveillance videos have become urgent to scientifically validate. In this paper, we propose inter-frame forgeries (frame deletion, frame insertion, and frame duplication) detection system using 2D convolution neural network (2D-CNN) of spatiotemporal information and fusion for deep automatically feature extraction; Gaussian RBF multi-class support vector machine (RBF-MSVM) is used for classification process. The experimental results show that the efficiency of the proposed system for detecting all inter-frame forgeries, even when the forged videos have undergone additional post-processing operations such as Gaussian noise, Gaussian blurring, brightness modifications and compression.     

Automatic video segmentation and tracking for content-based applications

Advanced multimedia applications have to provide content-related functionalities such as search and retrieval of meaningful objects, detection and analysis of events, and understanding of scenes, which allow the user to access and manipulate the multimedia content with greater flexibility. This greatly depends on automatic techniques for extracting such objects from multimedia data. In this article we intend to provide a tutorial on the state-of-the-art in video segmentation and tracking technology with particular attention paid to the recent developments in attention-based object extraction. Performance results are included to highlight this emerging technology     

Performance measures for video object segmentation and tracking

We propose measures to evaluate quantitatively the performance of video object segmentation and tracking methods without ground-truth (GT) segmentation maps. The proposed measures are based on spatial differences of color and motion along the boundary of the estimated video object plane and temporal differences between the color histogram of the current object plane and its predecessors. They can be used to localize (spatially and/or temporally) regions where segmentation results are good or bad; and/or they can be combined to yield a single numerical measure to indicate the goodness of the boundary segmentation and tracking results over a sequence. The validity of the proposed performance measures without GT have been demonstrated by canonical correlation analysis with another set of measures with GT on a set of sequences (where GT information is available). Experimental results are presented to evaluate the segmentation maps obtained from various sequences using different segmentation approaches.     

一种压缩视频流的视频分段和关键帧提取方法

基于压缩域视频片段检测可以省略解压步骤,直接从原始视频数据流提取特征,可以加快检测率.本文首先分析了视频数据的特性,视频的分割和关键帧的选取,阐述现有的典型方法,通过分析视频检索技术的关键技术,采用基于视频分割和关键帧的压缩视频流提取方法,提出了一种基于DC系数和运动矢量从MPEG压缩视频中提取关键帧的方法.实验表明,所提出的方法可以减少计算负担,并可以更好地表示视频内容.     

Dynamic programming framework for automatic video object segmentation and vision-assisted video pre-processing

A generic definition of video objects, which is a group of pixels with temporal motion coherence, is considered. The generic video object (GVO) is the superset of the conventional video objects considered in the object segmentation literature. Because of its motion coherence, the GVO can be easily recognised by the human visual system. However, due to its arbitrary spatial distribution, the GVO cannot be easily detected by the existing algorithms which often assume the spatial homogeneousness of the video objects. The concept of extended optical flow is introduced and a dynamic programming framework for the GVO detection and segmentation is developed, whose solution is given by the Viterbi algorithm. Using this dynamic programming formulation, the proposed object detection algorithm is able to discover the motion path of the GVO automatically and refine its spatial region of support progressively. In addition to object segmentation, the proposed algorithm can also be applied to video pre-processing, removing the so-called 'video mask' noise in digital videos. Experimental results show that this type of vision-assisted video pre-processing significantly improves the compression efficiency.     

Block-based motion field segmentation for video coding

In the past few years, motion compensation has been widely used in the coding of image sequences. Most of motion estimation and compensation schemes belong to block-based framework. The framework simplifies the complexity of motion estimation, but gives over constraints to the motion field, which results in worse accuracy on the boundary of moving objects. This paper presents a novel technique for raising motion field accuracy. It uses several pre-defined pattern types to segment the motion fields of the previous frame of a sequence. The segmentation is based on the MAP framework that uses iterative method to obtain the solution. In addition, we develop a predictive scheme to predict the location of motion field discontinuities in the current frame, which further reduces the side information for the representation of segmentation.     

红外视频帧运动模糊复原技术研究

当前红外空域监测探测系统常用视频的形式存储和传输图像信号,但是视频图像在形成、传输和记录过程中,易受运动模糊和噪声的污染,为了使该系统适用于当前空域形势,提出基于该系统的视频运动模糊复原算法。首先构建基于视频流运动模糊复原模型,综合序列图像各帧之间的互信息,估计有效的点扩散函数,然后描述运动模糊复原流程,提出相应算法,构建各功能模块。操作中视频以降频采样的方式减少计算复杂度,提高图像质量,获取较高复原效果。最后,通过引入主、客观两套评价体系对使用的算法以及其他经典算法作对照,评估复原结果。实验结果表明:复原视频各帧的峰值信噪比达到37,均方误差在9以下,均优于对照算法。基本满足监测系统发现目标,监测空域的要求。     

Spatiotemporal video segmentation based on graphical models.

This paper proposes a probabilistic framework for spatiotemporal segmentation of video sequences. Motion information, boundary information from intensity segmentation, and spatial connectivity of segmentation are unified in the video segmentation process by means of graphical models. A Bayesian network is presented to model interactions among the motion vector field, the intensity segmentation field, and the video segmentation field. The notion of the Markov random field is used to encourage the formation of continuous regions. Given consecutive frames, the conditional joint probability density of the three fields is maximized in an iterative way. To effectively utilize boundary information from the intensity segmentation, distance transformation is employed in local objective functions. Experimental results show that the method is robust and generates spatiotemporally coherent segmentation results. Moreover, the proposed video segmentation approach can be viewed as the compromise of previous motion based approaches and region merging approaches.     

A spatiotemporal super-resolution algorithm for a hybrid stereo video system

This paper presents a spatiotemporal super-resolution method to enhance both the spatial resolution and the frame rate in a hybrid stereo video system. In this system, a scene is captured by two cameras to form two videos, including a low spatial resolution with high-frame-rate video and a high spatial resolution with low-frame-rate video. For the low-spatial-resolution video, the low-resolution frames are spatially super-resolved by the high-resolution video via the stereo matching, the bilateral overlapped block motion estimation, and the adaptive overlapped block motion compensation algorithms, while for the low-frame-rate video, those missed frames are interpolated using the high-resolution frames obtained by fusing the disparity compensation and the motion compensation frame rate up-conversion. Experimental results demonstrate that the proposed mixed spatiotemporal super-resolution method has a more significant contribution to both the subjective and objective qualities than the pure spatial super-resolution or the frame rate up-conversion.     

Texture as pixel feature for video object segmentation

As texture represents one of the key perceptual attributes of any object, integrating textural information into existing video object segmentation frameworks affords the potential to achieve semantically improved performance. While object segmentation is fundamentally pixel-based classification, texture is normally defined for the entire image, which raises the question of how best to directly specify and characterise texture as a pixel feature. Introduced is a generic strategy for representing textural information so it can be seamlessly incorporated as a pixel feature into any video object segmentation paradigm. Both numerical and perceptual results upon various test sequences reveal considerable improvement in the object segmentation performance when textural information is embedded.     

Motion field modeling for video sequences

We propose a model for the interframe correspondences existing between pixels of an image sequence. These correspondences form the elements of a field called the motion field. In our model, spatial neighborhoods of motion elements are related based on a generalization of autoregressive (AR) modeling of the time-series. We also propose a joint spatio-temporal model by including spatial neighborhoods of pixel intensities in the motion model. A fundamental difference of our approach with most previous approaches to modeling motion is in basing our model on concepts from statistical signal processing. The developments in this paper give rise to the promise of extending well-understood tools of signal processing (e.g., filtering) to the analysis and processing of motion fields. Simulation results presented show the performance of our models in interframe prediction; specifically, on average the motion model performs 29% better in terms of the mean squared error energy over a commonly used pel-recursive approach. The spatio-temporal model improves the prediction efficiencies by 8% over the motion model. Our model can also be used to obtain estimates of the optical flow field as the simulations demonstrate.     

一种新的视频兴奋内容建模和视频摘要提取方法

本文对基于运动矢量的运动量的建模方法进行了修正,同时引入语速这种音频特征,在综合了镜头变换率和声音能量两个特征基础之上,提出了一种新的兴奋建模和视频摘要统一框架.在该框架之内,首先对视频的兴奋内容进行建模得到兴奋时间曲线,之后,依据曲线中的极大值和极小值提取关键帧和精彩片断两种形式的视频摘要.实验证明本文建模方法是有效的,提取的视频摘要能够有效表示视频的内容,且具有良好的面向用户性和自适应性.提取的关键帧的有效性达到78%,足球片断的精彩有效性和排序的有效性分别达到81%和82%,篮球片断的精彩有效性和排序的有效性分别达到75%和76%.     

An interactive authoring system for video object segmentation and annotation

An interactive authoring system is proposed for semi-automatic video object (VO) segmentation and annotation. This system features a new contour interpolation algorithm, which enables the user to define the contour of a VO on multiple frames while the computer interpolates the missing contours of this object on every frame automatically. Typical active contour (snake) model is adapted and the contour interpolation problem is decomposed into a two-directional contour tracking problem and a merging problem. In addition, new user interaction models are designed for the user to interact with the computer. Experiments indicate that this system offers a good balance between algorithm complexity and user interaction efficiency.