Real-time human segmentation from RGB-D video sequence based on adaptive geodesic distance computation

Multimedia Tools and Applications - In this paper, we propose a method for extracting humans in the foreground of video frames using color and depth information. To ensure real-time performance and...     

Boundary-aware texture region segmentation from manga

Computational Visual Media - Due to the lack of color in manga (Japanese comics), black-and-white textures are often used to enrich visual experience. With the rising need to digitize manga,...     

Fuzzy color histogram-based video segmentation

We present a fuzzy color histogram-based shot-boundary detection algorithm specialized for content-based copy detection applications. The proposed method aims to detect both cuts and gradual transitions (fade, dissolve) effectively in videos where heavy transformations (such as cam-cording, insertions of patterns, strong re-encoding) occur. Along with the color histogram generated with the fuzzy linking method on L*a*b* color space, the system extracts a mask for still regions and the window of picture-in-picture transformation for each detected shot, which will be useful in a content-based copy detection system. Experimental results show that our method effectively detects shot boundaries and reduces false alarms as compared to the state-of-the-art shot-boundary detection algorithms.     

Genetic algorithms for video segmentation

The current paper presents a new genetic algorithm (GA)-based method for video segmentation. The proposed method is specifically designed to enhance the computational efficiency and quality of the segmentation results compared to standard GAs. The segmentation is performed by chromosomes that independently evolve using distributed genetic algorithms (DGAs). However, unlike conventional DGAs, the chromosomes are initiated using the segmentation results of the previous frame, instead of random values. Thereafter, only unstable chromosomes corresponding to moving object parts are evolved by crossover and mutation. As such, these mechanisms allow for effective solution space exploration and exploitation, thereby improving the performance of the proposed method in terms of speed and segmentation quality. These advantages were confirmed based on experiments where the proposed method was successfully applied to both synthetic and natural video sequences.     

Graph-based multilevel temporal video segmentation

This paper presents a graph-based multilevel temporal video segmentation method. In each level of the segmentation, a weighted undirected graph structure is implemented. The graph is partitioned into clusters which represent the segments of a video. Three low-level features are used in the calculation of temporal segments’ similarities: visual content, motion content and shot duration. Our strength factor approach contributes to the results by improving the efficiency of the proposed method. Experiments show that the proposed video scene detection method gives promising results in order to organize videos without human intervention.     

视频会议中视频分割合成的研究

视频会议系统正在吸引越来越多在地域上分散的人们的注意。随着多媒体信息处理和通信技术的飞速发展,虚拟视频会议系统越来越成为可能。和传统的基于二维图像表述的视频会议系统比较,三维虚拟会议系统能为人们提供更好的以人为中心的交流。在这两种情况下,高效精确的视频分割和视图合成都是关键问题。就以下3方面的主要技术进行了研究：背景分离、视差估计以及视图合成。     

Feature fusion for basic behavior unit segmentation from video sequences

It has become increasingly popular to study animal behaviors with the assistance of video recordings. An automated video processing and behavior analysis system is desired to replace the traditional manual annotation. We propose a framework for automatic video based behavior analysis systems, which consists of four major modules: behavior modeling, feature extraction from video sequences, basic behavior unit (BBU) discovery and complex behavior recognition. BBU discovery is performed based on features extracted from video sequences, hence the fusion of multiple dimensional features is very important. In this paper, we explore the application of feature fusion techniques to BBU discovery with one and multiple cameras. We applied the vector fusion (SBP) method, a multi-variate vector visualization technique, in fusing the features obtained from a single camera. This technique reduces the multiple dimensional data into two dimensional (SBP) space, and the spatial and temporal analysis in SBP space can help discover the underlying data groups. Then we present a simple feature fusion technique for BBU discovery from multiple cameras with the affinity graph method. Finally, we present encouraging results on a physical system and a synthetic mouse-in-a-cage scenario from one, two, and three cameras. The feature fusion methods in this paper are simple yet effective.     

Full-duplex strategy for video object segmentation

Computational Visual Media - Previous video object segmentation approaches mainly focus on simplex solutions linking appearance and motion, limiting effective feature collaboration between these...     

多帧时空注意力引导的半监督视频分割

目的 传统的半监督视频分割多是基于光流的方法建模关键帧与当前帧之间的特征关联。而光流法在使用过程中容易因遮挡、特殊纹理等情况产生错误,从而导致多帧融合存在问题。为了更好地融合多帧特征,本文提取第1帧的外观特征信息与邻近关键帧的位置信息,通过Transformer和改进的PAN（path aggregation network）模块进行特征融合,从而基于多帧时空注意力学习并融合多帧的特征。方法 多帧时空注意力引导的半监督视频分割方法由视频预处理（即外观特征提取网络和当前帧特征提取网络）以及基于Transformer和改进的PAN模块的特征融合两部分构成。具体包括以下步骤：构建一个外观信息特征提取网络,用于提取第1帧图像的外观信息;构建一个当前帧特征提取网络,通过Transformer模块对当前帧与第1帧的特征进行融合,使用第1帧的外观信息指导当前帧特征信息的提取;借助邻近数帧掩码图与当前帧特征图进行局部特征匹配,决策出与当前帧位置信息相关性较大的数帧作为邻近关键帧,用来指导当前帧位置信息的提取;借助改进的PAN特征聚合模块,将深层语义信息与浅层语义信息进行融合。结果 本文算法在DAVIS（densely annotated video segmentation）-2016数据集上的J和F得分为81.5%和80.9%,在DAVIS-2017数据集上为78.4%和77.9%,均优于对比方法。本文算法的运行速度为22帧/s,对比实验中排名第2,比PLM（pixel-level matching）算法低1.6%。在YouTube-VOS（video object segmentation）数据集上也取得了有竞争力的结果,J和F的平均值达到了71.2%,领先于对比方法。结论 多帧时空注意力引导的半监督视频分割算法在对目标物体进行分割的同时,能有效融合全局与局部信息,减少细节信息丢失,在保持较高效率的同时能有效提高半监督视频分割的准确率。     

Production model based digital video segmentation

Effective and efficient tools for segmenting and content-based indexing of digital video are essential to allow easy access to video-based information. Most existing segmentation techniques do not use explicit models of video. The approach proposed here is inspired and influenced by well established video production processes. Computational models of these processes are developed. The video models are used to classify the transition effects used in video and to design automatic edit effect detection algorithms. Video segmentation has been formulated as a production model based classification problem. The video models are also used to define segmentation error measures. Experimental results from applying the proposed technique to commercial cable television programming are presented.     

Discovery and segmentation of activities in video

Hidden Markov models (HMMs) have become the workhorses of the monitoring and event recognition literature because they bring to time-series analysis the utility of density estimation and the convenience of dynamic time warping. Once trained, the internals of these models are considered opaque; there is no effort to interpret the hidden states. We show that by minimizing the entropy of the joint distribution, an HMM's internal state machine can be made to organize observed activity into meaningful states. This has uses in video monitoring and annotation, low bit-rate coding of scene activity, and detection of anomalous behavior. We demonstrate with models of office activity and outdoor traffic, showing how the framework learns principal modes of activity and patterns of activity change. We then show how this framework can be adapted to infer hidden state from extremely ambiguous images, in particular, inferring 3D body orientation and pose from sequences of low-resolution silhouettes     

Spatial-temporal joint probability images for video segmentation

Effective annotation and content-based search for videos in a digital library require a preprocessing step of detecting, locating and classifying scene transitions, i.e., temporal video segmentation. This paper proposes a novel approach—spatial-temporal joint probability image (ST-JPI) analysis for temporal video segmentation. A joint probability image (JPI) is derived from the joint probabilities of intensity values of corresponding points in two images. The ST-JPT, which is a series of JPIs derived from consecutive video frames, presents the evolution of the intensity joint probabilities in a video. The evolution in a ST-JPI during various transitions falls into one of several well-defined linear patterns. Based on the patterns in a ST-JPI, our algorithm detects and classifies video transitions effectively.Our study shows that temporal video segmentation based on ST-JPIs is distinguished from previous methods in the following way: (1) It is effective and relatively robust not only for video cuts but also for gradual transitions; (2) It classifies transitions on the basis of predefined evolution patterns of ST-JPIs during transitions; (3) It is efficient, scalable and suitable for real-time video segmentation. Theoretical analysis and experimental results of our method are presented to illustrate its efficacy and efficiency.     

Non-interactive automatic video segmentation of moving targets

Extracting moving targets from video accurately is of great significance in the field of intelligent transport.To some extent,it is related to video segmentation or matting.In this paper,we propose a non-interactive automatic segmentation method for extracting moving targets.First,the motion knowledge in video is detected with orthogonal Gaussian-Hermite moments and the Otsu algorithm,and the knowledge is treated as foreground seeds.Second,the background seeds are generated with distance transformation based on foreground seeds.Third,the foreground and background seeds are treated as extra constraints,and then a mask is generated using graph cuts methods or closed-form solutions.Comparison showed that the closed-form solution based on soft segmentation has a better performance and that the extra constraint has a larger impact on the result than other parameters.Experiments demonstrated that the proposed method can effectively extract moving targets from video in real time.     

Color-based video segmentation using interlinked irregular pyramids

This paper presents a new segmentation technique for video sequences. It relies on building irregular pyramids based on its homogeneity over consecutive frames. Pyramids are interlinked to keep a relationship between the regions in the frames. Virtual nodes are considered to improve matching between low resolution levels of the pyramids. Its performance is good in real-world conditions because it does not depend on image constrains.     

Effective online refinement for video object segmentation

Multimedia Tools and Applications - In this paper, we propose a novel framework, which deeply explores the motion cue and the online fine-tuning strategy to tackle the task of semi-supervised video...     

基于视频分割的关键帧选取

综合考虑了基于帧分块的局部似然比特征和基于小波变换的全局的边缘信息,利用这两者进行镜头边界检测,从视频镜头重构度 (SRD) 角度出发,针对线性拟合方法,采用了最小二乘法曲线拟合优化方法,分段重构出视频曲线,求出曲线的极值点,并以极值点代表关键帧.通过实验数据比较表明,该方法与实际的情况相符并取得明显的数值效果.     

On multi-view video segmentation for object-based coding

A novel scheme for multi-view segmentation and tracking is proposed aiming to acquire perceptually consistent results for object-based coding. Firstly, a classic image segmentation technique is employed to perform initial segmentation to divide the whole image into spatially homogeneous regions. Secondly, the motion information is extracted based on frame differences and the disparity information is derived by employing a classic disparity estimation technique. Thirdly, a novel scheme is proposed to perform merging of the initial segmentation results based on both motion and disparity information to remove over-segmented regions and extract perceptually consistent semantic objects. Finally, a contour-based tracking algorithm is proposed to implement accurate and robust object tracking along both temporal and view directions. Experiments are conducted and the results demonstrate that the proposed scheme is effective and, compared with the existing technique, it can acquire more perceptually consistent results.     

Skin color-based video segmentation under time-varying illumination

A novel approach for real-time skin segmentation in video sequences is described. The approach enables reliable skin segmentation despite wide variation in illumination during tracking. An explicit second order Markov model is used to predict evolution of the skin-color (HSV) histogram over time. Histograms are dynamically updated based on feedback from the current segmentation and predictions of the Markov model. The evolution of the skin-color distribution at each frame is parameterized by translation, scaling, and rotation in color space. Consequent changes in geometric parameterization of the distribution are propagated by warping and resampling the histogram. The parameters of the discrete-time dynamic Markov model are estimated using Maximum Likelihood Estimation and also evolve over time. The accuracy of the new dynamic skin color segmentation algorithm is compared to that obtained via a static color model. Segmentation accuracy is evaluated using labeled ground-truth video sequences taken from staged experiments and popular movies. An overall increase in segmentation accuracy of up to 24 percent is observed in 17 out of 21 test sequences. In all but one case, the skin-color classification rates for our system were higher, with background classification rates comparable to those of the static segmentation.     

Saturation-aware human attention region of interest algorithm for efficient video compression

Multimedia Tools and Applications - We propose a saturation-aware human attention region-of-interest (SA-HAROI) video compression method that performs a perceptual adaptive quantization algorithm...     

GCA-Net: Gait contour automatic segmentation model for video gait recognition

Multimedia Tools and Applications - Gait recognition from videos is a very important task for surveillance video analysis. Although a number of studies have explored gait recognition models, they...