Multiview video summarization using video partitioning and clustering

Multiview video summarization plays a crucial role in abstracting essential information form multiple videos of the same location and time. In this paper, we propose a new approach for the multiview summarization. The proposed approach uses the BIRCH clustering algorithm for the first time on the initial set of frames to get rid of the static and redundant. The work presents a new approach for shot boundary detection using frame similarity measures Jaccard and Dice. The algorithm performs effectively synchronized merging of keyframes from all camera-views to obtain the final summary. Extensive experimentation conducted on various datasets suggests that the proposed approach significantly outperforms most of the existing video summarization approaches. To state a few, a 1.5% improvement on video length reduction, 24.28% improvement in compression ratio, and 6.4% improvement in quality assessment ratio is observed on the lobby dataset.     

A comprehensive survey and mathematical insights towards video summarization

Video Summarization is a technique to reduce the original raw video into a short video summary. Video summarization automates the task of acquiring key frames/segments from the video and combining them to generate a video summary. This paper provides a framework for summarization based on different criteria and also compares different literature work related to video summarization. The framework deals with formulating model for video summarization based on different criteria. Based on target audience/ viewership, number of videos, type of output intended, type of video summary and summarization factor; a model generating video summarization framework is proposed. The paper examines significant research works in the area of video summarization to present a comprehensive review against the framework. Different techniques, perspectives and modalities are considered to preserve the diversity of survey. This paper examines important mathematical formulations to provide meaningful insights for video summarization model creation.     

Online video summarization on compressed domain

Recent advances in technology have increased the availability of video data, creating a strong requirement for efficient systems to manage those materials. Making efficient use of video information requires that data to be accessed in a user-friendly way. Ideally, one would like to understand a video content, without having to watch it entirely. This has been the goal of a quickly evolving research area known as video summarization. In this paper, we present a novel approach for video summarization that works in the compressed domain and allows the progressive generation of a video summary. The proposed method relies on exploiting visual features extracted from the video stream and on using a simple and fast algorithm to summarize the video content. Experiments on a TRECVID 2007 dataset show that our approach presents high quality relative to the state-of-the-art solutions and in a computational time that makes it suitable for online usage.     

基于压缩感知和熵计算的关键帧提取算法

针对关键帧提取问题,提出了一种基于压缩感知理 论和熵计算的关键帧提取算法, 首先通过构造符合有限等距性质要求的稀疏随机投影矩阵,将高维多尺度帧图像特征变换为 低维多尺度帧图像特征, 并形成视频镜头低维多尺度特征列向量组;然后通过随机权值向量与低维多尺度特征向量的 阿达玛乘积运算生成各 帧图像的匹配特征,并根据匹配特征的相似性度量完成镜头内部的子镜头分割;最后通过交 叉熵计算在每个子镜头 中得到可能的关键帧,并由图像熵计算确定最终的关键帧。实验表明,与传统方法相比,本 文算法提取的关键帧能够更精确、更稳定描述视频镜头内容。     

多文档文摘句子优选算法研究

该文通过对文摘句的选择问题进行分析,提出了一种文摘句优选方法,相对于传统的逐个添加句子生成文摘的方法,该文提出的方法是在一定范围内逐个删除句子生成文摘。该方法分两阶段进行句子选择,第1阶段获取候选文摘句子集合,采用了直接获取算法和基于冗余信息处理的获取算法。第2阶段逐步删除句子,分别以不同特征项作为衡量句子对候选文摘句子集合的贡献,提出了文摘句优选算法。以DUC2004为实验语料,通过经句子选择后生成文摘的ROUGE得分,验证了句子选择在文摘生成过程中的必要性,与基于冗余信息处理的句子选择方法比较,验证了该文提出算法的有效性。     

On the use of hierarchical prediction structures for efficient summary generation of H.264/AVC bitstreams

Video summarization refers to an important set of abstraction techniques aimed to provide a compact representation of the video essential to effectively browse and retrieve video content from multimedia repositories. Most of these video summarization techniques, such as image storyboards, video skims and fast previews, are based on selecting some frames or segments. H.264/AVC has become a widely accepted coding standard and is expected that many of the content will be available in this format soon. This paper proposes a generic model of video summarization especially suitable for generating summaries of H.264/AVC bitstreams in a highly efficient manner, using the concept of temporal scalability via hierarchical prediction structures. Along with the model, specific examples of summarization techniques are given to prove the utility of the model.     

Retrospective analysis of time series for frame selection in surveillance video summarization

In recent days, we have witnessed a dramatical growth of videos in various real-life scenarios. In this paper, we address the problem of surveillance video summarization. We present a new method of key-frame selection for this task: By virtue of retrospective analysis of time series, temporal cuts are first detected by sequentially measuring dissimilarities on a given video with threshold-based decision making; then, with the detected cuts, the video is segmented into a number of consecutive clips containing similar video contents; key frames are last selected by performing a typical clustering procedure in each resulted clip for final video summary. We have conducted extensive experiments on the benchmarking ViSOR dataset and the publicly available IVY LAB dataset. Excellent performances outperforming state-of-the-art competitors were demonstrated on key-frame selection for surveillance video summarization, which suggests good potentials of the proposed method in real-world applications.     

Efficient visual attention based framework for extracting key frames from videos

The huge amount of video data on the internet requires efficient video browsing and retrieval strategies. One of the viable solutions is to provide summaries of the videos in the form of key frames. The video summarization using visual attention modeling has been used of late. In such schemes, the visually salient frames are extracted as key frames on the basis of theories of human attention modeling. The visual attention modeling schemes have proved to be effective in video summarization. However, the high computational costs incurred by these techniques limit their applicability in practical scenarios. In this context, this paper proposes an efficient visual attention model based key frame extraction method. The computational cost is reduced by using the temporal gradient based dynamic visual saliency detection instead of the traditional optical flow methods. Moreover, for static visual saliency, an effective method employing discrete cosine transform has been used. The static and dynamic visual attention measures are fused by using a non-linear weighted fusion method. The experimental results indicate that the proposed method is not only efficient, but also yields high quality video summaries.     

Two-stage sentence selection approach for multi-document summarization

Compared with the traditional method of adding sentences to get summary in multi-document summarization,a two-stage sentence selection approach based on deleting sentences in a candidate sentence set to generate summary is proposed,which has two stages,the acquisition of a candidate sentence set and the optimum selection of sentence.At the first stage,the candidate sentence set is obtained by redundancy-based sentence selection approach.At the second stage,optimum selection of sentences is proposed to delete sentences in the candidate sentence set according to its contribution to the whole set until getting the appointed summary length.With a test corpus,the ROUGE value of summaries gotten by the proposed approach proves its validity,compared with the traditional method of sentence selection.The influence of the token chosen in the two-stage sentence selection approach on the quality of the generated summaries is analyzed.     

Multi-layered attentional peephole convolutional LSTM for abstractive text summarization

Abstractive text summarization is a process of making a summary of a given text by paraphrasing the facts of the text while keeping the meaning intact. The manmade summary generation process is laborious and time-consuming. We present here a summary generation model that is based on multilayered attentional peephole convolutional long short-term memory (MAPCoL; LSTM) in order to extract abstractive summaries of large text in an automated manner. We added the concept of attention in a peephole convolutional LSTM to improve the overall quality of a summary by giving weights to important parts of the source text during training. We evaluated the performance with regard to semantic coherence of our MAPCoL model over a popular dataset named CNN/Daily Mail, and found that MAPCoL outperformed other traditional LSTM-based models. We found improvements in the performance of MAPCoL in different internal settings when compared to state-of-the-art models of abstractive text summarization.     

IM(S)2: Interactive movie summarization system

The need of summarization methods and systems has become more and more crucial as the audio-visual material continues its critical growth. This paper presents a novel vision and a novel system for movies summarization. A video summary is an audio-visual document displaying the essential parts of an original document. However, the definition of the term “essential” is user-dependent. The advantage of this work, unlike the others, is the involvement of users in the summarization process. By means of IM(S)², people generate on the fly customized video summaries responding to their preferences. IM(S)² is made up of an offline part and an online part. In the offline, we segment the movies into shots and we compute features describing them. In the online part users inform about their preferences by selecting interesting shots. After that, the system will analyze the selected shots to bring out the user’s preferences. Finally the system will generate a summary from the whole movie which will provide more focus on the user’s preferences. To show the efficiency of IM(S)², it was tested on the database of the European project MUSCLE made up of five movies. We invited 10 users to evaluate the usability of our system by generating for every movie of the database a semi-supervised summary and to judge at the end its quality. Obtained results are encouraging and show the merits of our approach.     

Détection de visages pour résumé de vidéos

In an image, the faces of the persons are the first information looked for. Performing face detection in a video, allows to obtain automatically « face » summaries by privileging in the summary the images where faces have been detected. Shots with a similar scene layout (same number of person, similar face position and size) are gathered within shot clusters.     

Automatic soccer video analysis and summarization

We propose a fully automatic and computationally efficient framework for analysis and summarization of soccer videos using cinematic and object-based features. The proposed framework includes some novel low-level processing algorithms, such as dominant color region detection, robust shot boundary detection, and shot classification, as well as some higher-level algorithms for goal detection, referee detection, and penalty-box detection. The system can output three types of summaries: i) all slow-motion segments in a game; ii) all goals in a game; iii) slow-motion segments classified according to object-based features. The first two types of summaries are based on cinematic features only for speedy processing, while the summaries of the last type contain higher-level semantics. The proposed framework is efficient, effective, and robust. It is efficient in the sense that there is no need to compute object-based features when cinematic features are sufficient for the detection of certain events, e.g., goals in soccer. It is effective in the sense that the framework can also employ object-based features when needed to increase accuracy (at the expense of more computation). The efficiency, effectiveness, and robustness of the proposed framework are demonstrated over a large data set, consisting of more than 13 hours of soccer video, captured in different countries and under different conditions.     

HEVC压缩域的视频摘要关键帧提取方法

为了提高关键帧提取的准确率，改善视频摘要的质量，提出了一种HEVC压缩域的视频摘要关键帧提取方法。首先，对视频序列进行编解码，在解码中统计HEVC帧内编码PU块的亮度预测模式数目。然后，特征提取是利用统计得到的模式数目构建成模式特征向量，并将其作为视频帧的纹理特征用于关键帧的提取。最后，利用融合迭代自组织数据分析算法(ISODATA）的自适应聚类算法对模式特征向量进行聚类，在聚类结果中选取每个类内中间向量对应的帧作为候选关键帧，并通过相似度对候选关键帧进行再次筛选，剔除冗余帧，得到最终的关键帧。实验结果表明，在Open Video Project数据集上进行的大量实验验证，该方法提取关键帧的精度为79.9%、召回率达到93.6%、F-score为86.2%，有效地改善了视频摘要的质量。      

A time-aware query-focused summarization of an evolving microblogging stream via sentence extraction

With the number of social media users ramping up, microblogs are generated and shared at record levels. The high momentum and large volumes of short texts bring redundancies and noises, in which the users and analysts often find it problematic to elicit useful information of interest. In this paper, we study a query-focused summarization as a solution to address this issue and propose a novel summarization framework to generate personalized online summaries and historical summaries of arbitrary time durations. Our framework can deal with dynamic, perpetual, and large-scale microblogging streams. Specifically, we propose an online microblogging stream clustering algorithm to cluster microblogs and maintain distilled statistics called Microblog Cluster Vectors (MCV). Then we develop a ranking method to extract the most representative sentences relative to the query from the MCVs and generate a query-focused summary of arbitrary time durations. Our experiments on large-scale real microblogs demonstrate the efficiency and effectiveness of our approach.     

Cross-Layer Optimization for Video Summary Transmission over Wireless Networks

Video summarization has gained increased popularity in the emerging multimedia communication applications, however, very limited work has been conducted to address the transmission problem of video summary frames. In this paper, we propose a cross-layer optimization framework for delivering video summaries over wireless networks. Within a rate-distortion theoretical framework, the source coding, allowable retransmission, and adaptive modulation and coding have been jointly optimized, which reflects the joint selection of parameters at physical, data link and application layers. The goal is to achieve the best video quality and content coverage of the received summary frames and to meet the delay constraint. The problem is solved using Lagirangian relaxation and dynamic programming. Experimental results indicate the effectiveness and efficiency of the proposed optimization framework, especially when the delay budget imposed by the upper layer applications is small, where more than 10% distortion gain can be achieved.     

Adaptive key frame extraction for video summarization using an aggregation mechanism

Video summarization is a method to reduce redundancy and generate succinct representation of the video data. One of the mechanisms to generate video summaries is to extract key frames which represent the most important content of the video. In this paper, a new technique for key frame extraction is presented. The scheme uses an aggregation mechanism to combine the visual features extracted from the correlation of RGB color channels, color histogram, and moments of inertia to extract key frames from the video. An adaptive formula is then used to combine the results of the current iteration with those from the previous. The use of the adaptive formula generates a smooth output function and also reduces redundancy. The results are compared to some of the other techniques based on objective criteria. The experimental results show that the proposed technique generates summaries that are closer to the summaries created by humans.     

基于颜色和灰度熵的视频镜头检测方法

视频镜头分割是视频检索和视频分类等应用的关键步骤,因此提出一种基于颜色直方图和灰度熵的镜头检测算法。该方法以HSV颜色直方图和图像的灰度熵为特征,计算视频相邻两图像帧的相似度,从而判断出是否发生了镜头变换。实验证明,该方法能准确检测出视频的突变镜头,对渐变镜头也有理想的检测效果。     

基于聚类的视频镜头分割和关键帧提取

镜头分割是基于内容的视频检索和浏览首先要解决的关键技术。视频分割为镜头后,下一步的工作就是进行关键帧提取,用以描述镜头的主要内容。提出了一种改进的基于聚类的镜头分割和关键帧提取算法．在无监督聚类中引入一个参考变量,解决了利用无监督聚类进行镜头分割和关键帧提取时可能产生的帧序不连续或分割错误的问题。在关键帧提取阶段,将镜头分割为子镜头后,引入图像熵的概念提取关键帧。实验结果表明了改进算法在镜头分割和关键帧提取方面的有效性。     

Video abstraction based on the visual attention model and online clustering

With the fast evolution of digital video, research and development of new technologies are greatly needed to lower the cost of video archiving, cataloging and indexing, as well as improve the efficiency and accessibility of stored video sequences. A number of methods to respectively meet these requirements have been researched and proposed. As one of the most important research topics, video abstraction helps to enable us to quickly browse a large video database and to achieve efficient content access and representation. In this paper, a video abstraction algorithm based on the visual attention model and online clustering is proposed. First, shot boundaries are detected and key frames in each shot are extracted so that consecutive key frames in a shot have the same distance. Second, the spatial saliency map indicating the saliency value of each region of the image is generated from each key frame and regions of interest (ROI) is extracted according to the saliency map. Third, key frames, as well as their corresponding saliency map, are passed to a specific filter, and several thresholds are used so that the key frames containing less information are discarded. Finally, key frames are clustered using an online clustering method based on the features in ROIs. Experimental results demonstrate the performance and effectiveness of the proposed video abstraction algorithm.