自注意力机制的视频摘要模型

针对如何高效地识别出视频中具有代表性的内容问题,提出了一种对不同的视频帧赋予不同重要性的视频摘要算法.首先使用长短期记忆网络来建模视频序列的时序关系,然后利用自注意力机制建模视频中不同帧的重要性程度并提取全局特征,最后通过每一帧回归得到的重要性得分进行采样,并使用强化学习策略优化模型参数.其中,强化学习的动作定义为每一帧选或者不选,状态定义为当前这个视频的选择情况,反馈信号使用多样性和代表性代价.在2个公开数据集SumMe和TVSum中进行视频摘要实验,并使用F-度量来衡量这2个数据集上不同视频摘要算法的准确度,实验结果表明,提出的视频摘要算法结果要优于其他算法.

Self-Attention Based Video Summarization

Video summarization aims to identify the most representative contexts in videos.In this paper,we propose a new video summarization method which assigns different importance to video frames.Specifically,we exploit bidirectional LSTMs to capture temporal information of video frames and then employ self-attention mechanism to pay different attention on different frames for extracting their global features.Finally,we sample an action for each frame by using the corresponding regression score and apply the reinforcement learning strategy to optimize parameters in our model,where actions are defined as select or not select the current frame,states are defined as actions for the whole video,and the reward is defined as the sum of representative and diversity costs.We conduct video summarization experiments on two public video summarization datasets including SumMe and TVSum and evaluate the performance by using F-measure.Experimental results demonstrate that our proposed video summarization method has achieved the superior performance,comparing to the state-of-the-arts.