结合关键帧提取的视频-文本跨模态实体分辨双重编码方法

现有的视频-文本跨模态实体分辨方法在视频处理上均采用均匀取帧的方法，必然导致视频信息的丢失，增加问题的复杂度。针对这一问题，提出一种结合关键帧提取的视频-文本跨模态实体分辨双重编码方法（DEIKFE）。以充分保留视频信息表征为前提，设计关键帧提取算法提取视频中的关键帧，获得视频关键帧集合表示。对于视频关键帧集合和文本，采用多级编码的方法，分别提取表征视频和文本的全局、局部和时序的特征，将其进行拼接形成多级编码表示。将该编码表示映射至共同嵌入空间，采用强负样本跨模态三元组损失对模型参数进行优化，使得匹配的视频-文本相似度越大，而不匹配的视频-文本相似度越小。通过在MSR-VTT、VATEX两个数据集上进行实验验证，与现有方法进行对比，在总体性能R@sum上分别提升了9.22%、2.86%，证明了该方法的优越性。

Dual Encoding Integrating Key Frame Extraction for Video-text Cross-modal Entity Resolution

Existing video-text cross-modal entity resolution methods all adopt a method of uniformly extracting frames in video processing，which inevitably leads to the loss of video information and increases the model complexity.A dual encoding integrating key frame extraction (DEIKFE) is proposed for video-text cross-modal entity resolution. On the premise of fully retaining the video information，a key frame extraction algorithm is designed to extract the key frames in the video，which makes up the video key frame set. For the video key frame set and the text，a multi-level encoding method is adopted to extract the global，local，and time-series features，which are spliced to form a multi-level encoding representation. And the encoding representation is mapped into a common embedding space，and the model parameters are optimized by cross-modal triplet ranking loss based on the hard negative sample to make the matched video-text similarity greater and the unmatched video-text similarity smaller. The experiments on MSR-VTT and VATEX datasets show that the overall performance of R@sum is increased by 9.22% and 2.86%，respectively，comparedwith the existing methods, which can fully demonstrate the superiority of the proposed method.