基于深度学习的行为识别多模态融合方法综述

行为识别是通过对视频数据进行处理分析从而让计算机理解人的动作和行为.不同模态数据在外观、姿态、几何、光照和视角等主要特征上各有优势,通过多模态融合将这些特征进行融合可以获得比单一模态数据更好的识别效果.本文对现有行为识别多模态融合方法进行介绍,对比了它们之间的特点以及获得的性能提升,包括预测分数融合、注意力机制、知识蒸馏等晚期融合方法,以及特征图融合、卷积、融合结构搜索、注意力机制等早期融合方法.通过这些分析和比较归纳出未来多模态融合的研究方向.

Survey on Multi-modality Fusion Methods for Action Recognition Based on Deep Learning

Action recognition aims to make computers understand human actions by the processing and analysis of video data. As different modality data have different strengths in the main features such as appearance, gesture, geometric shapes, illumination, and viewpoints, action recognition based on the multi-modality fusion of these features can achieve better performance than the recognition based on single modality data. In this study, a comprehensive survey of multi-modality fusion methods for action recognition is given, and their characteristics and performance improvements are compared. These methods are divided into the late fusion methods and the early fusion methods, where the former includes prediction score fusion, attention mechanisms, and knowledge distillation, and the latter includes feature map fusion, convolution, fusion architecture search, and attention mechanisms. Upon the above analysis and comparison, the future research directions are discussed.