文本词向量与预训练语言模型研究

介绍了文本词向量及预训练语言模型的发展体系,系统整理并分析了其中重点方法的思想特点。首先,阐述了传统的文本词向量表征方法及基于语言模型的文本表征方法;然后,详述了预训练语言模型方法的研究进展,包括动态词向量的表征方法和基于Transformer架构的预训练模型;最后,指出了未来探究多模态间更有效的融合方式和迁移学习将成为该领域的发展趋势。

A Survey of Research on Word Vectors and Pretrained Language Models

This paper mainly introduces the development system of text word vectors and pre-trained language models,systematically organizes and analyzes the ideological characteristics of key methods.Firstly,we describe the traditional text word vector representation method and the language model-based text representation method,then we elaborate the research progress of the pre-trained language model method,including the dynamic word vector representation method and the Transformer architecture-based pre-training model.Finally,it is pointed out that in the future,exploring more effective fusion methods and transfer learning between multi-modalities will become a development trend in this field.