知识感知的预训练语言模型综述

随着自然语言处理（NLP）领域中预训练技术的快速发展，将外部知识引入到预训练语言模型的知识驱动方法在NLP任务中表现优异，知识表示学习和预训练技术为知识融合的预训练方法提供了理论依据。概述目前经典预训练方法的相关研究成果，分析在新兴预训练技术支持下具有代表性的知识感知的预训练语言模型，分别介绍引入不同外部知识的预训练语言模型，并结合相关实验数据评估知识感知的预训练语言模型在NLP各个下游任务中的性能表现。在此基础上，分析当前预训练语言模型发展过程中所面临的问题和挑战，并对领域发展前景进行展望。

Survey of Knowledge-Aware Pre-Trained Language Models

In the field of Natural Language Processing(NLP), the recent years has witnessed a rapid development in pre-training technology, and the knowledge-driven method that injects external knowledge into a pre-trained language model performs excellently in NLP tasks.The techniques of knowledge representation learning and pre-training provide theoretical foundation for the knowledge-based pre-training method.This paper briefly introduces the development of the classical pre-trained methods.Then it analyzes the representative knowledge-aware pre-trained language models supported by new pre-training technology.According to the types of external knowledge, this paper introduces the pre-trained language models injected with different external knowledge.Based on relevant experimental data, it subsequently evaluates the performance of the knowledge-aware pre-trained language models in various downstream tasks of NLP.On this basis, the paper analyzes the problems and challenges faced by the developing pre-trained language models, and discusses the development trends of this field.