基于深度学习的语言模型研究进展

语言模型旨在对语言的内隐知识进行表示,作为自然语言处理的基本问题,一直广受关注.基于深度学习的语言模型是目前自然语言处理领域的研究热点,通过预训练-微调技术展现了内在强大的表示能力,并能够大幅提升下游任务性能.围绕语言模型基本原理和不同应用方向,以神经概率语言模型与预训练语言模型作为深度学习与自然语言处理结合的切入点,...

Language Models Based on Deep Learning: A Review

Language model, to express implicit knowledge of language, has been widely concerned as a basic problem of natural language processing in which the current research hotspot is the language model based on deep learning. Through pre-training and fine-tuning techniques, language models show their inherently power of representation, also improve the performance of downstream tasks greatly. Around the basic principles and different application directions, this paper takes the neural probability language model and the pre-training language model as a pointcut for combining deep learning and natural language processing. We introduce the application as well as challenges of neural probability and pre-training model, which is based on the basic concepts and theories of language model. Then the existing neural probability, pre-training language model include their methods are compared and analyzed. In addition, we elaborate on the training methods of pre-training language model from two aspects of new training tasks and improved network structure. Meanwhile the current research directions of pre-training model in scale compression, knowledge fusion, multi-modality and cross-language are summarized and evaluated. Finally, this paper sums up the bottleneck of language model in natural language processing application, afterwards prospects for possible future research priorities.