加入自注意力机制的BERT命名实体识别模型

命名实体识别属于自然语言处理领域词法分析中的一部分，是计算机正确理解自然语言的基础。为了加强模型对命名实体的识别效果，本文使用预训练模型BERT(bidirectional encoder representation from transformers)作为模型的嵌入层，并针对BERT微调训练对计算机性能要求较高的问题，采用了固定参数嵌入的方式对BERT进行应用，搭建了BERT-BiLSTM-CRF模型。并在该模型的基础上进行了两种改进实验。方法一，继续增加自注意力(self-attention)层，实验结果显示，自注意力层的加入对模型的识别效果提升不明显。方法二，减小BERT模型嵌入层数。实验结果显示，适度减少BERT嵌入层数能够提升模型的命名实体识别准确性，同时又节约了模型的整体训练时间。采用9层嵌入时，在MSRA中文数据集上F₁值提升至94.79%，在Weibo中文数据集上F₁值达到了68.82%。

BERT named entity recognition model with self-attention mechanism

Named entity recognition is a part of lexical analysis in the field of natural language processing. It is the basis for a computer to correctly understand natural language. In order to strengthen the recognition effect of the model on named entities, in this study, the pre-trained model BERT (bidirectional encoder representation from transformers) was used as the embedding layer of the model, and fixed parameter embedding was adopted to solve the problem of high computer performance required for BERT fine-tuning training. A BERT-BiLSTM-CRF model was built, and on the basis of this model, two improved experiments were carried out. Method one is to continue to add a self-attention layer. Experimental results show that the addition of the self-attention layer does not significantly improve the recognition effect of the model. Method two is to reduce the number of embedding layers of the BERT model. Experimental results show that moderately reducing the number of BERT embedding layers can improve the model’s named entity recognition accuracy, while saving the overall training time of the model. When using 9-layer embedding, thevalue on the MSRA Chinese data set increased to 94.79%, and thevalue on the Weibo Chinese data set reached 68.82%.