改进的SemBERT特征重组模型

SemBERT模型实现了对BERT模型的提升, 但存在两个明显的缺陷. 第一, 模型获得向量表示的能力不足. 第二, 没有从特征的种类出发, 直接使用传统特征进行任务分类. 针对这两个缺陷, 提出了一种新的特征重组网络. 该模型在SemBERT内部添加自注意力机制, 外接特征重组机制, 得到更好的向量表示并且重新分配特征权重. 实验数据表明新的方法在MRPC数据集上比经典的SemBERT模型在F1值上提高了1%. 实现在小数据集上的明显提升, 并且超越了大多数优秀模型.

Improved SemBERT-based Feature Reorganization Model

Although the SemBERT model is an improved version of the BERT model, it has two obvious defects. One is its poor ability to obtain vector representation. The other is that it directly uses conventional features to classify tasks without considering the category of the features. A new feature reorganization network is proposed to address these two defects. This model adds a self-attention mechanism into the SemBERT model and obtains better vector representation with an external feature reorganization mechanism. Feature weights are also reassigned. Experimental data show that the F1 score of the new method on the Microsoft Research Paraphrase Corpus (MRPC) dataset is one percentage point higher than that of the classical SemBERT model. The proposed model has significantly improved performance on small datasets, and it outperforms most of the current outstanding models.