基于新词发现的跨领域中文分词方法

深度神经网络(DNN)是目前中文分词的主流方法，但将针对某一领域训练的网络模型用于其他领域时，会因存在跨领域的未登录词(OOV)和表达鸿沟而造成性能显著下降，而在实际中对所有未知领域的训练语料进行人工标注和训练模型并不可行。为了解决这个问题，该文构建了一个基于新词发现的跨领域中文分词系统，可以自动完成从目标领域语料中提取新词、标注语料和训练网络模型的工作。此外，针对现有新词发现算法提取出的词表垃圾词串多以及自动标注语料中存在噪声样本的问题，提出了一种基于向量增强互信息和加权邻接熵的无监督新词发现算法以及一种基于对抗式训练的中文分词模型。实验中将使用北大开源新闻语料训练的网络模型提取出的特征迁移到医疗、发明专利和小说领域，结果表明该文所提方法在未登录词率、准确率、召回率和分词F值方面均优于现有方法。

Cross-domain Chinese Word Segmentation Based on New Word Discovery

Deep Neural Network (DNN) is the major method in current Chinese word segmentation. However, its performance is significantly degraded when the network trained for one domain is used in other domains due to the Out Of Vocabulary (OOV) words and expression gaps. In this paper, a cross domain Chinese word segmentation system based on new word discovery is built to handle the OOV word and expression gap problems. An unsupervised new word discovery algorithm based on vector enhanced mutual information and weighted adjacency entropy, and a Chinese word segmentation model based on adversarial training are also proposed to improve the performance of the baseline system. Experimental results show that the proposed method is superior to the conventional methods in the OOV rates, precisions, recalls and F-scores.