一种基于主动学习的中文新词识别算法

分词是中文自然语言处理的重要基础，新词的不断涌现是分词的最大难题。针对新词识别定义不清、语料缺乏的实际问题，提出了一种以大规模神经网络预训练模型为基础，并结合主动学习和人工规则的新词识别算法。利用预训练模型高效识别候选新词，使用基于不确定性和代表性样本选择的主动学习策略辅助标注新词，利用热度规则、突发性规则和合成性规则识别和过滤新词发现结果。针对新词识别评价标准不一致的问题，给出了一般性准确率和受限制准确率两条规范测试指标。与现有最优算法进行实验对比，所提算法两项指标分别提高了16%和4%。

Chinese New Words Recognition Based on Active Learning

Word segmentation is an essential basis of Chinese natural language processing,and the continuous emergence of new words is the biggest problem of word segmentation.To solve the problems of unclear definition and lack of corpus,a new word recognition algorithm based on a large-scale neural network pre-training model combined with active learning and standard rules is proposed.The pre-training model is used to identify new word candidates efficiently.The active learning strategy based on uncertainty and representative sample selection is used to assist new word labelling.The heat rule,the burst rule,and the compound rule are used to identify and filter new word discovery results.Two standard test metrics,general-accuracy rate and restricted-accuracy rate are given to solve the problem of inconsistent evaluation criteria.The experimental comparison of relevant research results shows that the two metrics are improved by 16% and 4%,respectively,compared with the existing algorithms.