一种新词检测方法研究

汉语自动分词是进行中文信息处理的基础。目前,困扰汉语自动分词的一个主要难题就是新词自动识别,尤其是非专名新词的自动识别。同时,新词自动识别对于汉语词典的编纂也有着极为重要的意义。提出一种新的未登录词识别方法,该方法混合了互信息、信息熵及词频等3个评价指标评价新词,并在此基础上添加了垃圾串过滤机制,大幅度提高了新词识别准确率和召回率。

Research on new word detection

Chinese automatic segmentation is the basis of Chinese information processing. Currently, a major problem that has plagued Chinese automatic segmentation is new words identity. At the same time, it is important for the compilation of Chinese dictionary. This paper presents a new method for new word identify. It includes three parameters such as mutual information, entropy of information, word frequency and also includes garbage string filtering mechanism. It improves the accuracy and the recall rate of new words greatly.