基于互信息和邻接熵的新词发现算法

如何快速高效地识别新词是自然语言处理中一项非常重要的任务，针对当前新词发现存在的问题，提出了一种从左至右逐字在未切词的微博语料中发现新词的算法。通过计算候选词语与其右邻接字的互信息来逐字扩展，得到候选新词；并通过计算邻接熵、删除候选新词的首尾停用词和过滤旧词语等方法来过滤候选新词，最终得到新词集。解决了因切词错误导致部分新词无法识别以及通过n-gram方法导致大量重复词串和垃圾词串识别为新词的问题，最后通过实验验证了该算法的有效性。

New word discovery algorithm based on mutual information and branch entropy

How to identify new words quickly and efficiently is a very important task in natural language processing. Aiming at the problems existing in the discovery of new words, there is an algorithm for word-finding new words verbatim from left to right in the uncut word Weibo corpus. One way to get a candidate new word is by computing the candidate word and its right adjacent word mutual information to expand word by word; There are some ways to filter candidate new words to get new word sets. The included methods include calculating the branch entropy, deleting stop words contained in the first or last word of each candidate new word and deleting old words included in the candidate new word set. It solves the problem that some new words can not be recognized due to the mistakes in the word segmentation and It also solves the problem that the large number of repetitive word strings and rubbish words strings generated by the n-gram method are identified as new words. Finally, experiments verified the effectiveness of the algorithm.