一种基于独立性统计的子串归并算法

现行的子串归并算法都是采用一对一的方式针对同频子串提出的。但是在使用词法分析工具对文本进行切分时，不可避免地会产生很多的分词碎片，这直接导致了很多无意义子串的产生。通过分析这些无意义子串和众多父串之间的这种一对多关系，提出了一种基于独立性统计的子串归并算法。最后将该子串归并算法应用在中文术语抽取系统中，使得系统的准确率从91.3%提升到了93.32%。

Substring reduction algorithm based on independence statistic

The substring reduction algorithm applied in most cases is mainly focusing on the substrings having the same frequency with the parent string in one to one mode.After being processed by the morphological analysis tool,it＇s unavoidable to product many segment fragments which compose many meaningless substrings.According to the analysis of the one to multiple relationship between the meaningless substring and its parent strings,a substring reduction algorithm based on independence statistic is proposed to filter these meaningless substrings.Finally,this substring reduction algorithm is applied in the Chinese multi-words terminology extraction system,and the precision of the term extraction results is improved from 91.3% to 93.32%.