基于文本相似度的网页消重策略

针对在网页检索结果中经常出现内容相同或相似的问题,提出了一种通过计算网页相似度的方法进行网页消重。该算法通过提取网页特征串,特征串的提取在参考以往特征码提取的基础上,加入了文本结构特征的提取,通过比较特征串之间差异性的基础上得到网页的相似度。经与相似方法比较,结果表明,该方法减少了时间复杂度,具有较高的查全率和查准率,适于大规模网页消重。

STRATEGY OF ELIMINATING DUPLICATED WEB PAGES BASED ON TEXT SIMILARITY

In order to solve the problems of identical or similar contents frequently occur in web pages retrieval,a method of eliminating duplicated web pages by calculating the similarity of the web pages was put forward.This algorithm uses the extraction of feature string of the web pages.Such extraction of feature string is based on previous feature code extraction as the reference and is combined with the extraction of text structural features.By comparing differences between the feature strings the similarities ...