基于 Counting Bloom Filter 的海量网页快速去重研究

网页去重是从给定的大量的数据集合中检测出冗余的网页,然后将冗余的网页从该数据集合中去除的过程,其中基于同源网页的URL去重的研究已经取得了很大的发展,但是针对海量网页去重问题,目前还没有很好的解决方案,文章在基于MD5指纹库网页去重算法的基础上,结合Counting Bloom Filter算法的特性,提出了一种快速去重算法IMP-CBFilter。该算法通过减少I/O频繁操作,来提高海量网页去重的效率。实验表明,IMP-CBFilter算法的有效性。

Rapid De-Duplication of Massive Web Page Based on Counting Bloom Filter

Web page de-duplication is a process which detects redundant duplicate content pages from a given amount of data collection , and then removes from the collection .The research of web de-duplica-tion based on URL filter has achieved great development .But there is no ideal solution to solve this kind of problem with massive web pages filter .Based on MD5 fingerprint database web de-duplication algo-rithm, and the utilization of Counting Bloom filter algorithm , an algorithm for rapid de-duplication named as IMP-CBFilter has been proposed in this paper .It could improve the efficiency of mass web pages filter by reducing the frequent operation of I/O.The results indicate higher performance by using IMP-CBFilter algorithm .