| 基于Rabin指纹方法的URL去重算法 |
| |
| 引用本文： | 梁正友,张林才. 基于Rabin指纹方法的URL去重算法[J]. 计算机应用, 2008, 28(Z2) |
| |
| 作者姓名： | 梁正友 张林才 |
| |
| 作者单位： | 1. 广西大学计算机与电子信息学院,南宁,530004
2. 广西大学计算机与电子信息学院,南宁,530004;辽宁石油化工大学计算机与通信工程学院,辽宁,抚顺,113001 |
| |
| 基金项目： | 广西科学基金资助项目 ， 广西教育厅科研项目 |
| |
| 摘 要： | 针对现有URL检索算法占用存储空间较大,对重复率高的URL集合检索速度较慢,使Web Spider的效率降低的问题,提出了一种改进的URL去重算法.此算法基于Rabin指纹方法,以URL的指纹为地址,仅用一位数据标识一条URL,每次检索仅需对相应的一位数据的值做一次判断.实验表明,该算法能有效去除URL集合中重复的URL.提高检索速度.
|
| 关 键 词： | URL去重 Rabin指纹方法 |
Duplicated URL detection based on Rabin's fingerprint method |
| |
| LIANG Zheng-you,ZHANG Lin-cai. Duplicated URL detection based on Rabin's fingerprint method[J]. Journal of Computer Applications, 2008, 28(Z2) |
| |
| Authors: | LIANG Zheng-you ZHANG Lin-cai |
| |
| Affiliation: | LIANG Zheng-you1,ZHANG Lin-cai1,2(1.School of Computer , Electronic Information,Guangxi University,Nanning Guangxi 530004,China,2.School of Computer , Communication Engineering,Liaoning Shihua University,Fushun Liaoning 113001,China) |
| |
| Abstract: | The existing URL retrieval algorithms take up large memory,so their retrieval speed becomes slow when they retrieval the URL set with high repetition rate,and the gathering efficiency of the Web Spider is low.To resolve these problems,an improved duplicated URL detection algorithm based on Rabin's fingerprint method was proposed.In the new algorithm,the URL fingerprint was taken as its address.Marking an URL only needs one bit and each retrieval only needs once judgment to a corresponding bit of the data.Th... |
| |
| Keywords: | Web spider |
| 本文献已被 CNKI 万方数据 等数据库收录！ |
|
