面向信息检索需要的网络数据清理研究

Web数据中的质量参差不齐、可信度不高以及冗余现象造成了网络信息检索工具存储和运算资源的极大浪费,并直接影响着检索性能的提高。现有的网络数据清理方式并非专门针对网络信息检索的需要,因而存在着较大不足。本文根据对检索用户的查询行为分析,提出了一种利用查询无关特征分析和先验知识学习的方法计算页面成为检索结果页面的概率,从而进行网络数据清理的算法。基于文本信息检索会议标准测试平台的实验结果证明,此算法可以在保留近95%检索结果页面的基础上清理占语料库页面总数45%以上的低质量页面,这意味着使用更少的存储和运算资源获取更高的检索性能将成为可能。

Web Data Cleansing for Effective Information Retrieval

The existence of low quality Web pages affects the effectiveness and efficiency of Web search.In this paper,we define the Web page quality estimation as a learning problem.First,several query-independent features are investigated which can separate search target page from ordinary ones.Bayes estimation based on these features is then used to train a model to assign importance scores to Web pages.In TREC based experiments,the top-scored set reduces 45% low quality pages as well as retains 95% high quality ones.It shows the possibility to gain better performance with less storage and computing resource for search engines.