一种提高中文搜索引擎检索质量的HTML解析方法

中文搜索引擎经常会返回大量的无关项或者不含具体信息的间接项,产生这类问题的一个原因是网页中存在着大量与主题无关的文字。对使用关键字检索方法的搜索引擎来说,想在检索或者后处理阶段解决这类问题不仅要付出一定代价,而且在大多数情况下是不可能的。在这篇论文中,我们提出了网页噪声的概念,并针对中文网页的特点,实现了一种对网页自动分块并去噪的HTML解析方法,从而达到在预处理阶段消除潜在无关项和间接项的目的。实验结果表明,该方法能够在不占用查询时间的前提下100%地消除中文搜索引擎隐藏的间接项,以及大约11%的无法过滤或隐藏的无关项或间接项,从而大幅度提高检索结果的查准率。

A HTML Parser to Improve Chinese Search Engines

While using search engine, people always find so many irrelevant or peripherally relevant items in the result list. Most of them are produced by the words irrelevant to the topic of a web page. It is costly or even impossible to remove such items using traditional keyword methods. In this paper, we define the concept of noise in web pages, and propose a novel approach to clean the noise information of web pages in the pre-processing stage. A novel model of Chinese web pages and 4 simple rules are build to discard noise from HTML files. Experimental results show that, all the indirect items that appear in the results of site grouping are removed correctly and about 11% irrelevant or indirect items that cannot be excluded by commercial Chinese search engines are removed by our approach.