有效HTML文本信息抽取方法的研究*

从新闻网页和博客网页中抽取出正文内容是一个非常有意义的研究问题,但是多数网页中含有大量与正文无关的噪声内容，导致很难从网页中获取正确的文本信息。分析了中文新闻与博客网页的正文特征，用实验表明了利用HTML与文本的密度比可以进行文本的识别与抽取。提出了机器学习、统计估计以及FDR三种HLML正文抽取方法，并作了大量的实验比较和分析。实验结果表明，该算法可以有效地过滤噪声而且算法的复杂度很低，效率与效果均达到一个很好的平衡。

Research on methods for extracting text information from HTML pages

Extracting text information from news and blog HTML pages is a very important and interesting research problem. There are too many noises to extract precise texts. The paper analyzed statistic characterizes of HTML pages and showed it was possible to extract texts using information about the density of text vs. HTML code. Proposed three methods based on machine leaning, statistic and false discovery rate(FDR). Conducted comprehensive experiments and the result show these methods can effectively extract texts and balance effectivity and efficiency.