一种主题爬虫文本分类器的构建

该文利用DF与CHI统计量相结合的特征选取方法,针对互联网上对外汉语相关领域的网页进行特征提取,并在此基础上,构建了基于标题与正文相结合的两步式主题相关度判定分类器。基于该分类器做对外汉语相关主题的网页爬取工作,实验表明,效率和召回率比传统分类器都有较大程度的提高,目前该分类器已经用于为大型对外汉语语料库构建提供数据源。

A Method of Text Classifier for Focused Crawler

This paper combines DF and CHI to select features of web pages related to the area of teaching Chinese as a second language (TCSL). A classifier is first constructed based on two-step topic similarity measurement by the title and the main text. The classifier is then applied to crawling web pages related to TCSL, and the results show substantial improvements on efficiency and recall rate compared with traditional methods. Now this classifier has been deployed for data collection for a big TCSL corpus in actual practice.
Key wordsDF; CHI; classifier; focused crawler