一种基于新词发现的Web文本表示方法

Web文本表示是Web文本特征提取和分类的前提，最常用的文本表示是向量空间模型（VSM），其中向量一般是基于词的特征项。由于向量空间模型本身没有考虑文本上下文间的潜在概念结构（如词汇间的共现关系），而Web文本是一种半结构化文本，同时经常有新词出现，因此在VSM基础上提出了一种基于新词发现的Web文本表示方法：首先进行预处理将网页转化为文本；然后进行文本分词；接着通过二元互信息进行新词发现，同时把新词加入字典重新分词；最后用词和新词共同来表示Web文本。实验结果表明，该方法可以帮助识别未登录词并扩充现有字典，能够增强Web文本表示能力，改善Web文本的特征项质量，提高Web文本分类效果。

Web document representation method based on new-word discovery

Web document representation is important foundation of Web page feature selection and categorization. At present, the commonly used method is word-based vector space model. Because the model does not consider the latent concept construction of text, and a Web document is in semi-structural form where many new-words perhaps occur, a new Web document representation method based on new-word discovery was presented in this paper. This method first preprocessed the Web page, and segmented words for the converted documents; then tried to discover new-words by using bi-gram and mutual information, added new-words into its original dictionary, and segmented words for text again, finally represented the Web document using words and new-words. The experimental results show that it can help us to identify unknown words and extend the current dictionary, strengthen the representation of Web documents, improve the quality of the adopted vector, and increase the effect of Web document categorization.