融合词语类别特征和语义的短文本分类方法

针对短文本内容简短、特征稀疏等特点,提出一种新的融合词语类别特征和语义的短文本分类方法。该方法采用改进的特征选择方法从短文本中选择最能代表类别特征的词语构造特征词典,同时结合利用隐含狄利克雷分布LDA主题模型从背景知识中选择最优主题形成新的短文本特征,在此基础上建立分类器进行分类。采用支持向量机SVM与k近邻法k-NN分类器对搜狗语料库数据集上的搜狐新闻标题内容进行分类,实验结果表明该方法对提高短文本分类的性能是有效的。

A short text classification method combining lexical category features and semantics

Classification of short texts is challenging due to their typical characteristics of severe sparseness and high dimension. We propose a novel approach to classify short texts by combining both lexical and semantic features. To construct the term dictionary, we firstly select lexical features of the most distinctive words of a certain category, and then extract the optimal topic distribution from the background knowledge repository based on the Latent Dirichlet Allocation so as to construct the new features of short texts. Experiments on classifying Sohu news titles which are typical short texts via SVM and K-NN show that our method can greatly improve the classification results.