基于关键词相似度的短文本分类方法研究

在传统的文本分类中,文本向量空间矩阵存在“维数灾难”和极度稀疏等问题,而提取与类别最相关的关键词作为文本分类的特征有助于解决以上两个问题。针对以上结论进行研究,提出了一种基于关键词相似度的短文本分类框架。该框架首先通过大量语料训练得到word2vec词向量模型;然后通过TextRank获得每一类文本的关键词,在关键词集合中进行去重操作作为特征集合。对于任意特征,通过词向量模型计算短文本中每个词与该特征的相似度,选择最大相似度作为该特征的权重。最后选择K近邻（KNN）和支持向量机（SVM）作为分类器训练算法。实验基于中文新闻标题数据集,与传统的短文本分类方法相比,分类效果约平均提升了6%,从而验证了该框架的有效性。

Research on short text classification based on keyword similarity

In order to cope with the problem of data sparsity and "curse of dimensionality" in text classification, this paper proposed a short text classification framework by taking keyword as featrues and assigning keyword similarity as feature weight. First, it trained a word2vec model with large corpus data, then got keywords of each category text by textrank. And it selected unique keywords from the keywords collection as features. For each feature, it calculated the similarity of words in the short text by word2vec model, and assigned the maximum similarity as the weight of the feature. Finally, it chose KNN and SVM as classifier. Experiments on dataset of Chinese news headlines demonstrate that the accuracy outperforms other usual methods by 6%.