具有词判别力学习能力的短文本聚类概率模型研究

社交媒体的广泛使用使短文本聚类成为一个重要的研究课题。但短文本词向量的高维、稀疏性限制了传统文本聚类方法在短文本中的效果,并且由于词的稀疏性,词对簇结构的判别能力对短文本类结构的学习显得尤为重要。本文我们提出了一种基于概率模型的具有词判别力学习能力的短文本聚类框架,并在经典文本聚类模型LDA(Ldatant Drichilet Allocation)、BTM(Biterm Topic Model)和GSDMM(Gibbs Sampling Drichilet Mutitional Mixture model)模型中验证了词判别力学习对类结构学习的有效性。通过Gibbs采样算法对模型中的参数进行求解。最后在真实数据集上的实验结果显示具有词判别力学习的概率模型可以提高已有模型的聚类效果。

Research on Short - text Clustering Probability Model with Word Discrimination Ability

The widespread use of social media makes short texts clustering an important research topic. However, the high-dimensions and sparseness of the short-text word vector limits the effect of the conventional document clustering approach in short text, and because of the sparseness of the word, the discriminant of the word on class structure is very important for short text clustering. In this paper, we propose a short text clustering framework based on probabilistic model with word discriminant learning ability, and validates the effectiveness of word discrimination learning in class structure learning in LDA (Latent Dirichlet Allocation), BTM (Biterm Topic Model) and GSDMM (Gibbs Sampling algorithm for the Dirichlet Multinomial Mixture model). The experimental results on the real data set show that the probability model with word discriminant learning can improve the clustering effect of the existing model.