基于核心词项平均划分相似度的短文本聚类算法

针对短文本特征极度稀疏、上下文依赖性强等特点,以自顶向下的策略,提出一种基于核心词项平均划分相似度的短文本聚类算法CTMPS。该方法首先在整个短文本语料库中计算词项之间的概率相关性,以此为基础对短文本中词项进行加权,将权值较大的词项作为最能代表该短文本的核心词项形成核心词项集;以信息论为基础,将核心词项作为划分依据计算平均划分相似度,选择平均划分相似度值最大包含该核心词项的短文本形成一类,用此策略反复迭代直到满足要求。最后,实验结果表明,本文提出的方法显著地提高了短文本聚类的性能。

Core term based mean partition similarity for short text clustering

Aiming at the characteristics of extreme sparse and context dependent features of short texts, we propose a novel core term based mean partition similarity for short text clustering algorithm (CTMPS) with top-down strategy. The CTMPS firstly determines probabilistic correlation among terms in the corpus. Secondly, based on the probabilistic correlation,terms in a short text are weighted. The terms with larger weight are considered as the most representative terms of the short text and they then form the core terms set. On the basis of information theory, mean partition similarity (MPS) is calculated via core terms, and the MPS with the maximum core terms in the short text forms one class. Finally, experimental results show that the CTMPS outperforms the baseline algorithm in term of performance and clustering efficiency.