News Text Topic Clustering Optimized Method Based on TF-IDF Algorithm on Spark |
| |
Authors: | Zhuo Zhou Jiaohua Qin Xuyu Xiang Yun Tan Qiang Liu Neal N Xiong |
| |
Affiliation: | 1.School of Computer & Software, Jiangsu Engineering Center of Network Monitoring, Nanjing University of Information Science & Technology, Nanjing, 210044, China.
2 State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, China.
3 Department of Economics, Finance, Insurance and Risk Management University of Central Arkansas, Conway, 72035, USA. |
| |
Abstract: | Due to the slow processing speed of text topic clustering in stand-alone
architecture under the background of big data, this paper takes news text as the research
object and proposes LDA text topic clustering algorithm based on Spark big data
platform. Since the TF-IDF (term frequency-inverse document frequency) algorithm
under Spark is irreversible to word mapping, the mapped words indexes cannot be traced
back to the original words. In this paper, an optimized method is proposed that TF-IDF
under Spark to ensure the text words can be restored. Firstly, the text feature is extracted
by the TF-IDF algorithm combined CountVectorizer proposed in this paper, and then the
features are inputted to the LDA (Latent Dirichlet Allocation) topic model for training.
Finally, the text topic clustering is obtained. Experimental results show that for large data
samples, the processing speed of LDA topic model clustering has been improved based
Spark. At the same time, compared with the LDA topic model based on word frequency
input, the model proposed in this paper has a reduction of perplexity. |
| |
Keywords: | News text topic clustering spark platform countvectorizer algorithm TFIDF algorithm latent dirichlet allocation model |
|
| 点击此处可从《》浏览原始摘要信息 |
|
点击此处可从《》下载全文 |
|