News Text Topic Clustering Optimized Method Based on TF-IDF Algorithm on Spark

Due to the slow processing speed of text topic clustering in stand-alonearchitecture under the background of big data, this paper takes news text as the researchobject and proposes LDA text topic clustering algorithm based on Spark big dataplatform. Since the TF-IDF (term frequency-inverse document frequency) algorithmunder Spark is irreversible to word mapping, the mapped words indexes cannot be tracedback to the original words. In this paper, an optimized method is proposed that TF-IDFunder Spark to ensure the text words can be restored. Firstly, the text feature is extractedby the TF-IDF algorithm combined CountVectorizer proposed in this paper, and then thefeatures are inputted to the LDA (Latent Dirichlet Allocation) topic model for training.Finally, the text topic clustering is obtained. Experimental results show that for large datasamples, the processing speed of LDA topic model clustering has been improved basedSpark. At the same time, compared with the LDA topic model based on word frequencyinput, the model proposed in this paper has a reduction of perplexity.