Spark框架结合分布式KNN分类器的网络大数据分类处理方法

针对现有大数据分类方法难以满足大数据应用中时间和储存空间的限制，提出了一种基于Apache Spark框架的大数据并行多标签K最近邻分类器设计方法。为了通过使用其他内存操作来减轻现有MapReduce方案的成本消耗，首先，结合Apache Spark框架的并行机制将训练集划分成若干分区；然后在Map阶段找到待预测样本每个分区的K近邻，进一步在reduce阶段根据map阶段的结果确定最终的K近邻；最后并行地对近邻的标签集合进行聚合，通过最大化后验概率输出待预测样本的目标标签集合。在PokerHand等四个大数据分类数据集上进行实验，提出方法取得了较低的汉明损失，证明了其有效性。

Network big data classification processing method based on Spark and distributed KNN classifier

Aiming at the limitation that the existing big data classification methods can not meet the time and storage space in big data applications, this paper proposed a design method of big data parallel multi-label K-nearest neighbor classifier based on Apache Spark framework. In order to reduce the cost of the existing MapReduce scheme by using other memory operations, firstly, it divided the training set into several partitions in conjunction with the parallel mechanism of the Apache Spark framework. Then in the Map stage, it found the K nearest neighbors of each partition of the sample to be predicted, and in the Reduce phase, it determined the final K nearest neighbors according to the results of the Map phase. Finally, it aggregated the neighboring tag sets in parallel, and output the target tag set of the sample to be predicted by maximizing the posterior probability. Experiments were carried out on four big data classification datasets such as PokerHand. The proposed method achieved a lower Hamming loss and proved its effectiveness.