基于Spark框架的FP-Growth大数据频繁项集挖掘算法

针对大数据中的频繁项集挖掘问题,提出一种基于Spark框架的FP-Growth频繁项集并行挖掘算法。首先,根据垂直布局思想将数据按照事务标识符垂直排列,以此解决扫描整个数据集的缺陷。然后,通过FP-Growth算法构建频繁模式树,并生成频繁1-项集。接着,通过扫描垂直数据集来计算项集的支持度,从而识别出非频繁项,并将其从数据集中删除以降低数据尺寸。最后,通过迭代过程来生成频繁 -项集。在标准数据集上的实验结果表明,该算法能够有效挖掘出频繁项集,在执行时间方面具有很大的优越性。

Frequent Item sets Mining Algorithm for big data based on FP-Growth and Spark Framework

For the issues of the frequent item sets mining in big data, a parallel frequent item sets mining algorithm based on FP-Growth and Spark framework is proposed. Firstly, the data is arranged vertically according to the transaction identifier, in order to solve the defects of scan the entire data set. Then, the FP-Growth algorithm is used to construct the frequent pattern tree and generate frequent 1-item sets. After that, the support of the item set is calculated by scanning the vertical data set, so as to identify the non-frequent items, and delete them from the data set to reduce the data size. Finally, the iterative process is used to generate frequent item sets. The experimental results on the standard dataset show that the algorithm can effectively excavate frequent item sets and have great superiority in execution time.