Spark环境下并行立方体计算方法

针对传统联机分析处理(OLAP)处理大数据时实时响应能力差的问题,研究基于分布式内存计算框架Spark加速的数据立方体计算方法,设计基于Spark内存集群的自底向上构造(BUC)算法——BUCPark,来提高BUC的并行度和大数据适应能力。在此基础上,为避免内存中迭代的立方体单元膨胀,基于内存重复利用和共享的思想设计改进的BUCPark算法——LBUCPark。实验结果表明:LBUCPark算法性能优于BUC算法和BUCPark算法,能够胜任大数据背景下的快速数据立方体计算任务。

Parallel cube computing in Spark

In view of the poor real-time response capability of traditional OnLine Analytical Processing (OLAP) when processing big data, how to accelerate computation of data cubes based on Spark was investigated, and a memory-based distributed computing framework was put forward. To improve parallelism degree and performance of Bottom-Up Construction (BUC), a novel algorithm for computation of data cubes was designed based on Spark and BUC, referred to as BUCPark (BUC on Spark). Moreover, to avoid the expansion of iterative data cube in memory, BUCPark was fruther improved to LBUCPark (Layered BUC on Spark) which could take full advantage of reused and shared memory mechanism. The experimental results show that LBUCpark outperforms BUC and BUCPark algorithms in terms of computing performace, and it is capable of computing data cube efficiently in big data era.