Spark中一种高效RDD自主缓存替换策略研究

由于并行计算框架Spark缓存替换算法的粗糙性，LRU（least recently used）算法并未考虑RDD的重复使用导致易把高重用数据块替换出内存且作业执行效率较低等问题。通过优化权重模型和改进替换策略，提出了一种高效RDD自主缓存替换策略（efficient RDD automatic cache，ERAC），包括高重用自主缓存算法和缓存替换分级算法，可实现高效RDD的自主缓存和缓存目标的分级替换。最后利用SNAP（Stanford network analysis project）提供的标准数据集将ERAC和LRU、RA（register allocation）等算法进行了对比实验，结果显示ERAC算法能够有效提高Spark的内存利用率和任务执行效率。

Research on efficient RDD self-cache replacement strategy in Spark

Due to the roughness of cache replacement mechanism in Spark parallel computing framework, LRU algorithm does not consider the RDD data block reuse problem that easily results in high-reuse block to swap out of the cache and execution inefficiency. By optimizing the weight model and improving the replacement strategy, this paper put forward ERAC cache strategy, which could achieve efficient RDD autonomous cache and hierarchical replacement, included high-reuse self-caching algorithm and cache replacement hierarchical algorithm. Applied ERAC, RA and LRU in standard dataset of SNAP, experimental shows that ERAC can effective improve the memory utilization and task execution efficiency.