| Spark框架并行度推断算法 |
| |
| 引用本文： | 卞琛,于炯,修位蓉,廖彬,英昌甜,鲁亮. Spark框架并行度推断算法[J]. 电子科技大学学报(自然科学版), 2019, 48(4): 567-574. DOI: 10.3969/j.issn.1001-0548.2019.04.014 |
| |
| 作者姓名： | 卞琛 于炯 修位蓉 廖彬 英昌甜 鲁亮 |
| |
| 作者单位： | 广东金融学院互联网金融与信息工程学院 广州 510521;新疆大学信息科学与工程学院 乌鲁木齐 830046;新疆大学信息科学与工程学院 乌鲁木齐 830046;新疆财经大学统计与信息学院 乌鲁木齐 830012 |
| |
| 基金项目： | 新疆维吾尔自治区自然科学基金2017D01A20 |
| |
| 摘 要： | 分布式计算集群Spark宽依赖并行度取决于用户设定参数，对于不同的作业类型或数据集，硬编码的并行度参数设定难以发挥集群的最大计算能效。针对这一问题，首先对Spark作业执行方式进行深入分析，建立作业调度模型，提出宽依赖计算代价、资源空置率和溢写概率的定义；然后分析任务并行度对作业执行时间的影响，证明并行度取值具有合理区间，提出并行度推断算法的优化目标。最后根据模型定义进行目标求解，设计批处理内存计算框架的并行度推断算法（parallelism deduction algorithm，PDA），通过构建的数据总量、执行区预留比、操作闭包集合、资源表等多个基础数据，计算符合资源需求表且具有最大资源利用率和最小开销的任务并行度；PDA算法在作业的各个Stage中迭代执行，根据计算环境优化调度方案提高性能。实验表明，PDA算法提高了Spark框架的作业执行效率，针对不同类型作业均具有良好的普适性。
|
| 关 键 词： | 内存计算 并行度推断 性能优化 Spark 溢写概率 |
| 收稿时间： | 2017-04-24 |
Parallelism Deduction Algorithm for Spark |
| |
| Affiliation: | 1.College of Internet Finance and Information Engineering, Guangdong University of Finance Guangzhou 5105212.College of Information Science and Engineering, Xinjiang University Urumqi 8300463.College of Statistics and Information, Xinjiang University of Finance and Economics Urumqi 830012 |
| |
| Abstract: | Inappropriate parallelism parameter may result in the performance degradation on in-memory computing framework. For this issue, we analyze the execution mechanism of Spark jobs, establish job scheduling model, and give the definition of the computing cost, resource idle rate and spill probability. Based on the analysis of the relationship between parallelism parameter and job execution efficiency, the optimization objective of algorithm is given. To solve the problem of optimizing, a parallelism deduction algorithm (PDA) for in-memory computing framework is proposed. Firstly, PDA calculates the best parallelism of job execution by size of input data, worker computing resource and additional overhead of spill, and thus enhances the resource utilization of cluster and speeds up the state synchronization of job execution. The algorithm optimizes the task scheduling for each Stage, accelerates the job execution and improves the calculation efficiency. Experiment results demonstrate that the proposed algorithm can improve the computational efficiency of in-memory computing framework and accelerate data-intensive and compute-intensive applications. |
| |
| Keywords: | |
| 本文献已被 万方数据 等数据库收录！ |
| 点击此处可从《电子科技大学学报(自然科学版)》浏览原始摘要信息 |
|
点击此处可从《电子科技大学学报(自然科学版)》下载全文 |
