面向MapReduce的中间数据传输流水线优化机制

MapReduce是一种适用于大数据处理的重要并行计算框架,通过在大量集群节点上并行执行多个任务,极大地提高了数据的处理性能。然而,由于中间数据需要等到Mapper任务完成之后才能被发送给Reducer任务,由此导致的大量传输延迟成为MapReduce框架性能的重要瓶颈。为此,文中提出了一种面向MapReduce的中间数据传输流水线优化机制,将有效计算与中间数据传输解耦,以流水线的方式重叠执行各个阶段,有效隐藏数据传输开销。文中还给出了中间数据传输流水线执行机制和实现策略,包括流水线划分、数据细分、数据归并和数据传输粒度等。在公开数据集上对所提中间数据传输流水线优化机制进行了评价,当Shuffle数据量较大时,该优化机制比默认框架的整体性能提高了60.2%。

Intermediate Data Transmission Pipeline Optimization Mechanism for MapReduce Framework

MapReduce is an important parallel computing framework for large data processing,which greatly improves the performance of data processing by performing multiple tasks in parallel on a large number of cluster nodes.However,since the intermediate data needs to wait until the Mapper task is completed,it can be sent to the Reducer task.The massive transmission delay becomes an important bottleneck of the MapReduce framework performance.To this end,an intermediate data transmission pipeline mechanism for MapReduce is proposed.It decouples the effective computation from intermediate data transmission,overlaps each stage in pipeline mode,and effectively hides data transmission delay.The execution mechanism and implementation strategy of the approach are given,including pipeline partition,data subdivision,data merging and data transmission granularity.The proposed mechanism is evaluated on public data sets.When the Shuffle data volume is large,the overall performance improves by 60.2% compared with the default framework.