L-PowerGraph: a lightweight distributed graph-parallel communication mechanism

In order to process complex and large-scale graph data, numerous distributed graph-parallel computing platforms have been proposed. PowerGraph is an excellent representative of them. It has exhibited better performance, such as faster graph-processing rate and higher scalability, than others. However, like in other distributed graph computing systems, unnecessary and excessive communications among computing nodes in PowerGraph not only aggravate the network I/O workload of the underlying computing hardware systems but may also cause a decrease in runtime performance. In this paper, we propose and implement a mechanism called L-PowerGraph, which reduces the communication overhead in PowerGraph. First, L-PowerGraph identifies and eliminates the avoidable communications in PowerGraph. Second, in order to further reduce the required communications L-PowerGraph proposes an edge direction-aware master appointment strategy, in which L-PowerGraph appoints the replica with both incoming and outgoing edges as master. Third, L-PowerGraph proposes an edge direction-aware graph partition strategy, which optimally isolates the outgoing edges from the incoming edges of a vertex during the graph partition process. We have conducted extensive experiments using real-world datasets, and our results verified the effectiveness of the proposed mechanism. For example, compared with PowerGraph under Random partition scenario L-PowerGraph can not only reduce up to 30.5% of the communication overhead but also cut up to 20.3% of the runtime for PageRank algorithm while processing Live-journal dataset. The performance improvement achieved by L-PowerGraph over our precursor work, LightGraph, which only reduces the synchronizing communication overhead, is also verified by our experimental results.