一种基于Hadoop+CUDA实现相关器的方法

根据21CMA相关器的算法特点,在对比基于CPU并行的MPI集群、MPI+CUDA异构并行集群和Hadoop+CUDA异构并行集群的架构特点的基础上,提出了一种基于Hadoop+CUDA平台实现软相关器的方法。本方法利用GPU在计算FFT、向量乘和向量加等密集型计算模型的优势,设计相关器的并行模型,使其性能较前期在CPU并行的MPI集群实现的相关器有了大幅提升。同时,本文选择广泛应用于大数据处理平台的Hadoop软件架构,利用Hadoop Streaming工具实现非Java编写的程序在分布式系统中并行执行,非常便捷地获得了集群系统的线性加速比。Hadoop HDFS并行文件系统管理结果数据和过程日志更加灵活可靠,为后续的大数据分析提供了支撑环境。

A correlator implementation method based on Hadoop+CUDA

According to the characteristics of the 21CMA correlator algorithm, we propose a novel high efficient method to implement this specific algorithm on the Hadoop+CUDA platform, and it outperforms the MPI alone and MPI+CUDA solutions. The proposed method improves the parallel model of the correlator. Compared to the earlier MPI solution, it greatly enhances the running performance by utilizing the advantages of GPU for FFT processing, vector multiplication and vector addition. The Hadoop software architecture, a big data platform, is employed in the method by using Hadoop Streaming tool to realize parallel execution of non Java programs running on distributed systems, and linear speed ups on clusters are easily obtained. In addition， the result data and procedure logs can be flexibly managed in the parallel file system of the Hadoop HDFS, which provides a well precondition for future big data analysis.