基于Spark框架的乘潮水位计算与可视化平台

乘潮水位计算是海洋环境信息处理的重要组成部分,具有计算量大、计算复杂度高、计算时间长等特性。采用传统集群计算模式实现乘潮水位计算业务,存在计算成本高、计算伸缩性和交互性差的问题。针对以上问题,提出一种基于Spark框架的乘潮水位计算和可视化平台。结合对Spark任务调度算法的研究,设计和实现了一种基于节点计算能力的任务调度算法,实现了长时间序列的多任务乘潮水位数据的检索、获取、数值计算、特征可视化的并行处理,达到了海量海洋环境数据计算和可视化处理的目的。实验结果表明,提出的基于Spark的乘潮水位计算和可视化平台可以有效地提高海量乘潮水位数据的分布式并行处理的效率,为更加快速和高效的乘潮水位计算提供了一种新的方法。

Tide-bound water level computing and visualization platform based on Spark

Tide bound water level computing is an important part of ocean environment information processing, which features huge amount of data, high complexity, and prolonged computing time. The traditional computing model implemented by HPC has a number of problems, such as high computation cost, poor scalability and interactivity. Aiming at all these problems, we propose an interactive computing and visualization platform based on the Spark scheduling algorithm. We design a computing capacity scheduling algorithm, realize the parallel processing of large scale tide bound water level data, such as data retrieval, data extraction, numerical calculation, feature based visualization, and achieve the purpose of parallel processing and visualization of large scale ocean environmental data on Spark. Experimental results show that the computing and visualization platform based on Spark can improve the traditional computing model, lessen the dependence of tidal level calculation on high performance cluster and reduce computation cost. In addition, the newly developed task scheduling algorithm can make task allocation more rational and scientific, and therefore further enhance its efficiency. In conclusion, the proposed platform provides a new method for tide bound water level computing.