基于Hadoop的海量数据存储平台设计与开发

随着北部湾海洋生态资源的开发和利用,海量海洋科学数据飞速涌现出来,利用海量数据存储平台合理管理和存储这些科学数据显得极为重要.这里提出了一种基于分布式计算技术进行管理和存储海量海洋科学数据方法,构建了海量海洋科学数据存储平台解决方案,采用Linux集群技术,设计开发一个基于Hadoop的海量数据存储平台.系统由五大模块组成,有系统管理模块、并行加载存储模块、并行查询模块、数据字典模块、备份恢复模块,能够实现存储海量海洋科学数据.系统模块实现结果表明,该系统安全可靠、易维护、具有良好的可扩展性.

Design and Development of the Mass Data Storage Platform Based on Hadoop

With the development and utilization of BeiBu Bay Marine ecological resources, mass marine science data rapidly emerge in large numbers and it is very important to use a mass data storage platform to manage and store these science data reasonable. This paper puts forward the management and storage the mass marine science data methods based on the distributed computing technology, builds the mass marine science data storage platform solutions, designs and develops a mass data storage platform based on Hadoop by using Linux cluster technology. This system which consists of five modules includes system management module, parallel loading storage module, parallel query module, data dictionary module, backup and recovery module and it can achieve to store massive amounts of marine science data. The system module achieving result shows that this system enjoys good safety, reliability, easy maintenance and good expansibility.