地学大数据处理架构与关键技术研究

【目的】大数据以其独特的数据科学思维为地学研究知识发现带来重大机遇,但地学数据独特的多源异构、时空关联、多尺度和不确定性等特征亦给地学大数据处理带来一系列挑战。【方法】本文在分析地学数据特点基础上,结合数据关联、中间件系统、微服务及容器等技术手段,提出一种面向地学大数据的处理框架,重点解决地学领域多源数据汇集融合、异构数据综合集成处理问题,并将地学模型引入框架,增强数据处理的地学专业性。【结果】框架及其关键技术已在国家冰川冻土科学数据中心建设、高寒环境联合观测研究云及中巴走廊灾害数据集制备中应用实施。【结论】地学大数据平台处理框架拓宽数据处理维度,可为多主题、多尺度地学研究分析和知识发现提供支撑,未来框架将适应互联网、社交网络、平面媒体等更广泛来源的地学数据处理,进一步融合人工智能技术,提供更智能更迅捷的地学数据处理结果。

Research on Geoscience Big Data Processing Framework and Key Techniques

[Objective]As one of the special data science methods,big data brings great opportunities for geological research.Meanwhile,the characteristics of geological data such as multi-source heterogeneity,spatial-temporal correlation,multi-scale and uncertainty bring great challenges for the data processing.[Methods]On the basis of a detailed analysis of aforementioned characteristics,this study proposes a geological data processing framework to solve problems of multi-source data integration and heterogeneous data synthesis in geoscience field combined with a variety of big data technologies like data association,middleware systems,micro services and container technique.Besides,geological models are embedded in this framework in order to improve the expertness of data process.[Results]The framework and its key technologies have been applied in the construction of the National Glacier and Frozen Soil Scientific Data Center,the disaster datasets for the China-Pakistan Corridor as well as the High and Cold Environment United Observation Cloud.[Conclusions]This study is expected to broaden the data processing dimension and support multi-theme,multi-scale research and knowledge discovery in geoscience.In future,it will be adapted to the processing of geological data from a wider range of sources such as the internet,social networks,and printed media.The integration of artificial intelligence technologies will enable the framework to provide smarter and faster geological data processing results.