科研大数据平台关键技术与实践

首先,以高能物理领域数据处理为例,分析了支撑科学研究的大数据平台在存储和处理能力、传输和共享等方面的挑战,说明现有技术已经难以满足日益快速增长的数据处理需求。然后,给出了科研大数据平台的典型架构,并讨论科研大数据平台的关键技术,包括数据采集与清洗、数据存储、数据处理、数据传输、数据共享与安全等技术,同时介绍了各种关键技术的研究现状或者主流系统。最后,介绍了中国科学院高能物理研究所科研大数据开放平台的建设思路和实现框架,该平台试图解决目前大数据发展过程中面临的一些问题,如数据开放和跨领域融合不够、跨地域数据传输性能低等,从而激活数据价值,降低应用门槛。

Key Technologies and Practice of Big Data Platform for Scientific Research

This paper firstly discusses the challenges of big data platform for scientific research in the abilities of data storage, processing, transferring and sharing based on the example of data processing in the field of high en-ergy physics. Then the paper presents a typical structure of big data platform for scientific research and gives a brief review on some key technologies, including data acquisition, cleaning, storage, analysis, transfer, sharing and security. The popular systems and the current research situation of these key technologies are also described in the paper. Fi-nally, the concept and framework of the big data platform of High Energy Physics Institute of Chinese Academy of Sciences is introduced in this paper. This open platform is established as the solution of challenges/problems in big data evolution, for example, data acquisition, data transferring, and application interface.