| 基于Hadoop的风力发电监测大数据存储优化及并行查询方法 |
| |
| 引用本文： | 王林童,赵 腾,张 焰,苏 运.基于Hadoop的风力发电监测大数据存储优化及并行查询方法[J].电测与仪表,2018,55(11):01-06. |
| |
| 作者姓名： | 王林童 赵 腾 张 焰 苏 运 |
| |
| 作者单位： | 上海交通大学电气工程系,上海,200240
国网上海市电力公司电力科学研究院,上海,200437 |
| |
| 基金项目： | 国家高技术研究发展计划项目(863计划)(2015AA050203)，国家电网公司科技项目(520900150037) |
| |
| 摘 要： | 随着风力发电的广泛发展以及智能化监测技术的推广应用,风力发电监测数据呈现出体量大、类型多、增长快的大数据特征.针对风力发电监测大数据高效存储和快速查询两方面核心问题,基于Hadoop平台进行大数据存储优化方法研究,提出考虑风力发电监测数据关联性的哈希分桶存储算法,实现了相关联数据的集中存储,从而提升后期数据查询及处理的效率.在数据存储优化的基础上,实现基于MapReduce的多源风力发电监测大数据并行关联查询.通过在Hadoop平台上进行测试表明,经过哈希分桶存储优化后的多源数据并行关联查询相比传统Hadoop方法查询时间显著缩短.
|
| 关 键 词： | 大数据 风力发电监测 Hadoop 哈希分桶算法 big data wind power monitoring Hadoop Hash bucket algorithm |
| 收稿时间： | 2017/6/13 0:00:00 |
| 修稿时间： | 2017/6/14 0:00:00 |
Storage optimization and parallel query method for big data of wind power monitoring based on Hadoop |
| |
| Wang Lintong,Zhao Teng,Zhang Yan and Su Yun.Storage optimization and parallel query method for big data of wind power monitoring based on Hadoop[J].Electrical Measurement & Instrumentation,2018,55(11):01-06. |
| |
| Authors: | Wang Lintong Zhao Teng Zhang Yan and Su Yun |
| |
| Affiliation: | Department of Electrical Engineering,Shanghai Jiaotong University,Department of Electrical Engineering,Shanghai Jiaotong University,Department of Electrical Engineering,Shanghai Jiaotong University,Electric Power Research Institute,SMEPC |
| |
| Abstract: | With the extensive development of wind power generation and the generalized application of intelligent monitoring technology,wind power monitoring data shows the big data characteristics of large volume,multi types and fast growth.In order to solve the two major problems of big data with efficient storage and quick query,in this paper,the optimization method of big data storage is studied based on Hadoop platform.A Hash bucket algorithm considering wind power monitoring data correlation is proposed,which realizes the centralized storage of related data,so as to enhance the efficiency of data query and processing.On the basis of data storage optimization,the parallel association query for multi-source big data of wind power monitoring based on MapReduce is realized.Tests on a Hadoop platform show that the time of the multi-source data parallel association query is significantly shortened than traditional Hadoop method after optimization of hash bucket storage. |
| |
| Keywords: | big data wind power monitoring Hadoop hash bucket algorithm |
| 本文献已被 万方数据 等数据库收录！ |
| 点击此处可从《电测与仪表》浏览原始摘要信息 |
|
点击此处可从《电测与仪表》下载全文 |
|
