首页 | 本学科首页   官方微博 | 高级检索  
     

配电网监测数据的分布式Map压缩-查询技术
引用本文:屈志坚,陈鼎龙,王群峰.配电网监测数据的分布式Map压缩-查询技术[J].电力自动化设备,2017,37(12).
作者姓名:屈志坚  陈鼎龙  王群峰
作者单位:华东交通大学 电气与自动化工程学院,江西 南昌 330013,华东交通大学 电气与自动化工程学院,江西 南昌 330013,华东交通大学 电气与自动化工程学院,江西 南昌 330013
基金项目:国家自然科学基金资助项目(51567008);江西省杰出青年人才计划项目(20162BCB23045);江西省自然科学基金资助项目(20161BAB206156,20171BAB206044)
摘    要:针对配电自动化中大量电力监测数据的处理问题,提出了回避聚合操作的配电网监测数据分布式Map压缩-查询新方法。通过将监测数据分布式Map压缩存储,利用HQL查询引擎及压缩接口将分布式Map压缩应用到连接查询的混洗阶段中,减小传递到查询聚合端的数据量,提高压缩数据的查询速度,并推导了时效性的相关公式。以北京某动车段10 kV电力远动监控系统的实测数据为例,搭建了四节点测试集群。压缩导入对比测试表明,分布式Map压缩速度快于分布式Reduce压缩,分布式Map的Map_Deflate压缩处理时间比分布式Reduce_Deflate减少了45.3%;压缩-查询测试表明,当数据量为2×10~7记录级时,分布式Map的Map_LZO格式压缩-查询耗时大幅降低,比混洗阶段不压缩-查询时减少了31.6%,验证了分布式Map压缩对加速查询的时效性。

关 键 词:配电网    大数据    集群平台    分布式压缩    压缩-查询
收稿时间:2016/12/19 0:00:00
修稿时间:2017/10/11 0:00:00

Distributed Map compression-query technology for distribution network monitoring data
QU Zhijian,CHEN Dinglong and WANG Qunfeng.Distributed Map compression-query technology for distribution network monitoring data[J].Electric Power Automation Equipment,2017,37(12).
Authors:QU Zhijian  CHEN Dinglong and WANG Qunfeng
Affiliation:School of Electrical and Automation Engineering, East China Jiaotong University, Nanchang 330013, China,School of Electrical and Automation Engineering, East China Jiaotong University, Nanchang 330013, China and School of Electrical and Automation Engineering, East China Jiaotong University, Nanchang 330013, China
Abstract:In view of the considerable quantity of monitoring data in distribution automation, an innovative method of distributed Map compression-query for distribution network monitoring data is proposed, in which the aggregation operation can be avoided. The monitoring data is compressed and stored through distributed Map compression. Then, the distributed Map compression is applied to the shuffle stage of join query by HQL query engine and compression interface, which reduces the data quantity transferred to the reducers, and improves the query speed of the compressed data. Besides, the formulas of time effectiveness are deduced. The measured data of a Beijing EMU(Electric Multiple Unit) depot 10 kV power remote monitoring system is taken as an example, where a four-node cluster is constructed to carry out experiments. The compression importing comparison test results demonstrate that the distributed Map compression speed is faster than the distributed Reduce compression, and the Map_Deflate compression processing time decreases by 45.3 % compared to distributed Reduce_Deflate. Meanwhile, the compression-query test results demonstrate that the compression-query time of Map_LZO greatly reduces when the amount of data is 2 × 107 records, which decreases by 31.6 % compared to uncompressed-query in shuffle stage. Hence, these results verify the efficiency of distributed Map compression for accelerating query.
Keywords:distribution network  massive data  cluster platform  distributed compression  compression-query
本文献已被 CNKI 等数据库收录!
点击此处可从《电力自动化设备》浏览原始摘要信息
点击此处可从《电力自动化设备》下载全文
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司  京ICP备09084417号