依托Hadoop架构的海量变压器实时监测与存储方案构建

随着智能电网的建设以及电力变压器在线监测技术的成熟,电力变压器在线监测数据呈现出体量大、类型多等特点。使用传统存储技术存储变压器在线监测数据,已不能满足实时、快速的需求。为此,设计基于Hadoop集群的变压器在线监测数据存储方案。该方案利用HBase(分布式列式数据库)具有快速实时读写数据的优势,将变压器在线监测系统采集的海量数据实时快速地存储。为能自动快速实时收集数据和避免因数据流过大造成系统崩溃,分别采用Flume(日志收集工具)和Kafka(分布式流处理平台)收集和缓存数据。以电力变压器在线监测的油色谱数据为例,验证了所提存储方案的可行性和有效性。

Construction of massive transformer real-time monitoring and storage solution relying on Hadoop architecture

With the construction of smart grids and the maturity of on-line monitoring technology for power transformers, the online monitoring data of power transformers are characterized by large volumes and types. Using traditional storage technology to store transformer online monitoring data can no longer meet real-time and fast requirements. To this end, a Hadoop cluster-based transformer online monitoring data storage scheme is designed. This solution utilizes the advantages of HBase (distributed columnar database) to quickly read and write data in real time, and stores the massive data collected by the transformer online monitoring system in real time and quickly. To automatically and quickly collect data in real time and avoid system crashes due to excessive data flow, Flume (log collection tool) and Kafka (distributed stream processing platform) are used to collect and cache data, respectively. Finally, the oil chromatogram data of power transformer online monitoring is taken as an example to verify the feasibility and effectiveness of the proposed storage scheme.