| 海量自动气象站分钟数据存储检索系统设计 |
| |
| 引用本文： | 王建荣,季 刚. 海量自动气象站分钟数据存储检索系统设计[J]. 计算机与现代化, 2017, 0(9): 40. DOI: 10.3969/j.issn.1006-2475.2017.09.008 |
| |
| 作者姓名： | 王建荣 季 刚 |
| |
| 基金项目： | 中国气象局关键技术集成项目(CMAGJ2015M29); 安徽省气象局科技发展基金资助项目(KM201604) |
| |
| 摘 要： | 地面自动气象站的时空密度不断增加，产生的观测数据量呈指数级增长，传统的关系型数据库在海量数据存储与检索方面存在能力不足、检索性能下降等问题。鉴于此，本文设计一种自动站分钟数据存储与检索系统。使用Quartz定时采集自动站分钟文件并解码入库；应用HBase分布式数据库建立分钟数据存储模型；针对多要素查询需求，应用Elasticsearch建立辅助索引，实现HBase的二级索引。系统测试结果表明，分钟数据入库平均耗时54.6 s，二级索引完整可靠，数据检索结果返回时效达到毫秒级，能够满足业务应用中对自动站分钟数据存储和检索时效的要求。
|
| 关 键 词： | 自动站分钟数据 Kafka HBase Elasticsearch 协处理器 |
| 收稿时间： | 2017-09-19 |
Design of Storage and Retrieval System for Mass Automatic Weather Station Minute Data |
| |
| WANG Jian-rong,JI Gang. Design of Storage and Retrieval System for Mass Automatic Weather Station Minute Data[J]. Computer and Modernization, 2017, 0(9): 40. DOI: 10.3969/j.issn.1006-2475.2017.09.008 |
| |
| Authors: | WANG Jian-rong JI Gang |
| |
| Abstract: | With the rapid growth of temporal and spatial density of ground automatic weather station, the quantity of observed data increases exponentially, traditional relational database has insufficient ability and low performance in mass data storage and retrieval. In view of this, an automatic station minute data storage and retrieval system is designed. Quartz is used to collect automatic station minute files and then the system decodes the minute files and storage decoded data into database. The system uses HBase to build minute data storage model. In order to meet the query demands of multiple elements, the system applies Elasticsearch to build secondary index of HBase. Test results show that the average storage time for minute data is 54.6 s and the secondary index with Elasticsearch is complete and reliable and the retrieval results returned time to milliseconds. Therefore, the system can satisfy the demand for storage and query performance of automatic weather station minute data. |
| |
| Keywords: | AWS minute data Kafka HBase Elasticsearch coprocessor |
|
| 点击此处可从《计算机与现代化》浏览原始摘要信息 |
|
点击此处可从《计算机与现代化》下载全文 |
