利用HDF5和Esper的高效外汇数据分析系统

为解决目前关系型数据库由于死锁和阻塞等原因造成系统随数据量增加性能急剧降低的问题,本文提出结合HDF5和Esper的数据处理方法,利用HDF5(层次型数据存储格式)的强大I/O性能和数据类型自定义功能做数据存储,加快数据存储的速度。同时,结合Esper实时计算的高运算效率特征,设计出一套针对外汇计算行业的可移植的数据格式和函数库,解决了即时数据分析系统在存储、管理、交换、归档需求中出现的性能和效率问题。实验结果表明,在10G/天的数据量下,HDF5的数据读写性能比MSSQL要高80倍左右,结合Esper的毫秒级处理能力,能够做到外汇数据的实时计算。系统对于数据处理分析和实时性要求极高的情况有较高的应用价值。

An Effective Data Analysis System of Foreign Exchange Using HDF5 and Esper

To solve the existing problem of the decreasing low performance of the relational database caused by dead lock and blocking as the bulk data increase,the authors put forward a method of handling data with HDF5 and Esper by taking the advantage of HDF5's powerful I/O performance and Esper's high operational efficiency property.Thus,the design of a portable data format and a function bank may solve the problems of performance and efficiency that exist in the instant systematic storage,management,exchange and the filing requirement of the data analysis.The experimental results indicate that HDF5's data reading and writing performance are 80 times higher than that of the MSSQL under the condition of 10G per day data bulk.Taking Esper's millisecond level handling capacity into account,the system has a relatively high application value in the field of data handling analysis and the real-time requirement.