一种基于动态拓扑的流计算性能优化方法及其在Storm中的实现

响应性和稳定性一直是流式计算中两个至关重要的问题，而流计算系统在过载时常常表现出数据计算延迟增加和拓扑不稳定的现象，无法适应数据负载的动态变化.针对这一问题本文研究提出了一种基于动态拓扑的流计算性能优化方法，主要包括：（1）动态逐级反压：拓扑中的任务可以根据当前自身负载情况，动态调整上游向其发送数据的速率.（2）无状态拓扑数据重放：拓扑不维持数据的计算状态，尽可能地实现数据容错.（3）自适应拓扑替换：在拓扑不暂停的情况下对任务并发度进行自发调整.（4）延迟持久化队列：拓扑中对磁盘的IO读写被延迟到数据处理之外，减缓IO高频阻塞对流计算系统的影响.本文在Apache Storm中实现了以上四种方案，性能测试结果表明优化后的流计算系统与Storm默认实现相比，不仅增强了大数据动态匹配能力，而且在最优情况下改善了17%的吞吐量，并提升了约20%的数据处理速度.

A Performance Optimization Method Based on Dynamic Topology for Stream Computing and Its Implementation in Storm

Responsiveness and stability have always been two important problems in stream computing.However,as the scale of data being processed in real-time has increased,along with an increase in the data processing latency and topology instability of stream computing,many limitations of stream processing system have become apparent.Aiming at these problems,we present a performance optimization method based on dynamic topology for stream computing:(1) Dynamic step-by-step backpressure:the task in the topology can dynamically adjust the rate of upstream data transmission according to the current load.(2) Stateless topology data replay:topology can achieve data fault tolerance autonomously without maintaining the calculation of data state.(3) Adaptive topology replacement:no need for topology to suspend,the system can adjust the task concurrency spontaneously.(4) Delayed persistent queue:it delays the IO reading and writing in the disk out of the data processing,which mitigates the impact of IO high-frequency blocking in stream computing system.In this paper,the four methods are implemented in Apache Storm.The experimental results show that the optimized system not only enhances the dynamic matching capability of big data,but also achieves 17% higher throughput and 20% better data processing speed in the best case.