大规模数据流统计中冷热流替换策略优化

对大规模数据流统计的问题进行了研究,针对大流统计的典型结构Elastic Sketch替换策略中存在的问题进行优化,优化策略解决了冷流被误判为热流插入重部的问题.针对重部中保存的不一定是最大流的问题进行优化,提出了基于最大值和组相连的替换策略,保证了存储在重部的一定是最大的流,提高了大流统计的精度,同时大大降低了热碰撞...

Optimization of cold and hot flows replacement in large-scale data flow statistics

This paper studies the problem of large-scale data flow statistics, and optimizes the problems in the replacement strategy of Elastic Sketch, which is a typical structure of large flow statistics. The optimization strategy solves the problem of cold flow being misjudged as hot flow inserted into the heavy part. In order to optimize the problem that the flow stored in the heavy part may not be the largest flow, a replacement strategy based on the maximum value and group connection is proposed to ensure that the largest flow stored in the heavy part is guaranteed to improve the accuracy of the large flow statistics. At the same time, the probability of thermal collisions is greatly reduced. Compared with the traditional measurement statistics method, the measurement accuracy is improved while the memory usage is reduced.