一种面向流数据频繁项挖掘的降载策略*

流数据产生速率具有不可预见性,当其速率超过系统处理能力时,部分数据元素不能被实时处理。降载技术是处理此问题的关键技术之一。分析了目前降载技术的不足,提出了一种面向挖掘流数据频繁项集的降载策略。该策略采用了基于元组出现频率的语义删除策略,优先删除出现频率相对较低的元组,从而有效解决了在挖掘流数据中的频繁项所遇到系统超载时所出现的问题,同时采用了根据流数据产生速率自动地控制是否启动降载策略,有效地解决了降载的适应性问题。最后,通过实验和分析,证明了该策略在流数据频繁项挖掘中有效性。     

基于延迟时间的实时降载策略

降载可以有效地解决数据流流速的突变所引发的系统的过载问题。为此,分析现有降载策略在实时性方面的不足,不仅考虑系统的处理能力,同时结合数据的处理延迟,提出基于延迟时间的降载策略(DSP)。实验结果表明,DSP可以有效地解决系统的过载,同时具有较高的实时性。     

流系统的降载技术

流数据连续查询及优化研究已成为当前国际数据库研究领域的一个热点。数据流的到达速率经常是不可预测的且具有很高的突发性,数据流速这种不规则的变化会引起系统负载急剧的波动。当输入速率超过系统处理能力时,系统会发生过载并且导致系统性能的恶化,降载技术是解决此问题最有效的途径之一。对降载技术中系统负载估计、降载器的最佳的放置位置、降载量的大小、降载器合并等关键问题进行了讨论。     

数据流滑动窗口连接的卸载策略研究

随着数据流应用系统的快速发展,数据流管理系统对数据库技术提出了巨大挑战.针对数据流上的滑动窗口连接操作,提出一些新的卸载技术,使得系统在过载的情况下卸载连接结果少的元组,从而最大化输出结果.双窗口模型和辅助窗口统计信息的建立保证了预估连接结果的可靠性,同时应用线段树使卸载的判断更加高效.当流速过快、系统处理能力无法与之同步时,通过前端卸载和后端卸载的配合使用达到理想的语义卸载,得到最大子集的连接结果.实验验证该卸载策略的性能好于现有其他方法.     

有限通信资源下多数据流连接的降载算法

针对数据源节点通信资源十分有限的缺陷,提出一种基于直方图的多数据流滑动窗口连接查询的降载策略。该算法综合中心处理节点和数据源节点的负载情况,给出降载比例计算公式,通过使用聚类技术构建中心直方图和数据源直方图,给出降载策略。结果证明该算法能够产生精确连接结果的极大子集,对多数据流窗口连接降载是高效的。     

数据流系统中卸载技术研究综述

随着数据流应用系统的快速流行,流数据管理对数据库技术提出了巨大挑战。由于数据流经常是爆发性的且数据特征可能随时变化,因此要求数据流管理系统具有很好的自适应性。当输入速率超过系统处理能力时,系统会产生过载且性能下降。为了解决这一问题,卸载技术是有效的途径之一。卸载时间、卸载地点和卸载数量是与卸载技术密切相关的三个主要问题,本文主要从这三个方面来综述和分析目前各个数据流系统所采用的卸载技术。     

数据流系统连接降载研究

连接运算在数据流系统中扮演了关键角色,其执行方式与传统DBMS的连接运算不同,流连接降载策略与传统的网络降载也不相同,已开发许多连接降载策略。论文在介绍流连接运算、数据流及数据流系统的模型后,对连接降载时的系统约束、输出质量目标进行了正式阐述。提出连接降载策略的分类方法,并着重分析了目前一些较为重要的连接降载策略,指出其特征和应用范围。最后总结了好的连接降载策略应具有的特点及未来研究的发展趋势。     

数据流系统降载研究综述

在介绍数据流及数据流系统的模型后,对降载时的系统约束、输出质量目标进行了正式阐述。提出数据流系统降载策略的分类方法 ,着重分析了目前一些较为重要的数据流系统降载策略,指出其特征和应用范围 ,最后总结了好的数据流降载策略应具有的特点以及未来研究的发展趋势。     

挖掘数据流近似频繁项的改进算法

数据流的无限性、连续性和速度快等特点,使得挖掘出所有准确的数据流频繁项通常是不可能的.算法的空间复杂度和时间复杂度通常是评价频繁项挖掘算法优劣的两个主要度量.通过引入局部性原理改进数据流近似频繁项的挖掘算法,该算法的空间复杂性为O（1/ε）,数据流每个数据项的最坏处理时间是O（1/ε）,其最好处理时间是O（1）,输出结果的频率值误差为∑_(i=2)^j（1-μ_i）×k_i。     

基于浮动车数据的快速交通拥堵监控

浮动车技术是近年来智能交通系统中所采用的、获取道路交通信息的先进技术手段之一,可作为大规模实时交通监控的数据源.由于浮动车数据规模庞大,从大量移动对象中有效处理流数据是其中一大难点.采用相似轨迹聚类的思想,结合与拥堵特征相关的交通参数,提出了拥堵同伴发现算法.该算法能从浮动车轨迹流数据中筛选出可能发生拥堵的浮动车数据,从而对拥堵区域变化趋势进行概化预测,由预测结果决定负载处理方式.此外,设计基于预测的多优先级调度算法用以实现整个监控流程.提出的方法可有效降低处理浮动车数据的代价,实现快速交通拥堵监控.通过在城市路网中大规模出租车轨迹数据上的实测,验证了这种算法的有效性和优势.     

数据流滑动窗口聚集查询降载策略研究*

滑动窗口聚集查询在数据流管理系统中应用广泛,数据流到达高峰期,必须考虑滑动窗口聚集查询中出现的降载问题。分析了子集模型的特点和已有降载策略的不足,给出了数据流滑动窗口聚集查询降载问题的约束条件,提出了能保证子集结果产生的基于丢弃窗口更新策略的降载算法。理论分析和实验结果表明,该算法对数据流滑动窗口聚集查询降载问题的处理具有较高的有效性和实用性。     

Finding top-k elements in data streams

Identifying the most frequent elements in a data stream is a well known and difficult problem. Identifying the most frequent elements for each individual, especially in very large populations, is even harder. The use of fast and small memory footprint algorithms is paramount when the number of individuals is very large. In many situations such analysis needs to be performed and kept up to date in near real time. Fortunately, approximate answers are usually adequate when dealing with this problem. This paper presents a new and innovative algorithm that addresses this problem by merging the commonly used counter-based and sketch-based techniques for top-k identification. The algorithm provides the top-k list of elements, their frequency and an error estimate for each frequency value. It also provides strong guarantees on the error estimate, order of elements and inclusion of elements in the list depending on their real frequency. Additionally the algorithm provides stochastic bounds on the error and expected error estimates. Telecommunications customer’s behavior and voice call data is used to present concrete results obtained with this algorithm and to illustrate improvements over previously existing algorithms.     

一种基于数据流模式表示的半懒惰式分类算法

依据从大规模数据中抽取的模式来建立分类模型是模式挖掘的重要研究问题之一。一种可行的方法是根据模式集合建立贝叶斯分类模型。然而,目前基于模式的贝叶斯分类模型大多是针对静态数据集合的,通常不能适应于高速动态变化与无限的数据流环境。对此,提出一种数据流环境下基于模式发现的贝叶斯分类学习模型,其采用半懒惰式学习策略,针对分类实例在不断更新的频繁项集合上建立局部的分类模型;为加快流数据处理的速度,提出了结构更为简单的混合树结构,同时提出了给定项限制的模式抽取机制以减少候选项集的生成;对数据流中模式抽取不完全的情况,使用平滑技术处理未被抽取的项。大量实验分析证明,相较于其他数据流分类器,所提模型具有更高的分类正确率。     

流式处理系统的动态数据分配技术

流式数据处理中,数据倾斜等原因易导致计算节点的负载不均衡,降低系统处理能力。传统的负载均衡方法,比如算子分配、算子迁移和负载脱落等技术因为相对较高的性能代价,在流式处理系统中没有得到广泛的应用。针对流式处理系统的特点,提出一种新的负载均衡方法。在该方法中,计算单元的数据被划分为若干分区,并且数据分区可以在计算单元中动态分配和迁移,在较少干扰系统运行的情况下,通过动态调整各计算单元的分区,平衡各个计算单元的输入流和利用率,以此达到负载平衡的目的。在此基础上,设计并实现了流式处理系统的负载均衡算法和数据在线迁移技术。实验结果表明,该方法能够显著减少数据处理的平均延迟,提高系统吞吐量。     

Sliding window-based frequent pattern mining over data streams

Finding frequent patterns in a continuous stream of transactions is critical for many applications such as retail market data analysis, network monitoring, web usage mining, and stock market prediction. Even though numerous frequent pattern mining algorithms have been developed over the past decade, new solutions for handling stream data are still required due to the continuous, unbounded, and ordered sequence of data elements generated at a rapid rate in a data stream. Therefore, extracting frequent patterns from more recent data can enhance the analysis of stream data. In this paper, we propose an efficient technique to discover the complete set of recent frequent patterns from a high-speed data stream over a sliding window. We develop a Compact Pattern Stream tree (CPS-tree) to capture the recent stream data content and efficiently remove the obsolete, old stream data content. We also introduce the concept of dynamic tree restructuring in our CPS-tree to produce a highly compact frequency-descending tree structure at runtime. The complete set of recent frequent patterns is obtained from the CPS-tree of the current window using an FP-growth mining technique. Extensive experimental analyses show that our CPS-tree is highly efficient in terms of memory and time complexity when finding recent frequent patterns from a high-speed data stream.     

Discovering frequent itemsets over transactional data streams through an efficient and stable approximate approach

A data stream is a massive and unbounded sequence of data elements that are continuously generated at a fast speed. Compared with traditional approaches, data mining in data streams is more challenging since several extra requirements need to be satisfied. In this paper, we propose a mining algorithm for finding frequent itemsets over the transactional data stream. Unlike most of existing algorithms, our method works based on the theory of Approximate Inclusion–Exclusion. Without incrementally maintaining the overall synopsis of the stream, we can approximate the itemsets’ counts according to certain kept information and the counts bounding technique. Some additional techniques are designed and integrated into the algorithm for performance improvement. Besides, the performance of the proposed algorithm is tested and analyzed through a series of experiments.     

一种新的演化文本流聚类算法

数据流的聚类作为聚类的一个分支,已经成为了数据挖掘的研究热点。虽然已经有不少数据流算法出现,但是大部分都是针对低维的数值型数据,很少有高维文本流的研究。本文在传统的数据流聚类框架基础上,提出了一种新的文本微聚类结构体,它更适合文本聚类,同时还将在线微聚类分为潜在微聚类和异常微聚类,提高了对孤立点的适应能力。实验表明该算法相对于其他文本流聚类算法更有效。     

流数据管理的降载技术：研究进展

在当今的网络监控、通信服务、传感网络和金融服务等应用领域中产生了大量的流数据.当流数据的输入速率超过系统的容量,系统性能就会下降.在这种情形下,降载虽然降低了结果的准确性,但可以改善系统的性能.本文讨论了流数据降载的研究现状,对相关技术进行了总结和对比.降载的关键是确定降载的方式和实施的降载计划,本文把降载方式概括为语义降载、随机降载和自适应降载,降载计划包含降载的时间、降载的位置和降栽的量等.讨论了面临的挑战和未来研究的发展方向.     

CPU load shedding for binary stream joins

We present an adaptive load shedding approach for windowed stream joins. In contrast to the conventional approach of dropping tuples from the input streams, we explore the concept ofselective processing for load shedding. We allow stream tuples to be stored in the windows and shed excessive CPU load by performing the join operations, not on the entire set of tuples within the windows, but on a dynamically changing subset of tuples that are learned to be highly beneficial. We support such dynamic selective processing through three forms of runtimeadaptations: adaptation to input stream rates, adaptation to time correlation between the streams and adaptation to join directions. Our load shedding approach enables us to integrateutility-based load shedding withtime correlation-based load shedding. Indexes are used to further speed up the execution of stream joins. Experiments are conducted to evaluate our adaptive load shedding in terms of output rate and utility. The results show that our selective processing approach to load shedding is very effective and significantly outperforms the approach that drops tuples from the input streams. Bugra Gedik received the B.S. degree in C.S. from the Bilkent University, Ankara, Turkey, and the Ph.D. degree in C.S. from the College of Computing at the Georgia Institute of Technology, Atlanta, GA, USA. He is with the IBM Thomas J. Watson Research Center, currently a member of the Software Tools and Techniques Group. Dr. Gedik's research interests lie in data intensive distributed computing systems, spanning data-centric peer-to-peer overlay networks, mobile and sensor-based distributed data management systems, and distributed data stream processing systems. His research focus is on developing system-level architectures and techniques to address scalability problems in distributed continual query systems and applications. He is the recipient of the ICDCS 2003 best paper award. He has served in the program committees of several international conferences, such as ICDE, MDM, and CollaborateCom. Kun-Lung Wu received the B.S. degree in E.E. from the National Taiwan University, Taipei, Taiwan, the M.S. and Ph.D. degrees in C.S. both from the University of Illinois at Urbana-Champaign. He is with the IBM Thomas J. Watson Research Center, currently a member of the Software Tools and Techniques Group. His recent research interests include data streams, continual queries, mobile computing, Internet technologies and applications, database systems and distributed computing. He has published extensively and holds many patents in these areas. Dr. Wu is a Senior Member of the IEEE Computer Society and a member of the ACM. He is the Program Co-Chair for the IEEE Joint Conference on e-Commerce Technology (CEC 2007) and Enterprise Computing, e-Commerce and e-Services (EEE 2007). He was an Associate Editor for the IEEE Trans. on Knowledge and Data Engineering, 2000–2004. He was the general chair for the 3rd International Workshop on E-Commerce and Web-Based Information Systems (WECWIS 2001). He has served as an organizing and program committee member on various conferences. He has received various IBM awards, including IBM Corporate Environmental Affair Excellence Award, Research Division Award, and several Invention Achievement Awards. He received a best paper award from IEEE EEE 2004. He is an IBM Master Inventor. Philip S. Yu received the B.S. Degree in E.E. from National Taiwan University, the M.S. and Ph.D. degrees in E.E. from Stanford University, and the M.B.A. degree from New York University. He is with the IBM Thomas J. Watson Research Center and currently manager of the Software Tools and Techniques group. His research interests include data mining, Internet applications and technologies, database systems, multimedia systems, parallel and distributed processing, and performance modeling. Dr. Yu has published more than 430 papers in refereed journals and conferences. He holds or has applied for more than 250 US patents. Dr. Yu is a Fellow of the ACM and a Fellow of the IEEE. He is associate editors of ACM Transactions on the Internet Technology and ACM Transactions on Knowledge Discovery in Data. He is a member of the IEEE Data Engineering steering committee and is also on the steering committee of IEEE Conference on Data Mining. He was the Editor-in-Chief of IEEE Transactions on Knowledge and Data Engineering (2001–2004), an editor, advisory board member and also a guest co-editor of the special issue on mining of databases. He had also served as an associate editor of Knowledge and Information Systems. In addition to serving as program committee member on various conferences, he will be serving as the general chair of 2006 ACM Conference on Information and Knowledge Management and the program chair of the 2006 joint conferences of the 8th IEEE Conference on E-Commerce Technology (CEC' 06) and the 3rd IEEE Conference on Enterprise Computing, E-Commerce and E-Services (EEE' 06). He was the program chair or co-chairs of the 11th IEEE Intl. Conference on Data Engineering, the 6th Pacific Area Conference on Knowledge Discovery and Data Mining, the 9th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery, the 2nd IEEE Intl. Workshop on Research Issues on Data Engineering: Transaction and Query Processing, the PAKDD Workshop on Knowledge Discovery from Advanced Databases, and the 2nd IEEE Intl. Workshop on Advanced Issues of E-Commerce and Web-based Information Systems. He served as the general chair of the 14th IEEE Intl. Conference on Data Engineering and the general co-chair of the 2nd IEEE Intl. Conference on Data Mining. He has received several IBM honors including 2 IBM Outstanding Innovation Awards, an Outstanding Technical Achievement Award, 2 Research Division Awards and the 84th plateau of Invention Achievement Awards. He received an Outstanding Contributions Award from IEEE Intl. Conference on Data Mining in 2003 and also an IEEE Region 1 Award for “promoting and perpetuating numerous new electrical engineering concepts” in 1999. Dr. Yu is an IBM Master Inventor. Ling Liu is an associate professor at the College of Computing at Georgia Tech. There, she directs the research programs in Distributed Data Intensive Systems Lab (DiSL), examining research issues and technical challenges in building large scale distributed computing systems that can grow without limits. Dr. Liu and the DiSL research group have been working on various aspects of distributed data intensive systems, ranging from decentralized overlay networks, exemplified by peer to peer computing, data grid computing, to mobile computing systems and location based services, sensor network computing, and enterprise computing systems. She has published over 150 international journal and conference articles. Her research group has produced a number of software systems that are either open sources or directly accessible online, among which the most popular ones are WebCQ and XWRAPElite. Dr. Liu is currently on the editorial board of several international journals, including IEEE Transactions on Knowledge and Data Engineering, International Journal of Very large Database systems (VLDBJ), International Journal of Web Services Research, and has chaired a number of conferences as a PC chair, a vice PC chair, or a general chair, including IEEE International Conference on Data Engineering (ICDE 2004, ICDE 2006, ICDE 2007), IEEE International Conference on Distributed Computing (ICDCS 2006), IEEE International Conference on Web Services (ICWS 2004). She is a recipient of IBM Faculty Award (2003, 2006). Dr. Liu's current research is partly sponsored by grants from NSF CISE CSR, ITR, CyberTrust, a grant from AFOSR, an IBM SUR grant, and an IBM faculty award.     

一种新的基于AR* -GRNN的QoS降载管理框架

在过载环境下不是很高的降载比率很可能要丢失要分析的全部信息,故有效地获取流数据的信息是流数据挖掘的关键问题.本文建立基于AR* -GRNN的QoS降载管理框架,提高过载环境下流数据的相似性查询效率.利用离散傅立叶变换(Discrete Fourier Transform,DFT)来获取数据的特征值,运用混合预测模型(时间序列类模型(AR*)-Generalized RegressionNeural Network,AR* -GRNN)和历史的特征值来预测降载部分数据的特征值,建立自适应的降载计划,从而完成降载发生情况下的相似性查询分析.