面向数据流的频繁模式挖掘研究*

数据流的无限性、高速性使得经典的频繁模式挖掘方法难以适用到数据流中。针对数据流的特点,对数据流中频繁模式挖掘问题进行了研究,提出了数据流频繁模式挖掘算法FP-SegCount。该算法将数据流分段并利用改进的FP-growth算法挖掘分段中的频繁项集,然后利用Count-Min Sketch进行项集计数。算法解决了压缩统计和计算快速高效的问题。通过实验分析,FP-SegCount算法是有效的。     

界标窗口数据流频繁模式挖掘特性

随着数据流应用领域的不断扩大,数据流频繁模式挖掘技术逐渐成为数据挖掘领域研究的核心问题。对DSFPM算法进行研究和改进,提出了一种基于界标窗口的数据流频繁模式挖掘算法DSMFP_LW。该算法实现了单边扫描数据流;利用扩展的前缀模式树存储全局临界频繁模式,实现数据增量更新。通过对比实验,结果证明DSMFP_LW算法有较好的时间开销和空间利用率,优于经典的Lossy Counting算法,适合数据流频繁模式挖掘。     

数据流频繁模式挖掘算法设计

介绍了数据流频繁模式的概念和定义,提出了数据流频繁模式挖掘算法的通用数据流处理模型,详细总结了数据流频繁模式挖掘算法的三种分类方式:"窗口模型"、"结果集类型"和"结果集精确性".基于这些分类方法提出了数据流频繁模式挖掘算法的设计立方体,该立方体不仅涵盖了现有的数据流频繁模式挖掘算法,还对设计新的算法具有指导意义.基于设计立方体,分析了设计算法时应当采取的有效策略,旨在为设计新算法提供一个有力参考.最后讨论了数据流频繁模式挖掘的进一步研究工作.     

面向数据流的频繁项集挖掘研究

针对数据流的特点,对数据流中频繁模式挖掘问题进行了研究,提出了数据流频繁项集挖掘算法FP-SegCount。该算法将数据流分段并利用改进的FP-growth算法挖掘分段中的频繁项集。然后,利用Count Min Sketch进行项集计数。算法解决了压缩统计和计算快速高效的问题。通过和FP-DS算法的实验对比,FP-SegCount算法具有较好的时间效率。     

数据流频繁模式挖掘综述

一些先进应用如欺诈检测和趋势学习等带来了数据流频繁模式挖掘的发展。不同于静态数据,数据流挖掘面临着时空约束和项集组合爆炸等问题。对已有数据流频繁模式挖掘算法进行综述并对经典和最新算法进行分析。按照模式集合的完整程度进行分类,数据流中频繁模式分为全集模式和压缩模式。压缩模式主要包括闭合模式、最大模式、top-k模式以及三者的组合模式。不同之处是闭合模式是无损压缩的,而其他模式是有损压缩的。为了得到有趣的频繁模式,可以挖掘基于用户约束的模式。为了处理数据流中的新近事务,将算法分为基于窗口模型和基于衰减模型的方法。数据流中模式挖掘常见的还包含序列模式和高效用模式,对经典和最新算法进行介绍。最后给出了数据流模式挖掘的下一步工作。     

基于时间衰减模型的数据流频繁模式挖掘

频繁模式挖掘是数据流挖掘中的重要研究课题. 针对数据流的时效性和流中心的偏移性特点, 提出了界标窗口模型与时间衰减模型相结合的数据流频繁模式挖掘算法. 该算法通过动态构建全局模式树, 利用时间指数衰减函数对模式树中各模式的支持数进行统计, 以此刻画界标窗口内模式的频繁程度; 进而, 为有效降低空间开销, 设计了剪枝阈值函数, 用于对预期难以成长为频繁的模式及时从全局树中剪除. 本文对出现在算法中的重要参数和阈值进行了深入分析. 一系列实验表明, 与现有同类算法MSW相比, 该算法挖掘精度高(平均超过90%), 内存开销小, 速度上可以满足高速数据流的处理要求, 且可以适应不同事务数量、不同事务平均长度和不同最大潜在频繁模式平均长度的数据流频繁模式挖掘.     

数据流中的频繁模式挖掘

近年来,数据流挖掘越来越引起研究人员的关注,已逐渐成为许多领域有用的工具。如何利用有限的存储空间高效地挖掘出频繁模式已成为数据流挖掘的基本问题,具有很强的现实意义和理论价值。在论述数据流管理系统模型的基础上,深入分析了国内外的各种频繁模式挖掘算法,并指出这些算法的特点及其局限性。最后对未来的研究方向进行了展望。     

一种面向高速度数据流的频繁模式挖掘算法

数据流频繁模式挖掘是从实时、连续、有序的数据序列中寻找频繁模式的过程,以往的相关研究通常将该过程分为两个阶段:首先监测数据流中各模式的频率,由于数据流环境对空间与时间的限制,需要对监测模式进行剪裁,因而频率的计算和剪裁需要重复进行;当用户提交查询时,从监控的模式中筛选出满足要求的输出.现有研究都注重解决如何对观测对象进行剪裁,而事实上在计算模式频率时,数据项集中不同数据项间的组合使得频率计算非常耗时.因此,对于高速数据流,算法通常没有足够的时间来处理数据流中的每个事务,这会影响挖掘结果的正确性.针对这一问题提出了一种新的面向高速数据流的频繁模式挖掘算法Delay. 在Delay算法中将模式频率的统计延迟到第2阶段进行,第1阶段只记录"必要信息",这样大大提高了算法所能处理的数据流流动速度的上限.实验结果表明,算法在效率上优于已有算法,LossyCounting和FDPM,尤其是在处理长数据项集数据流时优势更为明显.     

挖掘数据流频繁模式的相关技术和算法研究综述

数据流本身的特点使得静态挖掘方法不再满足要求。国内外学者已提出许多新的挖掘数据流频繁模式的方法和技术。对这些技术和算法进行了综述。首先介绍数据流的概念和特点,分析国内外的研究现状,总结了数据流中挖掘频繁模式的特点,并列出挖掘方法的常用技术和基于这些技术的代表性算法,最后讨论了将来的研究方向。     

数据流中的频繁模式挖掘

近年来，数据流挖掘越来越引起研究人员的关注，已逐渐成为许多领域有用的工具。如何利用有限的存储空间高效地挖掘出频繁模式已成为数据流挖掘的基本问题，具有很强的现实意义和理论价值。在论述数据流管理系统模型的基础上，深入分析了国内外的各种频繁模式挖掘算法，并指出这些算法的特点及其局限性。最后对未来的研究方向进行了展望。     

High utility pattern mining over data streams with sliding window technique

Processing changeable data streams in real time is one of the most important issues in the data mining field due to its broad applications such as retail market analysis, wireless sensor networks, and stock market prediction. In addition, it is an interesting and challenging problem to deal with the stream data since not only the data have unbounded, continuous, and high speed characteristics but also their environments have limited resources. High utility pattern mining, meanwhile, is one of the essential research topics in pattern mining to overcome major drawbacks of the traditional framework for frequent pattern mining that takes only binary databases and identical item importance into consideration. This approach conducts mining processes by reflecting characteristics of real world databases, non-binary quantities and relative importance of items. Although relevant algorithms were proposed for finding high utility patterns in stream environments, they suffer from a level-wise candidate generation-and-test and a large number of candidates by their overestimation techniques. As a result, they consume a huge amount of execution time, which is a significant performance issue since a rapid process is necessary in stream data analysis. In this paper, we propose an algorithm for mining high utility patterns from resource-limited environments through efficient processing of data streams in order to solve the problems of the overestimation-based methods. To improve mining performance with fewer candidates and search space than the previous ones, we develop two techniques for reducing overestimated utilities. Moreover, we suggest a tree-based data structure to maintain information of stream data and high utility patterns. The proposed tree is restructured by our updating method with decreased overestimation utilities to keep up-to-date stream information whenever the current window slides. Our approach also has an important effect on expert and intelligent systems in that it can provide users with more meaningful information than traditional analysis methods by reflecting the characteristics of real world non-binary databases in stream environments and emphasizing on recent data. Comprehensive experimental results show that our algorithm outperforms the existing sliding window-based one in terms of runtime efficiency and scalability.     

Sliding window-based frequent pattern mining over data streams

Finding frequent patterns in a continuous stream of transactions is critical for many applications such as retail market data analysis, network monitoring, web usage mining, and stock market prediction. Even though numerous frequent pattern mining algorithms have been developed over the past decade, new solutions for handling stream data are still required due to the continuous, unbounded, and ordered sequence of data elements generated at a rapid rate in a data stream. Therefore, extracting frequent patterns from more recent data can enhance the analysis of stream data. In this paper, we propose an efficient technique to discover the complete set of recent frequent patterns from a high-speed data stream over a sliding window. We develop a Compact Pattern Stream tree (CPS-tree) to capture the recent stream data content and efficiently remove the obsolete, old stream data content. We also introduce the concept of dynamic tree restructuring in our CPS-tree to produce a highly compact frequency-descending tree structure at runtime. The complete set of recent frequent patterns is obtained from the CPS-tree of the current window using an FP-growth mining technique. Extensive experimental analyses show that our CPS-tree is highly efficient in terms of memory and time complexity when finding recent frequent patterns from a high-speed data stream.     

数据流高效用模式挖掘综述

数据流高效用模式挖掘方法是以二进制的频繁模式挖掘方法为前提,引入项的内部效用和外部效用,在模式挖掘过程中可以考虑项的重要性,从而挖掘更有价值的模式。从关键窗口技术、常用方法、表示形式等角度对数据流高效用模式挖掘方法进行分析并总结其相关算法,从而研究其特点、优势、劣势以及其关键问题所在。具体来说,说明了数据流高效用模式常用的概念;对处理数据流高效用模式的关键窗口技术进行了分析,涉及到滑动、衰减、界标和倾斜窗口模型;研究了一阶段和两阶段的数据流高效用模式挖掘方法;分析了高效用模式的表示形式,即完全高效用模式和压缩高效用模式;介绍了其他的数据流高效用模式,包括序列高效用模式、混合高效用模式以及高平均效用模式等;最后展望了数据流高效用模式挖掘的进一步研究方向。     

Frequent pattern mining: current status and future directions

Frequent pattern mining has been a focused theme in data mining research for over a decade. Abundant literature has been dedicated to this research and tremendous progress has been made, ranging from efficient and scalable algorithms for frequent itemset mining in transaction databases to numerous research frontiers, such as sequential pattern mining, structured pattern mining, correlation mining, associative classification, and frequent pattern-based clustering, as well as their broad applications. In this article, we provide a brief overview of the current status of frequent pattern mining and discuss a few promising research directions. We believe that frequent pattern mining research has substantially broadened the scope of data analysis and will have deep impact on data mining methodologies and applications in the long run. However, there are still some challenging research issues that need to be solved before frequent pattern mining can claim a cornerstone approach in data mining applications. The work was supported in part by the U.S. National Science Foundation NSF IIS-05-13678/06-42771 and NSF BDI-05-15813. Any opinions, findings, and conclusions or recommendations expressed here are those of the authors and do not necessarily reflect the views of the funding agencies.     

面向事件流的频繁片断计数算法

在事件流上挖掘频繁片断已经成为近来研究的热点,在很多应用中起到重要作用。以往的研究提出了一些挖掘算法,包括基于滑动窗口和基于非重叠出现的方法。然而,这些算法在处理基于片断互异出现的支持度计数时,效率很低甚至无效。为此,提出了一种包含状态计数的有限状态自动机模型,并使用该模型给出了一种高效挖掘算法。从理论上对算法的效率和有效性进行了分析;实验结果证明了算法是有效且高效的。     

Interactive mining of high utility patterns over data streams

High utility pattern (HUP) mining over data streams has become a challenging research issue in data mining. When a data stream flows through, the old information may not be interesting in the current time period. Therefore, incremental HUP mining is necessary over data streams. Even though some methods have been proposed to discover recent HUPs by using a sliding window, they suffer from the level-wise candidate generation-and-test problem. Hence, they need a large amount of execution time and memory. Moreover, their data structures are not suitable for interactive mining. To solve these problems of the existing algorithms, in this paper, we propose a novel tree structure, called HUS-tree (high utility stream tree) and a new algorithm, called HUPMS (high utility pattern mining over stream data) for incremental and interactive HUP mining over data streams with a sliding window. By capturing the important information of stream data into an HUS-tree, our HUPMS algorithm can mine all the HUPs in the current window with a pattern growth approach. Furthermore, HUS-tree is very efficient for interactive mining. Extensive performance analyses show that our algorithm is very efficient for incremental and interactive HUP mining over data streams and significantly outperforms the existing sliding window-based HUP mining algorithms.     

Finding hot query patterns over an XQuery stream

Caching query results is one efficient approach to improving the performance of XML management systems. This entails the discovery of frequent XML queries issued by users. In this paper, we model user queries as a stream of XML query pattern trees and mine the frequent query patterns over the query stream. To facilitate the one-pass mining process, we devise a novel data structure called DTS to summarize the pattern trees seen so far. By grouping the incoming pattern trees into batches, we can dynamically mark the active portion of the current batch in DTS and limit the enumeration of candidate trees to only the currently active pattern trees. We also design another summary data structure called ECTree that provides for the incremental computation of the frequent tree patterns over the query stream. Based on the above two constructs, we present two mining algorithms called XQSMinerI and XQSMinerII. XQSMinerI is fast, but it tends to overestimate, while XQSMinerII adopts a filter-and-refine approach to minimize the amount of overestimation. Experimental results show that the proposed methods are both efficient and scalable and require only small memory footprints.Received: 17 October 2003, Accepted: 16 April 2004, Published online: 14 September 2004Edited by: J. Gehrke and J. Hellerstein.