Identifying frequent items in distributed data sets

Many practical problems in computer science require the knowledge of the most frequently occurring items in a data set. Current state-of-the-art algorithms for frequent items discovery are either fully centralized or rely on node hierarchies which are inflexible and prone to failures in massively distributed systems. In this paper we describe a family of gossip-based algorithms that efficiently approximate the most frequent items in large-scale distributed datasets. We show, both analytically and using real-world datasets, that our algorithms are fast, highly scalable, and resilient to node failures.     

分布式流数据频繁项发现算法的研究

对分布式流数据中频繁项的发现算法进行了研究,利用一种新颖的分布式概要算法(DSA)来发现从叶子节点直至根节点的概要结构,通过在不同的分布状态下设置相应的精确梯度来最小化通信负载,并利用真实数据集验证了该结构和算法的有效性。     

一种实时挖掘数据流近似频繁项的算法

数据流的无限性和流动性使得传统的频繁项挖掘算法难以适用.针对数据流的特点,提出了一种实时的挖掘数据流近似频繁项的算法.在允许的偏差范围内,新算法只需扫描一次数据项,使用的存储空间远远小于数据流的规模,能动态地挖掘数据流中的所有频繁项.将数据项存储到一种新的数据结构中,利用该数据结构可以快速地删除非频繁项.最后,理论分析和实验表明这种方法的有效性.     

流数据上的频繁项挖掘算法

提出了一种流数据上的频繁项挖掘算法（SW-COUNT）。该算法通过数据采样技术挖掘滑动窗口下的数据流频繁项。给定的误差ε,SW-COUNT可以在O(ε-1)空间复杂度下,检测误差在εn内的数据流频繁项,对每个数据项的平均处理时间为O(1)。大量的实验证明,该算法比其他类似算法具有较好的精度质量以及时间和空间效率。     

一种分布式全局频繁项集挖掘方法

提出一种基于频繁模式树与最大频繁项集的分布式全局频繁项集挖掘算法BFM-MGFIS,该算法引入子集枚举树以实现有序挖掘与全局剪枝策略,有效地减小了候选数据集且提高了并行性,实验表明本文提出的算法是有效可行的。     

Entity resolution for distributed probabilistic data

The problem of entity resolution over probabilistic data (ERPD) arises in many applications that have to deal with probabilistic data. In many of these applications, probabilistic data is distributed among a number of nodes. The simple, centralized approach to the ERPD problem does not scale well as large amounts of data need to be sent to a central node. In this paper, we present FD (Fully Distributed), a decentralized algorithm for dealing with the ERPD problem over distributed data, with the goal of minimizing bandwidth usage and reducing processing time. FD is completely distributed and does not depend on the existence of certain nodes. We validated FD through implementation over a 75-node cluster and simulation using the PeerSim simulator. We used both synthetic and real-world data in our experiments. Our performance evaluation shows that FD can achieve major performance gains in terms of bandwidth usage and response time.     

Methods for finding frequent items in data streams

The frequent items problem is to process a stream of items and find all items occurring more than a given fraction of the time. It is one of the most heavily studied problems in data stream mining, dating back to the 1980s. Many applications rely directly or indirectly on finding the frequent items, and implementations are in use in large scale industrial systems. However, there has not been much comparison of the different methods under uniform experimental conditions. It is common to find papers touching on this topic in which important related work is mischaracterized, overlooked, or reinvented. In this paper, we aim to present the most important algorithms for this problem in a common framework. We have created baseline implementations of the algorithms and used these to perform a thorough experimental study of their properties. We give empirical evidence that there is considerable variation in the performance of frequent items algorithms. The best methods can be implemented to find frequent items with high accuracy using only tens of kilobytes of memory, at rates of millions of items per second on cheap modern hardware.     

Mining frequent itemsets over distributed data streams by continuously maintaining a global synopsis

Mining frequent itemsets over data streams has attracted much research attention in recent years. In the past, we had developed a hash-based approach for mining frequent itemsets over a single data stream. In this paper, we extend that approach to mine global frequent itemsets from a collection of data streams distributed at distinct remote sites. To speed up the mining process, we make the first attempt to address a new problem on continuously maintaining a global synopsis for the union of all the distributed streams. The mining results therefore can be yielded on demand by directly processing the maintained global synopsis. Instead of collecting and processing all the data in a central server, which may waste the computation resources of remote sites, distributed computations over the data streams are performed. A distributed computation framework is proposed in this paper, including two communication strategies and one merging operation. These communication strategies are designed according to an accuracy guarantee of the mining results, determining when and what the remote sites should transmit to the central server (named coordinator). On the other hand, the merging operation is exploited to merge the information received from the remote sites into the global synopsis maintained at the coordinator. By the strategies and operation, the goal of continuously maintaining the global synopsis can be achieved. Rooted in the continuously maintained global synopsis, we propose a mining algorithm for finding global frequent itemsets. Moreover, the correctness guarantees of the communication strategies and merging operation, and the accuracy guarantee analysis of the mining algorithm are provided. Finally, a series of experiments on synthetic datasets and a real dataset are performed to show the effectiveness and efficiency of the distributed computation framework.     

Finding frequent items over sliding windows with constant update time

In this paper, we consider the problem of finding ?-approximate frequent items over a sliding window of size N. A recent work by Lee and Ting (2006) [7] solves the problem by giving an algorithm that supports query and update time, and uses space. Their query time and memory usage are essentially optimal, but the update time is not. We give a new algorithm that supports O(1) update time with high probability while maintaining the query time and memory usage as .     

Mining frequent subgraphs over uncertain graph databases under probabilistic semantics

Frequent subgraph mining has been extensively studied on certain graph data. However, uncertainty is intrinsic in graph data in practice, but there is very few work on mining uncertain graph data. This paper focuses on mining frequent subgraphs over uncertain graph data under the probabilistic semantics. Specifically, a measure called ${\varphi}$ -frequent probability is introduced to evaluate the degree of recurrence of subgraphs. Given a set of uncertain graphs and two real numbers ${0 < \varphi, \tau < 1}$ , the goal is to quickly find all subgraphs with ${\varphi}$ -frequent probability at least τ. Due to the NP-hardness of the problem and to the #P-hardness of computing the ${\varphi}$ -frequent probability of a subgraph, an approximate mining algorithm is proposed to produce an ${(\varepsilon, \delta)}$ -approximate set Π of “frequent subgraphs”, where ${0 < \varepsilon < \tau}$ is error tolerance, and 0 <?δ?< 1 is a confidence bound. The algorithm guarantees that (1) any frequent subgraph S is contained in Π with probability at least ((1 ? δ) /2) ^s , where s is the number of edges in S; (2) any infrequent subgraph with ${\varphi}$ -frequent probability less than ${\tau - \varepsilon}$ is contained in Π with probability at most δ/2. The theoretical analysis shows that to obtain any frequent subgraph with probability at least 1 ? Δ, the input parameter δ of the algorithm must be set to at most ${1 - 2 (1 - \Delta)^{1 / \ell_{\max}}}$ , where 0 <?Δ <?1, and ? _max is the maximum number of edges in frequent subgraphs. Extensive experiments on real uncertain graph data verify that the proposed algorithm is practically efficient and has very high approximation quality. Moreover, the difference between the probabilistic semantics and the expected semantics on mining frequent subgraphs over uncertain graph data has been discussed in this paper for the first time.     

Mining constrained frequent itemsets from distributed uncertain data

Nowadays, high volumes of massive data can be generated from various sources (e.g., sensor data from environmental surveillance). Many existing distributed frequent itemset mining algorithms do not allow users to express the itemsets to be mined according to their intention via the use of constraints. Consequently, these unconstrained mining algorithms can yield numerous itemsets that are not interesting to users. Moreover, due to inherited measurement inaccuracies and/or network latencies, the data are often riddled with uncertainty. These call for both constrained mining and uncertain data mining. In this journal article, we propose a data-intensive computer system for tree-based mining of frequent itemsets that satisfy user-defined constraints from a distributed environment such as a wireless sensor network of uncertain data.     

Joint probabilistic data association in distributed sensor networks

A distributed multitarget tracking problem is considered. The joint probabilistic data association (JPDA) algorithm, which has been successfully used for tracking multiple targets in a cluttered environment, assumes a centralized processing architecture. It assumes that measurements are transmitted to a central site and processed. In some applications, however, it may be desirable for the sensor measurements to be processed at or near the sensors instead of transmitting them to the central processor. The local processed results are then sent over a communication network to be used by other processors. This paper presents a distributed version of the JPDA algorithm which is applicable under such a situation.     

有效的不确定数据概率频繁项集挖掘算法

针对已有概率频繁项集挖掘算法采用模式增长的方式构建树时产生大量树节点,导致内存空间占用较大以及发现概率频繁项集效率低等问题,提出了改进的不确定数据频繁模式增长(PUFP-Growth)算法。该算法通过逐条读取不确定事务数据库中数据,构造类似频繁模式树(FP-Tree)的紧凑树结构,同时更新项头表中保存所有尾节点相同项集的期望值的动态数组。当所有事务数据插入到改进的不确定数据频繁模式树(PUFP-Tree)中以后,通过遍历数组得到所有的概率频繁项集。最后通过实验结果和理论分析表明:PUFP-Growth算法可以有效地发现概率频繁项集;与不确定数据频繁模式增长(UF-Growth)算法和压缩的不确定频繁模式挖掘(CUFP-Mine)算法相比,提出的PUFP-Growth算法能够提高不确定数据概率频繁项集挖掘的效率,并且减少了内存空间的使用。     

Methods for mining frequent items in data streams: an overview

In many real-world applications, information such as web click data, stock ticker data, sensor network data, phone call records, and traffic monitoring data appear in the form of data streams. Online monitoring of data streams has emerged as an important research undertaking. Estimating the frequency of the items on these streams is an important aggregation and summary technique for both stream mining and data management systems with a broad range of applications. This paper reviews the state-of-the-art progress on methods of identifying frequent items from data streams. It describes different kinds of models for frequent items mining task. For general models such as cash register and Turnstile, we classify existing algorithms into sampling-based, counting-based, and hashing-based categories. The processing techniques and data synopsis structure of each algorithm are described and compared by evaluation measures. Accordingly, as an extension of the general data stream model, four more specific models including time-sensitive model, distributed model, hierarchical and multi-dimensional model, and skewed data model are introduced. The characteristics and limitations of the algorithms of each model are presented, and open issues waiting for study and improvement are discussed.     

Finding frequent itemsets over online data streams

Conventional data mining methods for finding frequent itemsets require considerable computing time to produce their results from a large data set. Due to this reason, it is almost impossible to apply them to an analysis task in an online data stream where a new transaction is continuously generated at a rapid rate. An algorithm for finding frequent itemsets over an online data stream should support flexible trade-off between processing time and mining accuracy. Furthermore, the most up-to-date resulting set of frequent itemsets should be available quickly at any moment. To satisfy these requirements, this paper proposes a data mining method for finding frequent itemsets over an online data stream. The proposed method examines each transaction one-by-one without any candidate generation process. The count of an itemset that appears in each transaction is monitored by a lexicographic tree resided in main memory. The current set of monitored itemsets in an online data stream is minimized by two major operations: delayed-insertion and pruning. The former is delaying the insertion of a new itemset in recent transactions until the itemset becomes significant enough to be monitored. The latter is pruning a monitored itemset when the itemset turns out to be insignificant. The number of monitored itemsets can be flexibly controlled by the thresholds of these two operations. As the number of monitored itemsets is decreased, frequent itemsets in the online data stream are more rapidly traced while they are less accurate. The performance of the proposed method is analyzed through a series of experiments in order to identify its various characteristics.     

基于数据流频繁闭合模式的挖掘

用挖掘频繁闭合模式集代替挖掘频繁模式集是近年来提出的一个重要策略。根据数据流的特点,提出了一种基于滑动窗口的频繁闭合模式的新方法DSFC_Mine。该算法以滑动窗口中的基本窗口为更新单位,利用改进的CHARM算法计算每个基本窗口的潜在频繁闭合项集,将它们存储到一种新的数据结构中,利用该数据结构可以快速地挖掘滑动窗口中的所有频繁闭合项集。实验验证了该算法在时间上和空间上的可行性和有效性。     

Tracking in a cluttered environment with probabilistic data association

This paper presents a new approach to the problem of tracking when the source of the measurement data is uncertain. It is assumed that one object of interest (‘target’) is in track and a number of undesired returns are detected and resolved at a certain time in the neighbourhood of the predicted location of the target's return. A suboptimal estimation procedure that takes into account all the measurements that might have originated from the object in track but does not have growing memory and computational requirements is presented. The probability of each return (lying in a certain neighborhood of the predicted return, called ‘validation region’) being correct is obtained—this is called ‘probabilistic data association’ (PDA). The undesired returns are assumed uniformly and independently distributed. The estimation is done by using the PDA method with an appropriately modified tracking filter, called PDAF. Since the computational requirements of the PDAF are only slightly higher than those of the standard filter, the method can be useful for real-time systems. Simulation results obtained for tracking an object in a cluttered environment show the PDAF to give significantly better results than the standard filter currently in use for this type of problem.     

分布式数据流上的Skyline计算

为了降低分布式数据流上的连续Skyline计算过程中的通信开销,提出了基于远程过滤的思想并对相关理论基础进行了证明,描述了系统的体系结构并提出了两个过滤模型v_Max和Distance。理论分析和实验结果证明了所提方法在某些数据分布情况下降低通信开销的有效性。     

分布式环境下保持隐私的聚类挖掘算法

隐私保护是数据挖掘中一个重要的研究方向。针对如何在不共享精确数据的条件下,应用k-平均聚类算法从数据中发现有意义知识的问题,提出了一种基于安全多方计算的算法。算法利用半可信第三方参与下的安全求平均值协议,实现了在分布式数据中进行k-平均聚类挖掘时隐私保护的要求。实验表明算法能很好的隐藏数据,保护隐私信息,且对聚类的结果没有影响。