An efficient algorithm for mining top-<Emphasis Type="Italic">k</Emphasis> on-shelf high utility itemsets

High on-shelf utility itemset (HOU) mining is an emerging data mining task which consists of discovering sets of items generating a high profit in transaction databases. The task of HOU mining is more difficult than traditional high utility itemset (HUI) mining, because it also considers the shelf time of items, and items having negative unit profits. HOU mining can be used to discover more useful and interesting patterns in real-life applications than traditional HUI mining. Several algorithms have been proposed for this task. However, a major drawback of these algorithms is that it is difficult for users to find a suitable value for the minimum utility threshold parameter. If the threshold is set too high, not enough patterns are found. And if the threshold is set too low, too many patterns will be found and the algorithm may use an excessive amount of time and memory. To address this issue, we propose to address the problem of top-k on-shelf high utility itemset mining, where the user directly specifies k, the desired number of patterns to be output instead of specifying a minimum utility threshold value. An efficient algorithm named KOSHU (fast top-K on-shelf high utility itemset miner) is proposed to mine the top-k HOUs efficiently, while considering on-shelf time periods of items, and items having positive and/or negative unit profits. KOSHU introduces three novel strategies, named efficient estimated co-occurrence maximum period rate pruning, period utility pruning and concurrence existing of a pair 2-itemset pruning to reduce the search space. KOSHU also incorporates several novel optimizations and a faster method for constructing utility-lists. An extensive performance study on real-life and synthetic datasets shows that the proposed algorithm is efficient both in terms of runtime and memory consumption and has excellent scalability.     

关联规则的冗余删除与聚类

关联规则挖掘常常会产生大量的规则，这使得用户分析和利用这些规则变得十分困难，尤其是数据库中属性高度相关时，问题更为突出．为了帮助用户做探索式分析，可以采用各种技术来有效地减少规则数量，如约束性关联规则挖掘、对规则进行聚类或泛化等技术．本文提出一种关联规则冗余删除算法ADRR和一种关联规则聚类算法ACAR．根据集合具有的性质，证明在挖掘到的关联规则中存在大量可以删除的冗余规则，从而提出了算法ADRR；算法ACAR采用一种新的用项目间的相关性来定义规则间距离的方法，结合DBSCAN算法的思想对关联规则进行聚类．最后将本文提出的算法加以实现，实验结果表明该算法暑有数可行的．且具较高的效率。     

应用于入侵检测系统的报警关联的改进Apriori算法

在众多的关联规则挖掘算法中,Apriori算法是最为经典的一个,但Apriori算法有以下缺陷：需要扫描多次数据库、生成大量候选集以及迭代求解频繁项集。提出了一种一步交集操作得到最大频繁项目集的方法。支持度由交集的次数得到而无需再去扫描事务数据库,将其中一些属性进行编号能减少存储空间且方便搜索候选集列表,从而提高算法的效率。最后针对入侵检测系统形成关联规则。实验结果表明,优化后的算法能有效地提高关联规则挖掘的效率。     

基于分布数据库的快速关联规则挖掘算法

关联规则发现是数据挖掘的重要研究内容,随着数据库中数据的不断增加,大数据集环境下的关联规则发现日益受到重视,分布式关联规则发现是解决这一问题的有效方法。分布式数据库环境下的关联规则挖掘算法中,时间开销主要体现在两方面(:1)频繁项目集的确定;(2)网络的通讯量。为了解决第一个问题,文章提出了一种基于二进制形式的候选频繁项目集生成和相应的计算支持数算法,该算法只需对挖掘对象进行一些”或”、”与”、”异或”等逻辑运算操作,显著降低了算法的实现难度。将该算法与DMA算法相结合提出改进算法FDMA。理论分析和实验结果表明,算法FDMA大大提高了关联规则挖掘的效率,算法是有效可行的。     

Using a hash-based method with transaction trimming for miningassociation rules

We examine the issue of mining association rules among items in a large database of sales transactions. Mining association rules means that, given a database of sales transactions, to discover all associations among items such that the presence of some items in a transaction will imply the presence of other items in the same transaction. The mining of association rules can be mapped into the problem of discovering large itemsets where a large itemset is a group of items that appear in a sufficient number of transactions. The problem of discovering large itemsets can be solved by constructing a candidate set of itemsets first, and then, identifying, within this candidate set, these itemsets that meet the large itemset requirement. Generally, this is done iteratively for each large k-itemset in increasing order of k, where a large k-itemset is a large itemset with k items. To determine large itemsets from a huge number of candidate sets in early iterations is usually the dominating factor for the overall data mining performance. To address this issue, we develop an effective algorithm for the candidate set generation. It is a hash-based algorithm and is especially effective for the generation of a candidate set for large 2-itemsets. Explicitly, the number of candidate 2-itemsets generated by the proposed algorithm is, in orders of magnitude, smaller than that by previous methods, thus resolving the performance bottleneck. Note that the generation of smaller candidate sets enables us to effectively trim the transaction database size at a much earlier stage of the iterations, thereby reducing the computational cost for later iterations significantly. The advantage of the proposed algorithm also provides us the opportunity of reducing the amount of disk I/O required. An extensive simulation study is conducted to evaluate performance of the proposed algorithm     

一种矩阵和排序索引关联规则数据挖掘算法

在关联规则挖掘算法中,Apriori由于多次对数据库进行扫描会产生较多的候选集,在多次扫描数据库的情况下容易产生I/O开销问题,并引起数据挖掘效率低.矩阵关联规则在数据挖掘过程中没有删除非频繁项集,致使存在较多的无效扫描,对于挖掘效率的提高也不明显.该文提出了一种改进的矩阵和排序索引关联规则数据挖掘算法,首先,删除不需...     

基于存储改进的分区并行关联规则挖掘算法

针对现有算法存储结构简单、生成大量冗余的候选集、时间和空间复杂度高,挖掘效率不理想的情况,为了进一步提高关联规则算法挖掘频繁集的速度,优化算法的执行性能,提出基于内存结构改进的关联规则挖掘算法。该算法基于Spark分布式框架,分区并行挖掘出频繁集,提出在挖掘过程中利用布隆过滤器进行项目存储,并对事务集和候选集进行精简化操作,进而达到优化挖掘频繁集的速度、节省计算资源的目的。算法在占用较少内存的条件下,相比于YAFIM和MR-Apriori算法,在挖掘频繁集效率上有明显的提升,不但能较好地提升挖掘速度,降低内存的压力,而且具有很好的可扩展性,使得算法可以应用到更大规模的数据集和集群,从而达到优化算法性能的目的。     

分组多支持度关联规则研究

关联规则是数据挖掘的重要任务之一,传统关联规则算法只有一个最小支持度,假设项出现的频率大致相同,而在谮实际中并非如此,由此产生了多支持度关联规则问题.该问题针对每个项给定不同的支持度,而在实际应用中项可以划分成若干个组,每组有一个支持度.由此提出了分组多支持度关联规则问题,针对该问题给出了基于多支持度性质对项进行分组的方法.该方法可以降低2-项候选集的数目.在此基础上,进一步给出了相应的多支持度关联规则发现算法,并通过实验证明了算法的有效性.     

基于层次梯度分析的协同数据挖掘算法

传统的关联规则挖掘算法易形成大量频繁项目集,不适用于异构环境下海量交通数据的挖掘。为此,提出基于层次梯度且无候选项分析的协同数据挖掘算法。采用挖掘主题数据库和层次梯度构建层次业务数据库,逐层深度挖掘局部频繁项。利用弱化熵模型对频繁项主题数据库进行数据分析,并产生关联规则。实验结果表明,该算法适用于无候选项支持的协同挖掘。     

Mining the optimal class association rule set

We define an optimal class association rule set to be the minimum rule set with the same predictive power of the complete class association rule set. Using this rule set instead of the complete class association rule set we can avoid redundant computation that would otherwise be required for mining predictive association rules and hence improve the efficiency of the mining process significantly. We present an efficient algorithm for mining the optimal class association rule set using an upward closure property of pruning weak rules before they are actually generated. We have implemented the algorithm and our experimental results show that our algorithm generates the optimal class association rule set, whose size is smaller than 1/17 of the complete class association rule set on average, in significantly less rime than generating the complete class association rule set. Our proposed criterion has been shown very effective for pruning weak rules in dense databases.     

Design and evaluation of visualization support to facilitate association rules modeling

Association rules mining is a popular data mining modeling tool. It discovers interesting associations or correlation relationships among a large set of data items, showing attribute values that occur frequently together in a given dataset. Despite their great potential benefit, current association rules modeling tools are far from optimal. This article studies how visualization techniques can be applied to facilitate the association rules modeling process, particularly what visualization elements should be incorporated and how they can be displayed. Original designs for visualization of rules, integration of data and rule visualizations, and visualization of rule derivation process for supporting interactive visual association rules modeling are proposed in this research. Experimental results indicated that, compared to an automatic association rules modeling process, the proposed interactive visual association rules modeling can significantly improve the effectiveness of modeling, enhance understanding of the applied algorithm, and bring users greater satisfaction with the task. The proposed integration of data and rule visualizations can significantly facilitate understanding rules compared to their nonintegrated counterpart.     

一种基于主属性判定的关联规则挖掘约简算法

关联规则挖掘是经典的数据挖掘方法,越来越多的企业都把它看作是必不可少的战略分析工具。当前关联规则挖掘方法得到的规则过多,令用户在运用时难以理解,因此研究关联规则集的约简方法具有应用价值。研究了数据库模式中关键字包含的主属性对基于Apriori算法的关联规则挖掘产生的关联规则的影响,即部分函数依赖会导致关联规则挖掘的数据集中冗余信息的频繁出现,并产生没有实际价值的关联规则,识别并消除这样的规则就能实现规则集的约简。求全部主属性如同求所有候选关键字问题都是NP难题,因此提出了一种基于一个候选关键字进行验证的算法来判定主属性,从而完成基于主属性判定的关联规则挖掘约简算法的设计与实现,并在最后的实验中验证了该算法的有效性。      

A genetic-fuzzy mining approach for items with multiple minimum supports

Data mining is the process of extracting desirable knowledge or interesting patterns from existing databases for specific purposes. Mining association rules from transaction data is most commonly seen among the mining techniques. Most of the previous mining approaches set a single minimum support threshold for all the items and identify the relationships among transactions using binary values. In the past, we proposed a genetic-fuzzy data-mining algorithm for extracting both association rules and membership functions from quantitative transactions under a single minimum support. In real applications, different items may have different criteria to judge their importance. In this paper, we thus propose an algorithm which combines clustering, fuzzy and genetic concepts for extracting reasonable multiple minimum support values, membership functions and fuzzy association rules from quantitative transactions. It first uses the k-means clustering approach to gather similar items into groups. All items in the same cluster are considered to have similar characteristics and are assigned similar values for initializing a better population. Each chromosome is then evaluated by the criteria of requirement satisfaction and suitability of membership functions to estimate its fitness value. Experimental results also show the effectiveness and the efficiency of the proposed approach.     

Effective product assignment based on association rule mining in retail

Much academic research has been conducted about the process of association rule mining. More effort is now required for practical application of association rules in various commercial fields. A potential application of association rule mining is the problem of product assignment in retail. The product assignment problem involves how to most effectively assign items to sites in retail stores to grow sales. Effective product assignment facilitates cross-selling and convenient shopping for customers to promote maximum sales for retailers. However, little practical research has been done to address the issue. The current study approaches the product assignment problem using association rule mining for retail environments. There are some barriers to overcome in applying association rule mining to the product assignment problem for retail. This study conducts some generalizing to overcome drawbacks caused by the short lifecycles of current products. As a measure of cross-selling, lift is used to compare the effectiveness of various assignments for products. The proposed algorithm consists of three processes, which include mining associations among items, nearest neighbor assignments, and updating assignments. The algorithm was tested on synthetic databases. The results show very effective product assignment in terms of the potential for cross-selling to drive maximum sales for retailers.     

基于矩阵压缩的加权关联规则挖掘算法

关联规则挖掘作为近年来的研究热点之一,其经典算法Apriori算法因需要多次扫描数据库且会产生大量候选项集,严重影响了关联规则的挖掘效率.在此基础上提出了一种基于矩阵压缩的加权关联规则挖掘算法,只需扫描一次数据库,并将其转换为0-1矩阵,根据相关性质对矩阵进行压缩,从而降低了算法执行过程中的计算量;同时,考虑到项目的重要性,采取加权的方法,用求概率的方式设置项目属性的权值.同Apriori算法相比,本算法在挖掘过程中能直接查找高阶频繁项集.实验结果表明,本算法能有效提高关联规则的挖掘效率.     

一种快速挖掘Top-k高效用模式的算法

高效用模式挖掘是数据挖掘领域的一个基础研究方向,其中关于top-k高效用模式的挖掘算法也越来越多,其中k指的是用户需要挖掘的高效用模式的个数。它们可以归纳为两类：二阶段top-k算法和一阶段top-k算法。两者的主要区别是,前者在挖掘的过程中会产生大量的候选模式,这个是影响算法性能的主要因素;后者在挖掘的过程中不产生候选模式。为了更加高效地挖掘效用值最高的k个模式,一阶段算法TKHUP被提出。该算法在进行数据挖掘的过程中主要是通过四个有效策略来减少时间和空间消耗的。通过大量的实验数据表明,TKHUP在时间性能上优于其它top-k高效用模式挖掘算法。     

基于改进多层次模糊关联规则的定量数据挖掘算法

针对单一层次结构实现规则提取具有规则提取准确性不高、算法运行时间长、难以满足用户使用需求的问题,提出一种基于改进多层次模糊关联规则的定量数据挖掘算法。采用高频项目集合,通过不断深化迭代的方法形成自顶向下的挖掘过程,整合模糊集合理论、数据挖掘算法以及多层次分类技术,从事务数据集中寻找模糊关联规则,挖掘出储存在多层次结构事务数据库中定量值信息的隐含知识,实现用户的定制化信息挖掘需求。实验结果表明,提出的数据挖掘算法在挖掘精度和运算时间方面相较于其他算法具有突出优势,可为多层次关联规则提取方法的实际应用带来新的发展空间。     

特殊网络通信行为的数据挖掘与分析

为了挖掘可疑通信的行为模式,定位发生了可疑通信行为的上网账户,本文首先分析了可疑通信行为特点。然后针对已有关联规则挖掘算法不能同时满足多层次数据挖掘和加权关联规则挖掘的问题,分析对比两种典型的基本关联规则算法,以FP-tree为基础,提出了ML-WFP多层次加权关联规则挖掘算法。针对算法中数据项权重的确定问题,由用户设置数据项间的重要性比较关系,借鉴模糊一致矩阵的概念,利用模糊层次分析法计算数据项的权重。最后将该算法应用于可疑通信行为的挖掘。实验测试结果表明可疑通信行为挖掘方案合理有效。     

挖掘关联规则中Apriori算法的研究与改进

关联规则是数据挖掘中重要的研究课题。对关联规则挖掘算法Apriori算法的关键思想以及性能进行了研究,指出该算法的某些不足,并且提出了一种产生候选项集的新算法。该算法提高了原算法的效率。     

Top-<Emphasis Type="Italic">k</Emphasis> probabilistic prevalent co-location mining in spatially uncertain data sets

A co-location pattern is a set of spatial features whose instances frequently appear in a spatial neighborhood. This paper efficiently mines the top-k probabilistic prevalent co-locations over spatially uncertain data sets and makes the following contributions: 1) the concept of the top-k probabilistic prevalent co-locations based on a possible world model is defined; 2) a framework for discovering the top-k probabilistic prevalent co-locations is set up; 3) a matrix method is proposed to improve the computation of the prevalence probability of a top-k candidate, and two pruning rules of the matrix block are given to accelerate the search for exact solutions; 4) a polynomial matrix is developed to further speed up the top-k candidate refinement process; 5) an approximate algorithm with compensation factor is introduced so that relatively large quantity of data can be processed quickly. The efficiency of our proposed algorithms as well as the accuracy of the approximation algorithms is evaluated with an extensive set of experiments using both synthetic and real uncertain data sets.