一种基于关键域子空间的离群数据聚类算法

离群数据发现与分析是数据挖掘的重要组成部分,现有离群数据挖掘算法主要针对如何检测离群对象,缺乏对挖掘出的离群数据集进行解释与分析的有效方法.通过对离群数据来源及特性进行分析并结合粗糙集理论,定义了离群划分相似度的概念,提出了一种基于关键属性域子空间的离群数据聚类算法COKAS,该算法不仅揭示了离群数据子空间特性,进一步获取了扩展知识,而且有助于对整体数据集的理解.对两个多维数据集的实验结果表明,该算法具有良好的适应性及有效性.     

基于近邻关系的离群约简搜索算法

针对传统离群点检测过程中属性多、维度大等问题,结合粗糙集理论,提出一种基于近邻关系的离群约简搜索算法。利用属性约简技术解决对象不相容的问题,并有效缩减离群搜索的属性空间。计算任意点与其他所有点间的距离和,通过计算基于近邻的加权离群因子来判定离群点,并在通用数据集上进行测试。实验结果表明,该离群检测算法的搜索精度较高。     

高维数据集离群子空间特性研究

探讨对挖掘出的离群数据集进行解释与分析的有效方法。以粗糙集理论的属性约简技术为基础,定义了属性离群贡献度等概念对高维数据集离群特性进行了量化描述,提出了离群划分与离群约简思想以及离群数据关键属性域子空间分析方法,给出了一种离群约简算法并分析了算法复杂性。实验表明,这种方法可以有效地揭示离群数据产生来源,有助于对整体数据集的更全面理解,且提出的算法对于问题规模具有较好的适应性。     

基于邻接图的离群数据聚类算法

离群数据是数据中的小模式,因其固有的少数据与稀疏性等特征,使得基于距离或基于统计等常规聚类方式不适用于对离群数据的分类。该文根据离群对象关键域子空间的重合度,定义了离群共享属性集与离群相似度等概念,提出-离群簇分析技术。通过构建离群邻接图并将其稀疏化,将-离群簇搜索与相应的离群邻接图的最大完全子图搜索一一对应,给出一种基于邻接图的离群数据聚类算法。算例及实验结果表明,该方法具有较高的效率及良好的直观性。     

离群数据关键域子空间快速搜索技术

离群数据挖掘与分析在通信欺诈检测、疾病诊断和网络入侵检测等多个领域具有十分重要的意义。离群数据关键域子空间可以获得数据离群起源与特征等相应的延伸知识。通过对离群数据对象与其属性值的关系讨论并基于探索性数据分析方法,提出了一种离群数据关键域子空间实时快速搜索算法。实验结果表明提出的算法是有效的,可以满足大多数实时性检测与分析要求。     

基于属性子空间的孤立点内涵知识挖掘

孤立点通常都包含着重要的信息,挖掘出孤立点的内涵知识可以帮助用户更好地认知数据。通过给出的孤立点的原因属性子空间及其孤立度和孤立点的相似度等概念,提出了一个基于属性子空间的孤立点内涵知识挖掘算法,得到了每个孤立点的原因属性集,并结合聚类的思想把孤立点按照其相似性特征进行了分类,使每一类中的所有孤立点在一定精度下都具有相同的原因属性集。实验结果表明该算法是有效和实用的,且易用性较强。     

一种面向高维混合属性数据的异常挖掘算法

异常检测是数据挖掘领域研究的最基本的问题之一,它在欺诈甄别、气象预报、客户分类和入侵检测等方面有广泛的应用。针对网络入侵检测的需求提出了一种新的基于混合属性聚类的异常挖掘算法,并且依据异常点(outliers)是数据集中的稀有点这一本质,给出了一种新的数据相似性和异常度的定义。本文所提出算法具有线性时间复杂度,在KDDCUP99和WisconsinPrognosisBreastCancer数据集上的实验表明,算本法在提供了近似线性时间复杂度和很好的可扩展性的同时,能够较好的发现数据集中的异常点。     

基于距离的孤立点检测研究

孤立点检测是一个重要的知识发现任务,在分析基于距离的孤立点及其检测算法的基础上,文章提出了一个判定孤立点的新定义,并设计了基于抽样的近似检测算法,用实际数据进行了实验。实验结果表明,新的定义不仅与DB(p,d)孤立点定义有着相同的结果,而且简化了孤立点检测对用户的要求,同时给出了数据对象在数据集中的孤立程度。     

Trajectory outlier detection approach based on common slices sub-sequence

Trajectory outlier detection is one of the most popular trajectory data mining topics. It helps researchers obtain a lot of valuable information that can be used as important guidance in monitoring and forecasting. Existing methods have difficulty in detecting the outlying trajectories with continuous multi-segment exception. To address the problem, in this paper, we propose a novel trajectory outlier detection algorithm based on common slices sub-sequence (TODCSS). For each trajectory, the direction-code sequence is firstly calculated based on the direction of each trajectory segment. Secondly, the corresponding sequence consisting of trajectory slices is obtained by inflection point segmentation. And then, the common slices sub-sequences between two trajectories are found to measure their distance. Finally, the slice outliers and trajectory outliers are detected based on the new CSS distance calculation. Both the intuitive visualization presentation and the experimental results on real Atlantic hurricane dataset, real-life mobility trajectory dataset of taxis in San Francisco and synthetic labeled dataset show that the proposed TODCSS algorithm effectively detects slice and trajectory outliers, and improves accuracy and stability in trajectory outlier detection.     

基于全局最近邻的离群点检测算法

针对全局最近邻离群点检测算法的效率问题,为了能够在数据集中快速准确地检测离群点,运用属性约简技术,将离群点的搜索简约到较小的最具代表性的属性子空间中进行,从而有效降低属性空间搜索的复杂度。在此基础上,通过计算基于近邻的加权离群因子实现离群点的检测并提出了相应的算法。实验表明,该离群点算法具有较好的适应性和有效性。     

Meta-path-based outlier detection in heterogeneous information network

Mining outliers in heterogeneous networks is crucial to many applications,but challenges abound.In this paper,we focus on identifying meta-path-based outliers in heterogeneous information network(HIN),and calculate the similarity between different types of objects.We propose a meta-path-based outlier detection method(MPOutliers)in heterogeneous information network to deal with problems in one go under a unified framework.MPOutliers calculates the heterogeneous reachable probability by combining different types of objects and their relationships.It discovers the semantic information among nodes in heterogeneous networks,instead of only considering the network structure.It also computes the closeness degree between nodes with the same type,which extends the whole heterogeneous network.Moreover,each node is assigned with a reliable weighting to measure its authority degree.Substantial experiments on two real datasets(AMiner and Movies dataset)show that our proposed method is very effective and efficient for outlier detection.     

基于自适应的高对比性子空间的高维离群点检测

基于子空间解决高维离群点挖掘的问题已经引起人们的广泛关注,现有方法存在的主要问题是难以选取合适的子空间且选取计算量大、阈值等参数设置困难等。这些影响了检测精度和检测效率。利用高对比度子空间选取方法解决子空间选取问题,利用自适应方法解决阈值参数的确定问题,据此提出自适应的高对比性子空间离群点检测方法(AHiCS)。该方法利用统计检验算法选取高对比性子空间,在高对比性的子空间里自适应计算离群点得分,提高了离群点检测的精度与效率。理论和实验表明,该方法可以有效地挖掘高维离群点。     

基于最近邻相似度的孤立点检测及半监督聚类算法

传统的聚类算法是一种无监督的学习过程,聚类的精度受到相似性度量方式以及数据集中孤立点的影响,并且算法也没有很好的利用先验知识,无法体现用户的需求。因此提出了基于共享最近邻的孤立点检测及半监督聚类算法。该算法采用共享最近邻为相似度,根据数据点的最近邻居数目来判断是否为孤立点,并在删除孤立点的数据集上进行半监督聚类。在半监督聚类过程中加入了经过扩展的先验知识,同时根据图形分割原理对数据集进行聚类。文中使用真实的数据集进行仿真,其仿真结果表明,本文所提出的算法能有效的检测出孤立点,并具有很好的聚类效果。     

Mining class outliers: concepts, algorithms and applications in CRM

Outliers, or commonly referred to as exceptional cases, exist in many real-world databases. Detection of such outliers is important for many applications and has attracted much attention from the data mining research community recently. However, most existing methods are designed for mining outliers from a single dataset without considering the class labels of data objects. In this paper, we consider the class outlier detection problem ‘given a set of observations with class labels, find those that arouse suspicions, taking into account the class labels’. By generalizing two pioneer contributions [Proc WAIM02 (2002); Proc SSTD03] in this field, we develop the notion of class outlier and propose practical solutions by extending existing outlier detection algorithms to this case. Furthermore, its potential applications in CRM (customer relationship management) are also discussed. Finally, the experiments in real datasets show that our method can find interesting outliers and is of practical use.     

基于鲁棒尺度的统计建模数据中异常点去除算法的研究及应用

针对基于主元分析 (PCA)的统计监控模型受到历史数据中异常点强烈影响的不足,鉴于建模历史数据中存在的异常点会影响过程监控效果,分析目前常用的鲁棒异常值检测算法原理及其缺陷,提出将中心最短距离(CDC)法与椭球多变量整理(MVT)法相结合,构成一种基于鲁棒尺度的CDC-MVT异常值综合检测算法,更加准确地检测异常点。将该算法应用于工业发酵过程,与CDC法和MVT法相比较,该算法能够有效去除建模数据中的异常点。     

快速挖掘数据流中离群点

离群点检测是数据挖掘领域的一个重要分支,当前数据流的离群点检测研究越来越受到关注.为了快速准确地检测出数据流中离群点,提出一种在线数据流离群点检测算法ODDS(outlier detection in online data stream s).它利用数据与频繁模式的相异程度来度量数据的离群程度,通过构建ODDS-Tree树,能动态地更新数据流中候选离群点的离群信息.实验结果验证了该算法与其他同类算法相比具有较高的效率与优良的可扩展性能.     

无线传感器网络中异常读数检测算法研究*

针对无线传感器网络的离群点检测算法由于没有充分考虑数据的时空关联性和网络的分布特性,导致检测精度低、通信量大和计算复杂度高等局限,提出了基于时空关联的分布计算与过滤的在线离群点检测算法。该算法在各传感器节点上利用传感器读数的时间关联性生成候选离群点,并利用空间关联性对候选离群点进行过滤得到局部离群点,最终将所有传感器节点上的局部离群点集中到sink节点上获得全局离群点。利用时空关联性提高了检测精度,利用分布计算与过滤减少了通信量和计算量,理论分析和实验结果均表明该算法优于现有算法。     

Fast outlier detection for very large log data

Density-based outlier detection identifies an outlying observation with reference to the density of the surrounding space. In spite of the several advantages of density-based outlier detections, its computational complexity remains one of the major barriers to its application.The purpose of the present study is to reduce the computation time of LOF (Local Outlier Factor), a density-based outlier detection algorithm. The proposed method incorporates kd-tree indexing and an approximated k-nearest neighbors search algorithm (ANN). Theoretical analysis on the approximation of nearest neighbor search was conducted. A set of experiments was conducted to examine the performance of the proposed algorithm. The results show that the method can effectively detect local outliers in a reduced computation time.     

基于改进的OPTICS聚类和LOPW的离群数据检测算法

针对现有的离群数据检测算法时间复杂度过高,且检测质量不佳的不足,提出一种新的基于改进的OPTICS聚类和LOPW的离群数据检测算法。首先,使用改进的OPTICS聚类算法对原始数据集进行预处理,筛选由聚类形成的可达图得到初步离群数据集;然后,利用新定义的基于P权值的局部离群因子LOPW计算初步离群数据集中对象的离群程度,计算距离时引入去一划分信息熵增量确定属性的权重,提高离群检测准确性。实验结果表明,改进后的算法不仅提高了运算效率,而且提高了对离群数据检测的精确度。     

Distance-based detection and prediction of outliers

A distance-based outlier detection method that finds the top outliers in an unlabeled data set and provides a subset of it, called outlier detection solving set, that can be used to predict the outlierness of new unseen objects, is proposed. The solving set includes a sufficient number of points that permits the detection of the top outliers by considering only a subset of all the pairwise distances from the data set. The properties of the solving set are investigated, and algorithms for computing it, with subquadratic time requirements, are proposed. Experiments on synthetic and real data sets to evaluate the effectiveness of the approach are presented. A scaling analysis of the solving set size is performed, and the false positive rate, that is, the fraction of new objects misclassified as outliers using the solving set instead of the overall data set, is shown to be negligible. Finally, to investigate the accuracy in separating outliers from inliers, ROC analysis of the method is accomplished. Results obtained show that using the solving set instead of the data set guarantees a comparable quality of the prediction, but at a lower computational cost.