基于距离的孤立点检测及其应用

孤立点检测是一个有趣的知识发现任务，文章介绍了基于距离的孤立点检测及其相关概念，分析了几种有代表性的算法。最后，文章给出了一个判定孤立点的新的定义，并按此定义进行了检测算法，用实际数据进行了实验。实验结果表明，新的定义不仅与DB(p，d)孤立点定义有着相同的结果，而且简化了孤立点检测对用户的需求，同时给出了数据对象在数据集中的孤立程度。     

基于距离的孤立点检测研究

孤立点检测是一个重要的知识发现任务,在分析基于距离的孤立点及其检测算法的基础上,文章提出了一个判定孤立点的新定义,并设计了基于抽样的近似检测算法,用实际数据进行了实验。实验结果表明,新的定义不仅与DB(p,d)孤立点定义有着相同的结果,而且简化了孤立点检测对用户的要求,同时给出了数据对象在数据集中的孤立程度。     

基于相对密度的孤立点和边界点识别算法

根据孤立点是数据集合中与大多数数据的属性不一致的数据,边界点是位于不同密度数据区域边缘的数据对象,提出了基于相对密度的孤立点和边界点识别算法(OBRD)。该算法判断一个数据点是否为边界点或孤立点的方法是:将以该数据点为中心、r为半径的邻域按维平分为2个半邻域,由这些半邻域与原邻域的相对密度确定该数据点的孤立度和边界度,再结合阈值作出判断。实验结果表明,该算法能精准有效地对多密度数据集的孤立点和聚类边界点进行识别。     

基于灰色关联分析的孤立点挖掘算法

孤立点挖掘是数据挖掘的重要研究方向之一,其目标是发现数据集中不具备数据一般特性的数据对象。传统孤立点挖掘算法通常基于项集属性,不适用于多目标决策和综合评价。该文提出一种基于灰色关联分析的孤立点检测算法OMGRA,通过总评价判断数挖掘孤立点集,避免人工确定阈值。实例分析表明,该算法能有效检测数据集中的孤立点,挖掘出的孤立点符合实际情况。     

基于局部偏离因子的孤立点检测算法

孤立点检测是知识发现中的一个活跃领域,如信用卡欺诈、入侵检测等。研究孤立点的异常行为能发现隐藏在数据集中更有价值的知识。该文提出基于局部偏离因子(LDF)的孤立点检测算法,利用每个数据点的LDF衡量该数据点的偏离程度。实验结果表明,该算法能有效检测孤立点,其效率高于LSC算法。     

一种基于划分的孤立点检测算法

孤立点是不具备数据一般特性的数据对象.划分的方法是通过将数据集中的数据点分布的空间划分为不相交的超矩形单元集合,匹配数据对象到单元中,然后通过各个单元的统计信息来发现孤立点.由于大多真实数据集具有较大偏斜,因此划分后会产生影响算法性能的大量空单元.由此,提出了一种新的索引结构--CD-Tree(cell dimension tree),用于索引非空单元.为了优化CD-Tree结构和指导对数据的划分,提出了基于划分的数据偏斜度(skew of data,简称SOD)概念.基于CD-Tree与SOD,设计了新的孤立点检测算法.实验结果表明,该算法与基于单元的算法相比,在效率及有效处理的维数方面均有显著提高.     

ISAD:一种新的基于属性距离和的孤立点检测算法

孤立点是数据对象在某些属性(维)上波动形成的.由此,本文提出了关键属性的概念,用于描述影响数据稳定性的属性.在真实数据集中,只有一部分属性是能够决定某数据是否是孤立点的关键属性.由此,本文提出了关键属性隶属度的定义及其求解算法,并在此基础上提出了一种新的基于属性距离和的孤立点检测算法.实验结果表明,该算法较基于单元的算法在效率及雏数可扩展方面均有显著提高.     

基于反向K近邻的孤立点检测算法

提出了基于反向K近邻（RKNN）的孤立点检测算法ODRKNN。ODRKNN算法用每个数据点的反向k近邻个数来衡量该数据点的偏离程度,在综合数据集和真实数据集上的实验结果表明,该算法能有效地检测出孤立点,且算法的效率高于算法LOF和LSC的效率.     

基于单元的孤立点检测算法改进及应用

研究基于单元的孤立点检测算法,给出数据空间的单元格划分及数据对象分配算法。针对该算法中阈值M设置的不足,对算法进行改进并应用于纳税行为的分析。与其他孤立点检测算法对比的结果表明,该算法不仅能有效挖掘纳税行为中的孤立点,还能确定孤立点的位置,有利于对纳税行为的分析。     

XML数据中孤立点检测方法研究

针对XML数据中的孤立点问题,利用聚类分析思想和XML数据嵌套结构特性所蕴含的元素间的上下文信息,设计了一种在XML半结构数据中检测孤立点的算法.该算法把逻辑相关的结点聚集到相应的子空间中,并基于这些相关子空间计算孤立点兴趣度度量XO度量,以此来识别孤立点数据.实验结果表明,该算法在一定规模的孤立点数据下能够达到较高的识别效率.     

基于聚类的孤立点集探测算法

在分析现有的孤立点探测算法的基础上,提出一种新颖的基于聚类的孤立点集挖掘算法．该算法不但能够探测出所有的孤立点,还能根据孤立点产生的原因对这些孤立点进行分类。通过实验数据测试,本算法有较好的稳定性和性能的优越性。     

Outliers detection in environmental monitoring databases

Environmental monitoring is nowadays an important task in many industrial operations. In order to comply with strong environmental laws, they have implemented monitoring systems based on a network of air quality and meteorological stations providing real-time measurements of key variables associated to the distribution of pollutants in surrounding areas. These measurements can be contaminated by outliers, which must be discarded in order to have a consistent set of data. This work presents a nonlinear procedure for outliers detection based on residual analysis of regression with Partial Least Squares and Artificial Neural Networks. In order to minimize the negative effect of outliers in the training dataset a learning algorithm with regularization is proposed. This algorithm is based on a Quasi-Newton optimization method and it was tested on a simulated nonlinear process, on real data from environmental monitoring contaminated with synthetic outliers, and finally applied to a real environmental monitoring data obtained from a monitoring station and having natural outliers. The results are encouraging and further developments are foreseen for including information from neighboring stations and emission source operation.     

基于局部相关维度的流形离群点检测算法

传统的离群点检测算法不适合检测流形离群点,目前专门针对流形离群点检测的算法报道较少。为此,基于实验观察的启示,提出用流形局部相关维度检测流形离群点的算法。首先探讨内在维度的性质,并基于实验观察提出用流形局部相关维度来度量流形离群点,然后证明流形局部相关维度可表征数据样本离群的性质,最后基于此性质提出流形离群点检测算法。在人工数据与真实数据上的实验表明本算法可检测流形离群点,且本算法比最近报道的流形除噪算法具有更优的性能。     

Genetic algorithms for outlier detection and variable selection in linear regression models

This article addresses some problems in outlier detection and variable selection in linear regression models. First, in outlier detection there are problems known as smearing and masking. Smearing means that one outlier makes another, non-outlier observation appear as an outlier, and masking that one outlier prevents another one from being detected. Detecting outliers one by one may therefore give misleading results. In this article a genetic algorithm is presented which considers different possible groupings of the data into outlier and non-outlier observations. In this way all outliers are detected at the same time. Second, it is known that outlier detection and variable selection can influence each other, and that different results may be obtained, depending on the order in which these two tasks are performed. It may therefore be useful to consider these tasks simultaneously, and a genetic algorithm for a simultaneous outlier detection and variable selection is suggested. Two real data sets are used to illustrate the algorithms, which are shown to work well. In addition, the scalability of the algorithms is considered with an experiment using generated data.I would like to thank Dr Tero Aittokallio and an anonymous referee for useful comments.     

散乱点云近离群点识别算法

针对原始曲面变化度的局部离群系数(SVLOF)无法有效滤除三维实体的棱边或棱角处的离群点问题,提出了一种散乱点云近离群点的滤除算法。该算法首先将SVLOF定义在类k邻域上,并将SVLOF的定义内容进行了扩展,使其既能滤除平滑曲面上的离群点,又能滤除三维实体的棱边或棱角点处的离群点,同时仍然保留SVLOF原有的足够宽泛的阈值选取空间。仿真数据和实际数据的实验结果均表明,在效率基本保持不变的情况下,所提算法能比原始SVLOF算法更有效地检测出距离主体点云近的离群点。     

快速的基于单元格的离群数据挖掘算法

针对数据集中离群数据的挖掘速度的问题,提出了快速的基于单元格的离群数据挖掘算法。该算法根据网格的特点首先将数据划分成若干空间单元,从而减少区域查询次数,提高离群数据的挖掘速度,然后根据单元格的阈值来判定一个数据是否为离群数据。通过数据测试表明,该算法能够快速有效地挖掘出数据集中的离群数据。     

基于粗糙集的高维空间离群点发现算法研究

该文提出在高维空间下离群点发现技术的新方法,即利用粗糙集的属性约简技术减少高维空间的维数,并在各个关联规则子空间下对数据集进行基于密度的离群点挖掘,使高维空间下的离群点挖掘更具有实用性。数据分析表明,该算法能有效地发现高维空间数据集中的离群点。     

Rule-based OneClass-DS learning algorithm

One-class learning algorithms are used in situations when training data are available only for one class, called target class. Data for other class(es), called outliers, are not available. One-class learning algorithms are used for detecting outliers, or novelty, in the data. The common approach in one-class learning is to use density estimation techniques or adapt standard classification algorithms to define a decision boundary that encompasses only the target data. In this paper, we introduce OneClass-DS learning algorithm that combines rule-based classification with greedy search algorithm based on density of features. Its performance is tested on 25 data sets and compared with eight other one-class algorithms; the results show that it performs on par with those algorithms.     

基于距离和密度的多阶段聚类

随着聚类技术的发展．对不同密度的数据集的聚类需求也越来越迫切。为了解决不同密度数据集的聚类问题,提出一种基于距离和密度的多阶段聚类算法MCDD。该算法主要采用多阶段密度处理技术提取不同密度的聚类,同时使用密度因子提高聚类的精度．最后通过使用距离阈值的方法去除孤立点和噪声数据。实验表明,该算法在扩展性方面表现良好．对任意形状和大小的聚类都可以很好地处理,并能够很好地识别出孤立点或噪声,在处理多密度聚类方面有很好的精度。     

Incremental rare pattern based approach for identifying outliers in medical data

Outlier detection is an imperative field of data mining that has several applications in the field of medical research. Mining outliers based on the notion of rare patterns can be a promising solution for medical diagnosis as it attempts to identify the unconventional and abnormal risk patterns present in medical data. A crucial issue in medical data analysis is the continuous growth of medical databases due to the addition of new records. Existing outlier detection techniques are capable of handling only static data and thus re-execute from scratch to identify the outliers from incremental medical data. This paper introduces an efficient rare pattern based outlier detection (RPOD) method that identifies outliers by mining rare patterns from incremental data. To avoid multiple database scans and expensive candidate generation steps performed by existent rare pattern mining techniques and facilitate incremental mining, a single pass prefix tree-based rare pattern mining technique is proposed. The proposed rare pattern mining technique is a modification of the well-known FP-Growth frequent pattern mining algorithm. Furthermore, to identify the outliers based on the set of generated rare patterns, an outlier detection technique is also presented. The significance of proposed RPOD approach is demonstrated using several well-known medical datasets. Comparative performance evaluation substantiates the predominance of RPOD approach over existing outlier mining methods.