Outlier-resistant methods for estimation and model fitting

Least squares is perhaps the most widely used technique for model fitting. In this article, we illustrate the poor performance of least squares when there are spurious values, or outliers, in a sequence of measurements. A brief overview of three well-known classes of robust alternatives to the least-squares mean is presented. For robust regression, a recent proposal called least median squares (LMS) is decribed. LMS regression is compared to least-squares regression in an example involving the estimation of optical fiber geometry. References are provided for software that is available for robust estimation techniques surveyed in this article.     

一种放射CT重建的鲁棒估计算法

放射CT的重建问题可以归结为一种基于泊松分布的最大似然估计（MuxnimumLikelihood,ML），其中最著名的如EM算法[1]，这种算法存在的问题是鲁棒性不强，当观测数据中存在一些出格点（outlier）（如2％）时，EM算法的性能就变得很差。本文分析了算法性能变差的原因，将泊松分布近似为高斯分布后，仿照鲁律估计中的M-估计，构造了一种鲁棒性很强的EM改进算法。实验结果表明，即使在投影数据中混有10％的出格点，仍能重建出满意的结果来。     

Ellipsoidal neighbourhood outlier factor for distributed anomaly detection in resource constrained networks

Anomaly detection in resource constrained wireless networks is an important challenge for tasks such as intrusion detection, quality assurance and event monitoring applications. The challenge is to detect these interesting events or anomalies in a timely manner, while minimising energy consumption in the network. We propose a distributed anomaly detection architecture, which uses multiple hyperellipsoidal clusters to model the data at each sensor node, and identify global and local anomalies in the network. In particular, a novel anomaly scoring method is proposed to provide a score for each hyperellipsoidal model, based on how remote the ellipsoid is relative to their neighbours. We demonstrate using several synthetic and real datasets that our proposed scheme achieves a higher detection performance with a significant reduction in communication overhead in the network compared to centralised and existing schemes.     

A data porting tool for coupling models with different discretization needs

The presented work is part of a larger research program dealing with developing tools for coupling biogeochemical models in contaminated landscapes. The specific objective of this article is to provide researchers with a data porting tool to build hexagonal raster using information from a rectangular raster data (e.g. GIS format). This tool involves a computational algorithm and an open source software (written in C). The method of extending the reticulated functions defined on 2D networks is an essential key of this algorithm and can also be used for other purposes than data porting. The algorithm allows one to build the hexagonal raster with a cell size independent from the geometry of the rectangular raster. The extended function is a bi-cubic spline which can exactly reconstruct polynomials up to degree three in each variable. We validate the method by analyzing errors in some theoretical case studies followed by other studies with real terrain elevation data. We also introduce and briefly present an iterative water routing method and use it for validation on a case with concrete terrain data.     

一种基于自然最近邻的离群检测算法

任何涉及k近邻求解问题的算法被应用于处理不同特征的数据集时,参数k值的选择都会明显影响算法的性能和结果。因而,如何选择k近邻算法中敏感参数k值一直是一个研究难点。提出了一种新的近邻关系———自然最近邻,它不需要设置参数k,每个节点的邻居是由算法自适应计算而形成的。针对离群点检测的特殊性,通过确定自然最近邻居搜索算法的终止条件,提出一种基于自然最近邻的新的离群检测算法ODb3N。实验表明,该算法不仅避免了k近邻中参数的选择问题,而且能够更有效地发现离群簇。     

一种基于密度的不确定数据离群点检测算法

不确定数据普遍存在于如移动计算、RFID技术和传感器网络等大量应用之中.由于不确定数据的离群点检测算法可以提高服务质量,提出一种基于密度的不确定数据离群检测算法RLOF.该算法引入一种R2-tree结构,有效降低了计算局部离群因子时的时间复杂度,同时降低了不确定数据集中的数据更新成本以及海量数据维护成本.理论分析和实验结果充分证明了该算法是有效可行的.     

基于衰减模型的混合属性数据流离群检测

数据流离群检测因内存容量限制和实时检测需求而成为离群检测的一个难点。介绍了一种快速混合属性数据流离群检测算法。在衰减模型下增量聚类数据流,生成代表数据分布的聚类特征集合,半径阈值动态变化;当接收到检测请求时,计算满足条件的每个簇的离群因子,具有高离群因子的簇作为结果输出。同时提出了一种可有效区分离群簇与数据进化初始阶段的方法。算法的时间与空间复杂度同数据流规模近似成线性关系,在真实数据集上的实验结果显示,该算法可有效检测混合属性数据流中的离群点。     

一种孤立点挖掘的混合核方法

孤立点是不具备数据一般特性的数据对象。支持向量机(SVM)将数据点映射到高维特征空间,通过划分最大间隔的超平面来分离孤立点和正常点。利用支持向量机在处理小样本、高维数及泛化性能强等方面的优势,提出了一种新的基于高斯过程潜变量模型(GPLVM)和支持向量分类的检测模型算法。利用GPLVM提供潜变量到数据空间的平滑概率映射实现数据降维,然后通过SVM交叉验证进行孤立点检测。在KDD99数据集上进行了仿真实验,数值结果表明该算法在保证低误报率的前提下能有效地提高检测率,证明了方法的有效性。     

基于鲁棒规范变量分析的故障诊断方法

针对工业过程的建模数据中含有离群点的情况,提出一种基于鲁棒规范变量分析(CVA)的故障诊断方法.该方法使用相关系数的鲁棒估计代替传统的相关系数,通过基于粒子群算法的投影寻踪技术计算最大化鲁棒相关系数的规范变量,从而建立统计模型并监控统计量检测过程的变化.连续搅拌反应器(CSTR)系统的仿真结果说明,鲁棒规范变量分析方法能在含离群点数据的基础上建立准确的统计模型,比规范变量分析更有效地监控过程变化.     

基于密度加权的粗糙K-均值聚类改进算法

针对粗糙K-均值聚类算法中类均值计算式的特点,提出了一种改进的粗糙K-均值算法.改进后的算法基于数据对象所在区域的密度,在类的均值计算过程中对每个对象赋以不同的权重.不同测试数据集的实验结果表明,改进后的粗糙K-均值算法提高了聚类的准确性,降低了迭代次数,并且可以有效地减小孤立点对聚类的影响.