On-line anomaly detection and resilience in classifier ensembles

Detection of anomalies is a broad field of study, which is applied in different areas such as data monitoring, navigation, and pattern recognition. In this paper we propose two measures to detect anomalous behaviors in an ensemble of classifiers by monitoring their decisions; one based on Mahalanobis distance and another based on information theory. These approaches are useful when an ensemble of classifiers is used and a decision is made by ordinary classifier fusion methods, while each classifier is devoted to monitor part of the environment. Upon detection of anomalous classifiers we propose a strategy that attempts to minimize adverse effects of faulty classifiers by excluding them from the ensemble. We applied this method to an artificial dataset and sensor-based human activity datasets, with different sensor configurations and two types of noise (additive and rotational on inertial sensors). We compared our method with two other well-known approaches, generalized likelihood ratio (GLR) and One-Class Support Vector Machine (OCSVM), which detect anomalies at data/feature level.     

Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs

Hidden Markov models (HMMs) have been shown to provide a high level performance for detecting anomalies in sequences of system calls to the operating system kernel. Using Boolean conjunction and disjunction functions to combine the responses of multiple HMMs in the ROC space may significantly improve performance over a “single best” HMM. However, these techniques assume that the classifiers are conditional independent, and their of ROC curves are convex. These assumptions are violated in most real-world applications, especially when classifiers are designed using limited and imbalanced training data. In this paper, the iterative Boolean combination (IBC) technique is proposed for efficient fusion of the responses from multiple classifiers in the ROC space. It applies all Boolean functions to combine the ROC curves corresponding to multiple classifiers, requires no prior assumptions, and its time complexity is linear with the number of classifiers. The results of computer simulations conducted on both synthetic and real-world host-based intrusion detection data indicate that the IBC of responses from multiple HMMs can achieve a significantly higher level of performance than the Boolean conjunction and disjunction combinations, especially when training data are limited and imbalanced. The proposed IBC is general in that it can be employed to combine diverse responses of any crisp or soft one- or two-class classifiers, and for wide range of application domains.     

Adaptive anomaly detection with evolving connectionist systems

Anomaly detection holds great potential for detecting previously unknown attacks. In order to be effective in a practical environment, anomaly detection systems have to be capable of online learning and handling concept drift. In this paper, a new adaptive anomaly detection framework, based on the use of unsupervised evolving connectionist systems, is proposed to address these issues. It is designed to adapt to normal behavior changes while still recognizing anomalies. The evolving connectionist systems learn a subject's behavior in an online, adaptive fashion through efficient local element tuning. Experiments with the KDD Cup 1999 network data and the Windows NT user profiling data show that our adaptive anomaly detection systems, based on Fuzzy Adaptive Resonance Theory (ART) and Evolving Fuzzy Neural Networks (EFuNN), can significantly reduce the false alarm rate while the attack detection rate remains high.     

Feature selection for ensembles applied to handwriting recognition

Feature selection for ensembles has shown to be an effective strategy for ensemble creation due to its ability of producing good subsets of features, which make the classifiers of the ensemble disagree on difficult cases. In this paper we present an ensemble feature selection approach based on a hierarchical multi-objective genetic algorithm. The underpinning paradigm is the “overproduce and choose”. The algorithm operates in two levels. Firstly, it performs feature selection in order to generate a set of classifiers and then it chooses the best team of classifiers. In order to show its robustness, the method is evaluated in two different contexts:supervised and unsupervised feature selection. In the former, we have considered the problem of handwritten digit recognition and used three different feature sets and multi-layer perceptron neural networks as classifiers. In the latter, we took into account the problem of handwritten month word recognition and used three different feature sets and hidden Markov models as classifiers. Experiments and comparisons with classical methods, such as Bagging and Boosting, demonstrated that the proposed methodology brings compelling improvements when classifiers have to work with very low error rates. Comparisons have been done by considering the recognition rates only.     

LoGID: An adaptive framework combining local and global incremental learning for dynamic selection of ensembles of HMMs

In this work, we propose the LoGID (Local and Global Incremental Learning for Dynamic Selection) framework, the main goal of which is to adapt hidden Markov model-based pattern recognition systems during both the generalization and learning phases. Given that the baseline system is composed of a pool of base classifiers, adaptation during generalization is performed through the dynamic selection of the members of this pool that best recognize each test sample. This is achieved by the proposed K-nearest output profiles algorithm, while adaptation during learning consists of gradually updating the knowledge embedded in the base classifiers, by processing previously unobserved data. This phase employs two types of incremental learning: local and global. Local incremental learning involves updating the pool of base classifiers by adding new members to this set. The new members are created with the Learn++ algorithm. Global incremental learning, in contrast, consists of updating the set of output profiles used during generalization. The proposed framework has been evaluated on a diversified set of databases. The results indicate that LoGID is promising. For most databases, the recognition rates achieved by the proposed method are higher than those achieved by other state-of-the-art approaches, such as batch learning. Furthermore, the simulated incremental learning setting demonstrates that LoGID can effectively improve the performance of systems created with small training sets as more data are observed over time.     

Human perspective to anomaly detection for cybersecurity

Traditionally signature-based network Intrusion Detection Systems (IDS) rely on inputs from domain experts and can only identify the attacks if they occur as individual event. IDS generate large number of alerts and it becomes very difficult for human users to go through each message. Previous researches have proposed analytics based approaches to analyze IDS alert patterns based on anomaly detection models, multi-steps models or probabilistic approaches. However, due to the complexities of network intrusions, it is impossible to develop all possible attack patterns or to avoid false positives. With the advance in technologies and popularity of networks in our daily life, it is becoming more and more difficult to detect network intrusions. However, no matter how rapid the technologies change, the human behaviors behind the cyber attacks stay relatively constant. This provides us an opportunity to develop an improved system to detect the unusual cyber attacks. In this paper, we developed four network intrusion models based on consideration of human factors. We then tested these models on ITOC Cyber Defense Competition (CDX) 2009 data. Our results are encouraging. These Models are not only able to recognize most network attacks identified by SNORT log alerts, they are also able to distinguish the non-attack network traffic that was potentially missed by SNORT as indicated by ground truth validation of the data.     

Adaptive fusion and co-operative training for classifier ensembles

In this paper, architectures and methods of decision aggregation in classifier ensembles are investigated. Typically, ensembles are designed in such a way that each classifier is trained independently and the decision fusion is performed as a post-process module. In this study, however, we are interested in making the fusion a more adaptive process. We first propose a new architecture that utilizes the features of a problem to guide the decision fusion process. By using both the features and classifiers outputs, the recognition strengths and weaknesses of the different classifiers are identified. This information is used to improve overall generalization capability of the system. Furthermore, we propose a co-operative training algorithm that allows the final classification to determine whether further training should be carried out on the components of the architecture. The performance of the proposed architecture is assessed by testing it on several benchmark problems. The new architecture shows improvement over existing aggregation techniques. Moreover, the proposed co-operative training algorithm provides a means to limit the users’ intervention, and maintains a level of accuracy that is competitive to that of most other approaches.     

Variational restricted Boltzmann machines to automated anomaly detection

Neural Computing and Applications - Data-driven methods are implemented using particularly complex scenarios that reflect in-depth perennial knowledge and research. Hence, the available intelligent...     

人工智能集成学习方法在入侵检测中的运用

在监督学习的分类中,集成学习已经成功地运用于许多不同的领域。文献中许多研究者通过考虑不同的组合方式、训练数据集、基分类器以及其他因素提出了不同的集成学习方法。人工智能技术相比于其他技术有许多优点,在解决入侵检测问题的集成学习发展中发挥着重要的作用。但是,目前还没有一篇综述性的文章来回顾解决入侵检测问题的通用集成方法和基于人工智能技术的集成学习方法。对入侵检测问题的集成学习方法进行对比和总结,并对该领域今后的研究方向进行了展望,为理解入侵检测系统领域的集成提供帮助。     

Non-parametric bootstrap ensembles for detection of tumor lesions

We propose and assess a set of non-parametric ensembles, including bagging and boosting schemes, to recognize tumors in digital mammograms. Different approaches were examined as candidates for the two major components of the bagging ensembles, three spatial resampling schemes (residuals, centers and standardized centers), and four combination criteria (at least one, majority vote, top 25% models, and false discovery rate). A conversion to a classification problem prior to aggregation was employed for the boosting ensemble. The ensembles were compared at the lesion level against a single expert, and to a set of Markov Random Field (MRF) models in real images using three different criteria. The performance of the ensembles depended on its components, particularly the combination, with at least one and top 25% models offering a greater detection power independently of the type of lesion, and of the booststrapping scheme in a lesser degree. The ensembles were comparable in performance to MRFs in the unsupervised recognition of patterns exhibiting spatial structure.     

A hybrid machine learning approach to network anomaly detection

Zero-day cyber attacks such as worms and spy-ware are becoming increasingly widespread and dangerous. The existing signature-based intrusion detection mechanisms are often not sufficient in detecting these types of attacks. As a result, anomaly intrusion detection methods have been developed to cope with such attacks. Among the variety of anomaly detection approaches, the Support Vector Machine (SVM) is known to be one of the best machine learning algorithms to classify abnormal behaviors. The soft-margin SVM is one of the well-known basic SVM methods using supervised learning. However, it is not appropriate to use the soft-margin SVM method for detecting novel attacks in Internet traffic since it requires pre-acquired learning information for supervised learning procedure. Such pre-acquired learning information is divided into normal and attack traffic with labels separately. Furthermore, we apply the one-class SVM approach using unsupervised learning for detecting anomalies. This means one-class SVM does not require the labeled information. However, there is downside to using one-class SVM: it is difficult to use the one-class SVM in the real world, due to its high false positive rate. In this paper, we propose a new SVM approach, named Enhanced SVM, which combines these two methods in order to provide unsupervised learning and low false alarm capability, similar to that of a supervised SVM approach.We use the following additional techniques to improve the performance of the proposed approach (referred to as Anomaly Detector using Enhanced SVM): First, we create a profile of normal packets using Self-Organized Feature Map (SOFM), for SVM learning without pre-existing knowledge. Second, we use a packet filtering scheme based on Passive TCP/IP Fingerprinting (PTF), in order to reject incomplete network traffic that either violates the TCP/IP standard or generation policy inside of well-known platforms. Third, a feature selection technique using a Genetic Algorithm (GA) is used for extracting optimized information from raw internet packets. Fourth, we use the flow of packets based on temporal relationships during data preprocessing, for considering the temporal relationships among the inputs used in SVM learning. Lastly, we demonstrate the effectiveness of the Enhanced SVM approach using the above-mentioned techniques, such as SOFM, PTF, and GA on MIT Lincoln Lab datasets, and a live dataset captured from a real network. The experimental results are verified by m-fold cross validation, and the proposed approach is compared with real world Network Intrusion Detection Systems (NIDS).     

Clustering ellipses for anomaly detection

Comparing, clustering and merging ellipsoids are problems that arise in various applications, e.g., anomaly detection in wireless sensor networks and motif-based patterned fabrics. We develop a theory underlying three measures of similarity that can be used to find groups of similar ellipsoids in p-space. Clusters of ellipsoids are suggested by dark blocks along the diagonal of a reordered dissimilarity image (RDI). The RDI is built with the recursive iVAT algorithm using any of the three (dis) similarity measures as input and performs two functions: (i) it is used to visually assess and estimate the number of possible clusters in the data; and (ii) it offers a means for comparing the three similarity measures. Finally, we apply the single linkage and CLODD clustering algorithms to three two-dimensional data sets using each of the three dissimilarity matrices as input. Two data sets are synthetic, and the third is a set of real WSN data that has one known second order node anomaly. We conclude that focal distance is the best measure of elliptical similarity, iVAT images are a reliable basis for estimating cluster structures in sets of ellipsoids, and single linkage can successfully extract the indicated clusters.     

Temporal difference learning applied to sequential detection

This paper proposes a novel neural-network method for sequential detection, We first examine the optimal parametric sequential probability ratio test (SPRT) and make a simple equivalent transformation of the SPRT that makes it suitable for neural-network architectures. We then discuss how neural networks can learn the SPRT decision functions from observation data and labels. Conventional supervised learning algorithms have difficulties handling the variable length observation sequences, but a reinforcement learning algorithm, the temporal difference (TD) learning algorithm works ideally in training the neural network. The entire neural network is composed of context units followed by a feedforward neural network. The context units are necessary to store dynamic information that is needed to make good decisions. For an appropriate neural-network architecture, trained with independent and identically distributed (iid) observations by the TD learning algorithm, we show that the neural-network sequential detector can closely approximate the optimal SPRT with similar performance. The neural-network sequential detector has the additional advantage that it is a nonparametric detector that does not require probability density functions. Simulations demonstrated on iid Gaussian data show that the neural network and the SPRT have similar performance.     

异常检测测试平台的设计

提出了一种可以测试不同算法的异常检测测试平台.为适合大规模分布式网络,将网络分成不同网段,每个网段放置一个探测器IC,把不同IC提供的网络数据汇总至异常检测部件,在此进行异常分析,并根据分析结果对可能的入侵行为进行实时报警,其中的异常检测算法可以替换.最后,针对一种基于统计的异常检测算法进行了实验,并给出异常检测结果.     

Back-face culling applied to collision detection of polyhedra

Back-face culling is a preprocessing technique used in computer graphics to speed up the rendering of polyhedra. In this paper we show how this technique can be modified to reduce unnecessary checking of boundary elements in collison detection for a physical-based simulation and animation systems. At each time step, we determine a priori which faces cannot be part of the contact between two polyhedra and thus can be culled. In the computer graphics technique, the normal vector of a polygon is compared with the view direction. Here, the normal is compared to one or possibly several relative-velocity vectors, and the face is culled when its motion is in the opposite direction of the normal vector. We also give an algorithm that takes linear time in terms of the number of faces, and on the average eliminates half of the polygons. Owing to its low computational overhead, when it is used as a front end to a collision detection system, a noticeable improvement in performance can be achieved.     

基于数据挖掘的自适应异常分析

正常用户行为活动是随时间变化的,一个异常分析系统要能适应这种变化更新正常行为模型,避免误报警.对增量更新算法进行了研究,使用线性回归的方法对相似度进行估计,如果实际相似度与估计值差值大于某个阈值,则产生报警;否则采用改进的滑动窗增量挖掘的方法,更新正常活动模型.并用DARPA-MIT 1999数据集验证其可行性.     

Adaptive fuzzy neuro-observer applied to low cost INS/GPS

A combined MEMS Inertial Navigation System (INS) with GPS is used to provide position and velocity data of land vehicles. Data fusion of INS and GPS measurements are commonly achieved through a conventional Extended Kalman filter (EKF). Considering the required accurate model of system together with perfect knowledge of predefined error models, the performance of the EKF is decreased due to unmodeled nonlinearities and unknown bias uncertainties of MEMS inertial sensors. Universal knowledge based approximators comprising of neural networks and fuzzy logic methods are capable of approximating the nonlinearities and the uncertainties of practical systems. First, in this paper, a new fuzzy neural network (FNN) function approximator is used to model unknown nonlinear systems. Second, the process of design and real-time implementation of an adaptive fuzzy neuro-observer (AFNO) in integrated low-cost INS/GPS positioning systems is proposed. To assess the long time performance of the proposed AFNO method, wide range tests of a real INS/GPS with a car vehicle have been performed. The unbiased estimation results of the AFNO show the superiority of the proposed method compared with the classic EKF and the adaptive neuro-observer (ANO) including a pure artificial neural network (ANN) function approximator.     

Adaptive element-free Galerkin method applied to the limit analysis of plates

The implementation of an h-adaptive element-free Galerkin (EFG) method in the framework of limit analysis is described. The naturally conforming property of meshfree approximations (with no nodal connectivity required) facilitates the implementation of h-adaptivity. Nodes may be moved, discarded or introduced without the need for complex manipulation of the data structures involved. With the use of the Taylor expansion technique, the error in the computed displacement field and its derivatives can be estimated throughout the problem domain with high accuracy. A stabilized conforming nodal integration scheme is extended for use in error estimation and results in an efficient and truly meshfree adaptive method. To demonstrate its effectiveness the procedure is then applied to plates with various boundary conditions.     

Adaptive neuro-genetic control of chaos applied to the attitude control problem

Conventional adaptive control techniques have, for the most part, been based on methods for linear or weakly non-linear systems. More recently, neural network and genetic algorithm controllers have started to be applied to complex, non-linear dynamic systems. The control of chaotic dynamic systems poses a series of especially challenging problems. In this paper, an adaptive control architecture using neural networks and genetic algorithms is applied to a complex, highly nonlinear, chaotic dynamic system: the adaptive attitude control problem (for a satellite), in the presence of large, external forces (which left to themselves led the system into a chaotic motion). In contrast to the OGY method, which uses small control adjustments to stabilize a chaotic system in an otherwise unstable but natural periodic orbit of the system, the neuro-genetic controller may use large control adjustments and proves capable of effectively attaining any specified system state, with no a prioriknowledge of the dynamics, even in the presence of significant noise.This work was partly supported by SERC grant 90800355.     

A robust unsupervised anomaly detection framework

Applied Intelligence - Anomaly detection plays an essential role in monitoring dependable systems and networks such as computer clusters, water treatment systems, sensor networks, etc. However,...