On-line anomaly detection and resilience in classifier ensembles

Detection of anomalies is a broad field of study, which is applied in different areas such as data monitoring, navigation, and pattern recognition. In this paper we propose two measures to detect anomalous behaviors in an ensemble of classifiers by monitoring their decisions; one based on Mahalanobis distance and another based on information theory. These approaches are useful when an ensemble of classifiers is used and a decision is made by ordinary classifier fusion methods, while each classifier is devoted to monitor part of the environment. Upon detection of anomalous classifiers we propose a strategy that attempts to minimize adverse effects of faulty classifiers by excluding them from the ensemble. We applied this method to an artificial dataset and sensor-based human activity datasets, with different sensor configurations and two types of noise (additive and rotational on inertial sensors). We compared our method with two other well-known approaches, generalized likelihood ratio (GLR) and One-Class Support Vector Machine (OCSVM), which detect anomalies at data/feature level.     

Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs

Hidden Markov models (HMMs) have been shown to provide a high level performance for detecting anomalies in sequences of system calls to the operating system kernel. Using Boolean conjunction and disjunction functions to combine the responses of multiple HMMs in the ROC space may significantly improve performance over a “single best” HMM. However, these techniques assume that the classifiers are conditional independent, and their of ROC curves are convex. These assumptions are violated in most real-world applications, especially when classifiers are designed using limited and imbalanced training data. In this paper, the iterative Boolean combination (IBC) technique is proposed for efficient fusion of the responses from multiple classifiers in the ROC space. It applies all Boolean functions to combine the ROC curves corresponding to multiple classifiers, requires no prior assumptions, and its time complexity is linear with the number of classifiers. The results of computer simulations conducted on both synthetic and real-world host-based intrusion detection data indicate that the IBC of responses from multiple HMMs can achieve a significantly higher level of performance than the Boolean conjunction and disjunction combinations, especially when training data are limited and imbalanced. The proposed IBC is general in that it can be employed to combine diverse responses of any crisp or soft one- or two-class classifiers, and for wide range of application domains.     

自适应的Web攻击异常检测方法

针对传统建模容易引入不可信样本的问题,提出了一种自适应建立基于Web攻击异常检测模型的方法。依据样本中Request-URL的结构特征对样本集进行分类,并利用样本的各属性来构造样本分类子集的离散性函数,其中离散程度值将作为识别正常行为集的依据;在此基础上,使用改进的隐马尔可夫模型(HMM)算法对正常行为样本集进行建模,并利用HMM合并的方法实现检测模型的动态更新。实验结果表明,所提方法建立的模型能够有效地识别出Web攻击请求,并降低检测的误报率。     

Adaptive anomaly detection with evolving connectionist systems

Anomaly detection holds great potential for detecting previously unknown attacks. In order to be effective in a practical environment, anomaly detection systems have to be capable of online learning and handling concept drift. In this paper, a new adaptive anomaly detection framework, based on the use of unsupervised evolving connectionist systems, is proposed to address these issues. It is designed to adapt to normal behavior changes while still recognizing anomalies. The evolving connectionist systems learn a subject's behavior in an online, adaptive fashion through efficient local element tuning. Experiments with the KDD Cup 1999 network data and the Windows NT user profiling data show that our adaptive anomaly detection systems, based on Fuzzy Adaptive Resonance Theory (ART) and Evolving Fuzzy Neural Networks (EFuNN), can significantly reduce the false alarm rate while the attack detection rate remains high.     

一种基于正常简档聚类的自适应异常检测模型*

为解决入侵检测系统存在检测率低、网络数据变化适应性弱的问题,选取正常数据记录通过聚类算法建立正常简档,然后依据正常简档对网络数据记录进行检测,并结合已检测出来的正常数据记录对正常简档进行更新。KDD CUP99数据的实验表明,该系统能够适应数据的变化趋势,在保持较低的误报率前提下获得了较好的检测率。     

Feature selection for ensembles applied to handwriting recognition

Feature selection for ensembles has shown to be an effective strategy for ensemble creation due to its ability of producing good subsets of features, which make the classifiers of the ensemble disagree on difficult cases. In this paper we present an ensemble feature selection approach based on a hierarchical multi-objective genetic algorithm. The underpinning paradigm is the “overproduce and choose”. The algorithm operates in two levels. Firstly, it performs feature selection in order to generate a set of classifiers and then it chooses the best team of classifiers. In order to show its robustness, the method is evaluated in two different contexts:supervised and unsupervised feature selection. In the former, we have considered the problem of handwritten digit recognition and used three different feature sets and multi-layer perceptron neural networks as classifiers. In the latter, we took into account the problem of handwritten month word recognition and used three different feature sets and hidden Markov models as classifiers. Experiments and comparisons with classical methods, such as Bagging and Boosting, demonstrated that the proposed methodology brings compelling improvements when classifiers have to work with very low error rates. Comparisons have been done by considering the recognition rates only.     

An intelligent algorithm with feature selection and decision rules applied to anomaly intrusion detection

Intrusion detection system (IDS) is to monitor the attacks occurring in the computer or networks. Anomaly intrusion detection plays an important role in IDS to detect new attacks by detecting any deviation from the normal profile. In this paper, an intelligent algorithm with feature selection and decision rules applied to anomaly intrusion detection is proposed. The key idea is to take the advantage of support vector machine (SVM), decision tree (DT), and simulated annealing (SA). In the proposed algorithm, SVM and SA can find the best selected features to elevate the accuracy of anomaly intrusion detection. By analyzing the information from using KDD’99 dataset, DT and SA can obtain decision rules for new attacks and can improve accuracy of classification. In addition, the best parameter settings for the DT and SVM are automatically adjusted by SA. The proposed algorithm outperforms other existing approaches. Simulation results demonstrate that the proposed algorithm is successful in detecting anomaly intrusion detection.     

On developing an automatic threshold applied to feature selection ensembles

Feature selection ensemble methods are a recent approach aiming at adding diversity in sets of selected features, improving performance and obtaining more robust and stable results. However, using an ensemble introduces the need for an aggregation step to combine all the output methods that confirm the ensemble. Besides, when trying to improve computational efficiency, ranking methods that order all initial features are preferred, and so an additional thresholding step is also mandatory. In this work two different ensemble designs based on ranking methods are described. The main difference between them is the order in which the combination and thresholding steps are performed. In addition, a new automatic threshold based on the combination of three data complexity measures is proposed and compared with traditional thresholding approaches based on retaining a fixed percentage of features. The behavior of these methods was tested, according to the SVM classification accuracy, with satisfactory results, for three different scenarios: synthetic datasets and two types of real datasets (where sample size is much higher than feature size, and where feature size is much higher than sample size).     

LoGID: An adaptive framework combining local and global incremental learning for dynamic selection of ensembles of HMMs

In this work, we propose the LoGID (Local and Global Incremental Learning for Dynamic Selection) framework, the main goal of which is to adapt hidden Markov model-based pattern recognition systems during both the generalization and learning phases. Given that the baseline system is composed of a pool of base classifiers, adaptation during generalization is performed through the dynamic selection of the members of this pool that best recognize each test sample. This is achieved by the proposed K-nearest output profiles algorithm, while adaptation during learning consists of gradually updating the knowledge embedded in the base classifiers, by processing previously unobserved data. This phase employs two types of incremental learning: local and global. Local incremental learning involves updating the pool of base classifiers by adding new members to this set. The new members are created with the Learn++ algorithm. Global incremental learning, in contrast, consists of updating the set of output profiles used during generalization. The proposed framework has been evaluated on a diversified set of databases. The results indicate that LoGID is promising. For most databases, the recognition rates achieved by the proposed method are higher than those achieved by other state-of-the-art approaches, such as batch learning. Furthermore, the simulated incremental learning setting demonstrates that LoGID can effectively improve the performance of systems created with small training sets as more data are observed over time.     

Human perspective to anomaly detection for cybersecurity

Traditionally signature-based network Intrusion Detection Systems (IDS) rely on inputs from domain experts and can only identify the attacks if they occur as individual event. IDS generate large number of alerts and it becomes very difficult for human users to go through each message. Previous researches have proposed analytics based approaches to analyze IDS alert patterns based on anomaly detection models, multi-steps models or probabilistic approaches. However, due to the complexities of network intrusions, it is impossible to develop all possible attack patterns or to avoid false positives. With the advance in technologies and popularity of networks in our daily life, it is becoming more and more difficult to detect network intrusions. However, no matter how rapid the technologies change, the human behaviors behind the cyber attacks stay relatively constant. This provides us an opportunity to develop an improved system to detect the unusual cyber attacks. In this paper, we developed four network intrusion models based on consideration of human factors. We then tested these models on ITOC Cyber Defense Competition (CDX) 2009 data. Our results are encouraging. These Models are not only able to recognize most network attacks identified by SNORT log alerts, they are also able to distinguish the non-attack network traffic that was potentially missed by SNORT as indicated by ground truth validation of the data.     

Adaptive fusion and co-operative training for classifier ensembles

In this paper, architectures and methods of decision aggregation in classifier ensembles are investigated. Typically, ensembles are designed in such a way that each classifier is trained independently and the decision fusion is performed as a post-process module. In this study, however, we are interested in making the fusion a more adaptive process. We first propose a new architecture that utilizes the features of a problem to guide the decision fusion process. By using both the features and classifiers outputs, the recognition strengths and weaknesses of the different classifiers are identified. This information is used to improve overall generalization capability of the system. Furthermore, we propose a co-operative training algorithm that allows the final classification to determine whether further training should be carried out on the components of the architecture. The performance of the proposed architecture is assessed by testing it on several benchmark problems. The new architecture shows improvement over existing aggregation techniques. Moreover, the proposed co-operative training algorithm provides a means to limit the users’ intervention, and maintains a level of accuracy that is competitive to that of most other approaches.     

Adaptive genetic algorithms applied to dynamic multiobjective problems

This paper describes an adaptive genetic algorithm (AGA) with dynamic fitness function for multiobjective problems (MOPs) in a dynamic environment. In order to see performance of the algorithm, AGA was applied to two kinds of MOPs. Firstly, the algorithm was used to find an optimal force allocation for a combat simulation. The paper discusses four objectives that need to be optimized and presents a fuzzy inference system that forms an aggregation of the four objectives. A second fuzzy inference system is used to control the crossover and mutation rates based on statistics of the aggregate fitness. In addition to dynamic force allocation optimization problem, a simple example of a dynamic multiobjective optimization problem taken from Farina et al. [M. Farina, K. Deb, P. Amato, Dynamic multiobjective optimization problems: test cases, approximations, and applications, IEEE Trans. Evol. Comput. 8 (5) (2004) 425–442] is presented and solved with the proposed algorithm. The results obtained here indicate that performance of the fuzzy-augmented GA is better than a standard GA method in terms of improvement of convergence to solutions of dynamic MOPs.     

Variational restricted Boltzmann machines to automated anomaly detection

Neural Computing and Applications - Data-driven methods are implemented using particularly complex scenarios that reflect in-depth perennial knowledge and research. Hence, the available intelligent...     

神经网络集成模型在入侵检测中的应用

入侵检测是网络安全研究中的热点。提出了一种用于入侵检测的神经网络集成模型。该模型采用神经网络集成分类技术,去除训练集中的冗余数据,利用遗传算法优化成员网络的权值,在此基础上训练成员网络,最终通过神经网络对成员网络的输出结果进行融合。理论和实验表明,模型具有较好的检测能力。     

Non-parametric bootstrap ensembles for detection of tumor lesions

We propose and assess a set of non-parametric ensembles, including bagging and boosting schemes, to recognize tumors in digital mammograms. Different approaches were examined as candidates for the two major components of the bagging ensembles, three spatial resampling schemes (residuals, centers and standardized centers), and four combination criteria (at least one, majority vote, top 25% models, and false discovery rate). A conversion to a classification problem prior to aggregation was employed for the boosting ensemble. The ensembles were compared at the lesion level against a single expert, and to a set of Markov Random Field (MRF) models in real images using three different criteria. The performance of the ensembles depended on its components, particularly the combination, with at least one and top 25% models offering a greater detection power independently of the type of lesion, and of the booststrapping scheme in a lesser degree. The ensembles were comparable in performance to MRFs in the unsupervised recognition of patterns exhibiting spatial structure.     

人工智能集成学习方法在入侵检测中的运用

在监督学习的分类中,集成学习已经成功地运用于许多不同的领域。文献中许多研究者通过考虑不同的组合方式、训练数据集、基分类器以及其他因素提出了不同的集成学习方法。人工智能技术相比于其他技术有许多优点,在解决入侵检测问题的集成学习发展中发挥着重要的作用。但是,目前还没有一篇综述性的文章来回顾解决入侵检测问题的通用集成方法和基于人工智能技术的集成学习方法。对入侵检测问题的集成学习方法进行对比和总结,并对该领域今后的研究方向进行了展望,为理解入侵检测系统领域的集成提供帮助。     

A hybrid machine learning approach to network anomaly detection

Zero-day cyber attacks such as worms and spy-ware are becoming increasingly widespread and dangerous. The existing signature-based intrusion detection mechanisms are often not sufficient in detecting these types of attacks. As a result, anomaly intrusion detection methods have been developed to cope with such attacks. Among the variety of anomaly detection approaches, the Support Vector Machine (SVM) is known to be one of the best machine learning algorithms to classify abnormal behaviors. The soft-margin SVM is one of the well-known basic SVM methods using supervised learning. However, it is not appropriate to use the soft-margin SVM method for detecting novel attacks in Internet traffic since it requires pre-acquired learning information for supervised learning procedure. Such pre-acquired learning information is divided into normal and attack traffic with labels separately. Furthermore, we apply the one-class SVM approach using unsupervised learning for detecting anomalies. This means one-class SVM does not require the labeled information. However, there is downside to using one-class SVM: it is difficult to use the one-class SVM in the real world, due to its high false positive rate. In this paper, we propose a new SVM approach, named Enhanced SVM, which combines these two methods in order to provide unsupervised learning and low false alarm capability, similar to that of a supervised SVM approach.We use the following additional techniques to improve the performance of the proposed approach (referred to as Anomaly Detector using Enhanced SVM): First, we create a profile of normal packets using Self-Organized Feature Map (SOFM), for SVM learning without pre-existing knowledge. Second, we use a packet filtering scheme based on Passive TCP/IP Fingerprinting (PTF), in order to reject incomplete network traffic that either violates the TCP/IP standard or generation policy inside of well-known platforms. Third, a feature selection technique using a Genetic Algorithm (GA) is used for extracting optimized information from raw internet packets. Fourth, we use the flow of packets based on temporal relationships during data preprocessing, for considering the temporal relationships among the inputs used in SVM learning. Lastly, we demonstrate the effectiveness of the Enhanced SVM approach using the above-mentioned techniques, such as SOFM, PTF, and GA on MIT Lincoln Lab datasets, and a live dataset captured from a real network. The experimental results are verified by m-fold cross validation, and the proposed approach is compared with real world Network Intrusion Detection Systems (NIDS).     

Temporal difference learning applied to sequential detection

This paper proposes a novel neural-network method for sequential detection, We first examine the optimal parametric sequential probability ratio test (SPRT) and make a simple equivalent transformation of the SPRT that makes it suitable for neural-network architectures. We then discuss how neural networks can learn the SPRT decision functions from observation data and labels. Conventional supervised learning algorithms have difficulties handling the variable length observation sequences, but a reinforcement learning algorithm, the temporal difference (TD) learning algorithm works ideally in training the neural network. The entire neural network is composed of context units followed by a feedforward neural network. The context units are necessary to store dynamic information that is needed to make good decisions. For an appropriate neural-network architecture, trained with independent and identically distributed (iid) observations by the TD learning algorithm, we show that the neural-network sequential detector can closely approximate the optimal SPRT with similar performance. The neural-network sequential detector has the additional advantage that it is a nonparametric detector that does not require probability density functions. Simulations demonstrated on iid Gaussian data show that the neural network and the SPRT have similar performance.