Ellipsoidal neighbourhood outlier factor for distributed anomaly detection in resource constrained networks

Anomaly detection in resource constrained wireless networks is an important challenge for tasks such as intrusion detection, quality assurance and event monitoring applications. The challenge is to detect these interesting events or anomalies in a timely manner, while minimising energy consumption in the network. We propose a distributed anomaly detection architecture, which uses multiple hyperellipsoidal clusters to model the data at each sensor node, and identify global and local anomalies in the network. In particular, a novel anomaly scoring method is proposed to provide a score for each hyperellipsoidal model, based on how remote the ellipsoid is relative to their neighbours. We demonstrate using several synthetic and real datasets that our proposed scheme achieves a higher detection performance with a significant reduction in communication overhead in the network compared to centralised and existing schemes.     

Local anomaly detection for mobile network monitoring

Huge amounts of operation data are constantly collected from various parts of communication networks. These data include measurements from the radio connections and system logs from servers. System operators and developers need robust, easy to use decision support tools based on these data. One of their key applications is to detect anomalous phenomena of the network. In this paper we present an anomaly detection method that describes the normal states of the system with a self-organizing map (SOM) identified from the data. Large deviation in the data samples from the SOM nodes is detected as anomalous behavior. Large deviation has traditionally been detected using global thresholds. If variation of the data occurs in separate parts of the data space, the global thresholds either fail to reveal anomalies or reveal false anomalies. Instead of one global threshold, we can use local thresholds, which depend on the local variation of the data. We also present a method to find an adaptive threshold using the distribution of the deviations. Our anomaly detection method can be used both in exploration of history data or comparison of unforeseen data against a data model derived from history data. It is applicable to wide range of processes that produce multivariate data. In this paper we present examples of this method applied to server log data and radio interface data from mobile networks.     

基于孤立点检测的自适应入侵检测技术研究

传统的入侵检测技术主要是从已知攻击数据中提取出每种具体攻击的特征规则模式,然后使用这些规则模式来进行匹配。然而基于规则的入侵检测的主要问题是现有的规则模式并不能有效应对持续变化的新型入侵攻击。针对这一问题,基于数据挖掘的入侵检测方法成为了入侵检测技术新的研究热点。本文提出了一种基于孤立点挖掘的自适应入侵检测框架,首先,基于相似系数寻找孤立点,然后对孤立点集合进行聚类,并使用改进的关联规则算法来从孤立点聚类结果中提取出各类入侵活动的潜在特征模式,然后生成可使用的匹配规则模式来添加到现有的规则模式中去,进而达到自适应的目的。本文使用KDD99的UCI数据集进行孤立点挖掘,然后使用IDS Snort的作为实验平台,使用IDS Informer模拟攻击工具进行测试,这两个实验结果表明了本文所提出算法的有效性。     

The importance of generalizability for anomaly detection

In security-related areas there is concern over novel “zero-day” attacks that penetrate system defenses and wreak havoc. The best methods for countering these threats are recognizing “nonself” as in an Artificial Immune System or recognizing “self” through clustering. For either case, the concern remains that something that appears similar to self could be missed. Given this situation, one could incorrectly assume that a preference for a tighter fit to self over generalizability is important for false positive reduction in this type of learning problem. This article confirms that in anomaly detection as in other forms of classification a tight fit, although important, does not supersede model generality. This is shown using three systems each with a different geometric bias in the decision space. The first two use spherical and ellipsoid clusters with a k-means algorithm modified to work on the one-class/blind classification problem. The third is based on wrapping the self points with a multidimensional convex hull (polytope) algorithm capable of learning disjunctive concepts via a thresholding constant. All three of these algorithms are tested using the Voting dataset from the UCI Machine Learning Repository, the MIT Lincoln Labs intrusion detection dataset, and the lossy-compressed steganalysis domain. Gilbert “Bert” Peterson is an Assistant Professor of Computer Engineering at the Air Force Institute of Technology. Dr. Peterson received a BS degree in Architecture, and an M.S. and Ph.D. in Computer Science at the University of Texas at Arlington. He teaches and conducts research in digital forensics and artificial intelligence. Brent McBride is a Communications and Information Systems officer in the United States Air Force. He received a B.S. in Computer Science from Brigham Young University and an M.S. in Computer Science from the Air Force Institute of Technology. He currently serves as Senior Software Engineer at the Air Force Wargaming Institute.     

On-line anomaly detection and resilience in classifier ensembles

Detection of anomalies is a broad field of study, which is applied in different areas such as data monitoring, navigation, and pattern recognition. In this paper we propose two measures to detect anomalous behaviors in an ensemble of classifiers by monitoring their decisions; one based on Mahalanobis distance and another based on information theory. These approaches are useful when an ensemble of classifiers is used and a decision is made by ordinary classifier fusion methods, while each classifier is devoted to monitor part of the environment. Upon detection of anomalous classifiers we propose a strategy that attempts to minimize adverse effects of faulty classifiers by excluding them from the ensemble. We applied this method to an artificial dataset and sensor-based human activity datasets, with different sensor configurations and two types of noise (additive and rotational on inertial sensors). We compared our method with two other well-known approaches, generalized likelihood ratio (GLR) and One-Class Support Vector Machine (OCSVM), which detect anomalies at data/feature level.     

Open data for anomaly detection in maritime surveillance

Maritime surveillance has received increased attention from a civilian perspective in recent years. Anomaly detection is one of many techniques available for improving the safety and security in this domain. Maritime authorities use confidential data sources for monitoring the maritime activities; however, a paradigm shift on the Internet has created new open sources of data. We investigate the potential of using open data as a complementary resource for anomaly detection in maritime surveillance. We present and evaluate a decision support system based on open data and expert rules for this purpose. We conduct a case study in which experts from the Swedish coastguard participate to conduct a real-world validation of the system. We conclude that the exploitation of open data as a complementary resource is feasible since our results indicate improvements in the efficiency and effectiveness of the existing surveillance systems by increasing the accuracy and covering unseen aspects of maritime activities.     

Combining sketches and wavelet analysis for multi time-scale network anomaly detection

With the rapid development and the increasing complexity of computer and communication systems and networks, traditional security technologies and measures can not meet the demand for integrated and dynamic security solutions. In this scenario, the use of Intrusion Detection Systems has emerged as a key element in network security.In this paper we address the problem proposing a wavelet-based technique able to detect network anomalies almost in real-time. In more detail, our approach is based on the combined use of sketches and wavelet analysis to reveal the anomalies in data collected at the router level. Moreover, to improve the detection rate we propose a multi time-scale analysis. The performance analysis, presented in this paper, demonstrates the effectiveness of the proposed method.     

Algorithms for anomaly detection of traces in logs of process aware information systems

This paper discusses four algorithms for detecting anomalies in logs of process aware systems. One of the algorithms only marks as potential anomalies traces that are infrequent in the log. The other three algorithms: threshold, iterative and sampling are based on mining a process model from the log, or a subset of it. The algorithms were evaluated on a set of 1500 artificial logs, with different profiles on the number of anomalous traces and the number of times each anomalous traces was present in the log. The sampling algorithm proved to be the most effective solution. We also applied the algorithm to a real log, and compared the resulting detected anomalous traces with the ones detected by a different procedure that relies on manual choices.     

Hierarchical crowd analysis and anomaly detection

ObjectiveThis work proposes a novel approach to model the spatiotemporal distribution of crowd motions and detect anomalous events.MethodsWe first learn the regions of interest (ROIs) which inform the behavioral patterns by trajectory analysis with Hierarchical Dirichlet Processes (HDP), so that the main trends of crowd motions can be modeled. Based on the ROIs, we then build a series of histograms both on global and local levels as the templates for the observed movement distribution, which statistically describes time-correlated crowd events. Once the template has been built hierarchically, we import real data containing the discrete trajectory observations from video surveillance and detect abnormal events for individuals and for crowds.ResultsExperimental results show the effectiveness of our approach, which is able to analyze and extract the crowd motion information from observed trajectory dataset, and achieve the anomaly detection at the hierarchical levels.ConclusionThe proposed hierarchical approach can learn the moving trends of crowd both in global and local area and describe the crowd behaviors in statistical way, which build a template for pedestrian movement distribution that allows for the detection of time-correlated abnormal crowd events.     

A program-based anomaly intrusion detection scheme using multiple detection engines and fuzzy inference

In this paper, a hybrid anomaly intrusion detection scheme using program system calls is proposed. In this scheme, a hidden Markov model (HMM) detection engine and a normal database detection engine have been combined to utilise their respective advantages. A fuzzy-based inference mechanism is used to infer a soft boundary between anomalous and normal behaviour, which is otherwise very difficult to determine when they overlap or are very close. To address the challenging issue of high cost in HMM training, an incremental HMM training with optimal initialization of HMM parameters is suggested. Experimental results show that the proposed fuzzy-based detection scheme can reduce false positive alarms by 48%, compared to the single normal database detection scheme. Our HMM incremental training with the optimal initialization produced a significant improvement in terms of training time and storage as well. The HMM training time was reduced by four times and the memory requirement was also reduced significantly.     

An autonomous labeling approach to support vector machines algorithms for network traffic anomaly detection

In the past years, several support vector machines (SVM) novelty detection approaches have been applied on the network intrusion detection field. The main advantage of these approaches is that they can characterize normal traffic even when trained with datasets containing not only normal traffic but also a number of attacks. Unfortunately, these algorithms seem to be accurate only when the normal traffic vastly outnumbers the number of attacks present in the dataset. A situation which can not be always hold.This work presents an approach for autonomous labeling of normal traffic as a way of dealing with situations where class distribution does not present the imbalance required for SVM algorithms. In this case, the autonomous labeling process is made by SNORT, a misuse-based intrusion detection system. Experiments conducted on the 1998 DARPA dataset show that the use of the proposed autonomous labeling approach not only outperforms existing SVM alternatives but also, under some attack distributions, obtains improvements over SNORT itself.     

基于混合式聚类算法的离群点挖掘在异常检测中的应用研究

为了提高异常检测系统的检测率,降低误警率,解决现有异常检测所存在的问题,将离群点挖掘技术应用到异常检测中,提出了一种基于混合式聚类算法的异常检测方法(NADHC)。该方法将基于距离的聚类算法与基于密度的聚类算法相结合从而形成新的混合聚类算法,通过k-中心点算法找出簇中心,进而去除隐蔽性较高的少量攻击行为样本,再将重复增加样本的方法结合基于密度的聚类算法计算出异常度,从而判断出异常行为。最后在KDD CUP 99数据集上进行实验仿真,验证了所提算法的可行性和有效性。     

Negative selection algorithm with constant detectors for anomaly detection

In the paper, two novel negative selection algorithms (NSAs) were proposed: FB-NSA and FFB-NSA. FB-NSA has two types of detectors: constant-sized detector (CFB-NSA) and variable-sized detector (VFB-NSA). The detectors of traditional NSA are generated randomly. Even for the same training samples, the position, size, and quantity of the detectors generated in each time are different. In order to eliminate the effect of training times on detectors, in the proposed approaches, detectors are generated in non-random ways. To determine the performances of the approaches, the experiments on 2-dimensional synthetic datasets, Iris dataset and ball bearing fault data were performed. Results show that FB-NSA and FFB-NSA outperforms the other anomaly detection methods in most cases. Besides, CFB-NSA can detect the abnormal degree of mechanical equipment. To determine the performances of CFB-NSA, the experiments on ball bearing fault data were performed. Results show that the abnormal degree based on the CFB-NSA can be used to diagnose the different fault types with the same fault degree, and the same fault type with the different fault degree.     

MILA – multilevel immune learning algorithm and its application to anomaly detection

T-cell-dependent humoral immune response is one of the more complex immunological events in the biological immune system, involving interaction of B cells with antigen (Ag) and their proliferation, differentiation and subsequent secretion of antibody (Ab). Inspired by these immunological principles, a Multilevel Immune Learning Algorithm (MILA) is proposed for novel pattern recognition. This paper describes the detailed background of MILA, and outlines its main features in different phases: Initialization phase, Recognition phase, Evolutionary phase and Response phase. Different test problems are studied and experimented with MILA for performance evaluation. The results show MILA is flexible and efficient in detecting anomalies and novel patterns.     

Multi-stage change-point detection scheme for large-scale simultaneous events

Change-point detection schemes, which represent one type of anomaly detection schemes, are a promising approach for detecting network anomalies, such as attacks and epidemics by unknown viruses and worms. These events are detected as change-points. However, the schemes generally also detect false-positive change-points caused by other events, such as improper parameter setting of detectors. Therefore there is a requirement for a scheme that detects only true-positive change-points caused by attacks and epidemics by unknown viruses and worms. The true-positive change-points tend to occur simultaneously and intensively in very large numbers, while the false-positive change-points tend to occur independently. Therefore, we expect that the multi-stage change-point detection scheme, which performs change-point detection in a distributed manner and takes account of the correlation among multiple change-points, can exclude false-positive change-points by neglecting those that occur independently. In this paper, we propose the multi-stage change-point detection scheme and introduce a weighting function that gives smaller weight to LDs with higher false-positive rate inferred by GD in order to avoid a set of false-positive alerts generated by the low-accuracy detectors from causing high false-positive rate of the scheme. We evaluate the performance of the scheme by a simulation using the parameter values obtained in an experiment using real random scan worms. In the evaluation, we modify AAWP (Analytical Active Worm Propagation) model so that it can derive the number of infected hosts (i.e., attack hosts) more accurately by considering a failure of infection behavior by random scan worms. The simulation results show that our scheme can achieve an optimal performance (detection rate of 1.0 and false-positive rate of 0) while the stand-alone change-point detection scheme, which does not use the correlation among multiple change-points, cannot attain such optimal performance, and our scheme with alert weighting always shows better detection performance than the scheme without alert weighting.     

异常检测测试平台的设计

提出了一种可以测试不同算法的异常检测测试平台.为适合大规模分布式网络,将网络分成不同网段,每个网段放置一个探测器IC,把不同IC提供的网络数据汇总至异常检测部件,在此进行异常分析,并根据分析结果对可能的入侵行为进行实时报警,其中的异常检测算法可以替换.最后,针对一种基于统计的异常检测算法进行了实验,并给出异常检测结果.     

Information fusion for anomaly detection with the dendritic cell algorithm

Dendritic cells are antigen presenting cells that provide a vital link between the innate and adaptive immune system, providing the initial detection of pathogenic invaders. Research into this family of cells has revealed that they perform information fusion which directs immune responses. We have derived a dendritic cell algorithm based on the functionality of these cells, by modelling the biological signals and differentiation pathways to build a control mechanism for an artificial immune system. We present algorithmic details in addition to experimental results, when the algorithm was applied to anomaly detection for the detection of port scans. The results show the dendritic cell algorithm is successful at detecting port scans.     

Support vector regression for anomaly detection from measurement histories

This research focuses on the analysis of measurements from distributed sensing of structures. The premise is that ambient temperature variations, and hence the temperature distribution across the structure, have a strong correlation with structural response and that this relationship could be exploited for anomaly detection. Specifically, this research first investigates whether support vector regression (SVR) models could be trained to capture the relationship between distributed temperature and response measurements and subsequently, if these models could be employed in an approach for anomaly detection. The study develops a methodology to generate SVR models that predict the thermal response of bridges from distributed temperature measurements, and evaluates its performance on measurement histories simulated using numerical models of a bridge girder. The potential use of these SVR models for damage detection is then studied by comparing their strain predictions with measurements collected from simulations of the bridge girder in damaged condition. Results show that SVR models that predict structural response from distributed temperature measurements could form the basis for a reliable anomaly detection methodology.     

A hybrid machine learning approach to network anomaly detection

Zero-day cyber attacks such as worms and spy-ware are becoming increasingly widespread and dangerous. The existing signature-based intrusion detection mechanisms are often not sufficient in detecting these types of attacks. As a result, anomaly intrusion detection methods have been developed to cope with such attacks. Among the variety of anomaly detection approaches, the Support Vector Machine (SVM) is known to be one of the best machine learning algorithms to classify abnormal behaviors. The soft-margin SVM is one of the well-known basic SVM methods using supervised learning. However, it is not appropriate to use the soft-margin SVM method for detecting novel attacks in Internet traffic since it requires pre-acquired learning information for supervised learning procedure. Such pre-acquired learning information is divided into normal and attack traffic with labels separately. Furthermore, we apply the one-class SVM approach using unsupervised learning for detecting anomalies. This means one-class SVM does not require the labeled information. However, there is downside to using one-class SVM: it is difficult to use the one-class SVM in the real world, due to its high false positive rate. In this paper, we propose a new SVM approach, named Enhanced SVM, which combines these two methods in order to provide unsupervised learning and low false alarm capability, similar to that of a supervised SVM approach.We use the following additional techniques to improve the performance of the proposed approach (referred to as Anomaly Detector using Enhanced SVM): First, we create a profile of normal packets using Self-Organized Feature Map (SOFM), for SVM learning without pre-existing knowledge. Second, we use a packet filtering scheme based on Passive TCP/IP Fingerprinting (PTF), in order to reject incomplete network traffic that either violates the TCP/IP standard or generation policy inside of well-known platforms. Third, a feature selection technique using a Genetic Algorithm (GA) is used for extracting optimized information from raw internet packets. Fourth, we use the flow of packets based on temporal relationships during data preprocessing, for considering the temporal relationships among the inputs used in SVM learning. Lastly, we demonstrate the effectiveness of the Enhanced SVM approach using the above-mentioned techniques, such as SOFM, PTF, and GA on MIT Lincoln Lab datasets, and a live dataset captured from a real network. The experimental results are verified by m-fold cross validation, and the proposed approach is compared with real world Network Intrusion Detection Systems (NIDS).