Ellipsoidal neighbourhood outlier factor for distributed anomaly detection in resource constrained networks

Anomaly detection in resource constrained wireless networks is an important challenge for tasks such as intrusion detection, quality assurance and event monitoring applications. The challenge is to detect these interesting events or anomalies in a timely manner, while minimising energy consumption in the network. We propose a distributed anomaly detection architecture, which uses multiple hyperellipsoidal clusters to model the data at each sensor node, and identify global and local anomalies in the network. In particular, a novel anomaly scoring method is proposed to provide a score for each hyperellipsoidal model, based on how remote the ellipsoid is relative to their neighbours. We demonstrate using several synthetic and real datasets that our proposed scheme achieves a higher detection performance with a significant reduction in communication overhead in the network compared to centralised and existing schemes.     

Local anomaly detection for mobile network monitoring

Huge amounts of operation data are constantly collected from various parts of communication networks. These data include measurements from the radio connections and system logs from servers. System operators and developers need robust, easy to use decision support tools based on these data. One of their key applications is to detect anomalous phenomena of the network. In this paper we present an anomaly detection method that describes the normal states of the system with a self-organizing map (SOM) identified from the data. Large deviation in the data samples from the SOM nodes is detected as anomalous behavior. Large deviation has traditionally been detected using global thresholds. If variation of the data occurs in separate parts of the data space, the global thresholds either fail to reveal anomalies or reveal false anomalies. Instead of one global threshold, we can use local thresholds, which depend on the local variation of the data. We also present a method to find an adaptive threshold using the distribution of the deviations. Our anomaly detection method can be used both in exploration of history data or comparison of unforeseen data against a data model derived from history data. It is applicable to wide range of processes that produce multivariate data. In this paper we present examples of this method applied to server log data and radio interface data from mobile networks.     

基于孤立点检测的自适应入侵检测技术研究

传统的入侵检测技术主要是从已知攻击数据中提取出每种具体攻击的特征规则模式,然后使用这些规则模式来进行匹配。然而基于规则的入侵检测的主要问题是现有的规则模式并不能有效应对持续变化的新型入侵攻击。针对这一问题,基于数据挖掘的入侵检测方法成为了入侵检测技术新的研究热点。本文提出了一种基于孤立点挖掘的自适应入侵检测框架,首先,基于相似系数寻找孤立点,然后对孤立点集合进行聚类,并使用改进的关联规则算法来从孤立点聚类结果中提取出各类入侵活动的潜在特征模式,然后生成可使用的匹配规则模式来添加到现有的规则模式中去,进而达到自适应的目的。本文使用KDD99的UCI数据集进行孤立点挖掘,然后使用IDS Snort的作为实验平台,使用IDS Informer模拟攻击工具进行测试,这两个实验结果表明了本文所提出算法的有效性。     

Workload-aware anomaly detection for Web applications

The failure of Web applications often affects a large population of customers, and leads to severe economic loss. Anomaly detection is essential for improving the reliability of Web applications. Current approaches model correlations among metrics, and detect anomalies when the correlations are broken. However, dynamic workloads cause the metric correlations to change over time. Moreover, modeling various metric correlations are difficult in complex Web applications. This paper addresses these problems and proposes an online anomaly detection approach for Web applications. We present an incremental clustering algorithm for training workload patterns online, and employ the local outlier factor (LOF) in the recognized workload pattern to detect anomalies. In addition, we locate the anomalous metrics with the Student's t-test method. We evaluated our approach on a testbed running the TPC-W industry-standard benchmark. The experimental results show that our approach is able to (1) capture workload fluctuations accurately, (2) detect typical faults effectively and (3) has advantages over two contemporary ones in accuracy.     

The importance of generalizability for anomaly detection

In security-related areas there is concern over novel “zero-day” attacks that penetrate system defenses and wreak havoc. The best methods for countering these threats are recognizing “nonself” as in an Artificial Immune System or recognizing “self” through clustering. For either case, the concern remains that something that appears similar to self could be missed. Given this situation, one could incorrectly assume that a preference for a tighter fit to self over generalizability is important for false positive reduction in this type of learning problem. This article confirms that in anomaly detection as in other forms of classification a tight fit, although important, does not supersede model generality. This is shown using three systems each with a different geometric bias in the decision space. The first two use spherical and ellipsoid clusters with a k-means algorithm modified to work on the one-class/blind classification problem. The third is based on wrapping the self points with a multidimensional convex hull (polytope) algorithm capable of learning disjunctive concepts via a thresholding constant. All three of these algorithms are tested using the Voting dataset from the UCI Machine Learning Repository, the MIT Lincoln Labs intrusion detection dataset, and the lossy-compressed steganalysis domain. Gilbert “Bert” Peterson is an Assistant Professor of Computer Engineering at the Air Force Institute of Technology. Dr. Peterson received a BS degree in Architecture, and an M.S. and Ph.D. in Computer Science at the University of Texas at Arlington. He teaches and conducts research in digital forensics and artificial intelligence. Brent McBride is a Communications and Information Systems officer in the United States Air Force. He received a B.S. in Computer Science from Brigham Young University and an M.S. in Computer Science from the Air Force Institute of Technology. He currently serves as Senior Software Engineer at the Air Force Wargaming Institute.     

Autoencoder-based anomaly detection for surface defect inspection

In this paper, the unsupervised autoencoder learning for automated defect detection in manufacturing is evaluated, where only the defect-free samples are required for the model training. The loss function of a Convolutional Autoencoder (CAE) model only aims at minimizing the reconstruction errors, and makes the representative features widely spread. The proposed CAE in this study incorporates a regularization that improves the feature distribution of defect-free samples within a tight range. It makes the representative feature vectors of all training samples as close as possible to the mean feature vector so that a defect sample in the evaluation stage can generate a distinct distance from the trained center of defect-free samples. The proposed CAE model with regularizations has been tested on a variety of material surfaces, including textural and patterned surfaces in images. The experimental results reveal that the proposed CAE with regularizations significantly outperforms the conventional CAE for defect detection applications in the industry.     

On-line anomaly detection and resilience in classifier ensembles

Detection of anomalies is a broad field of study, which is applied in different areas such as data monitoring, navigation, and pattern recognition. In this paper we propose two measures to detect anomalous behaviors in an ensemble of classifiers by monitoring their decisions; one based on Mahalanobis distance and another based on information theory. These approaches are useful when an ensemble of classifiers is used and a decision is made by ordinary classifier fusion methods, while each classifier is devoted to monitor part of the environment. Upon detection of anomalous classifiers we propose a strategy that attempts to minimize adverse effects of faulty classifiers by excluding them from the ensemble. We applied this method to an artificial dataset and sensor-based human activity datasets, with different sensor configurations and two types of noise (additive and rotational on inertial sensors). We compared our method with two other well-known approaches, generalized likelihood ratio (GLR) and One-Class Support Vector Machine (OCSVM), which detect anomalies at data/feature level.     

Open data for anomaly detection in maritime surveillance

Maritime surveillance has received increased attention from a civilian perspective in recent years. Anomaly detection is one of many techniques available for improving the safety and security in this domain. Maritime authorities use confidential data sources for monitoring the maritime activities; however, a paradigm shift on the Internet has created new open sources of data. We investigate the potential of using open data as a complementary resource for anomaly detection in maritime surveillance. We present and evaluate a decision support system based on open data and expert rules for this purpose. We conduct a case study in which experts from the Swedish coastguard participate to conduct a real-world validation of the system. We conclude that the exploitation of open data as a complementary resource is feasible since our results indicate improvements in the efficiency and effectiveness of the existing surveillance systems by increasing the accuracy and covering unseen aspects of maritime activities.     

Combining sketches and wavelet analysis for multi time-scale network anomaly detection

With the rapid development and the increasing complexity of computer and communication systems and networks, traditional security technologies and measures can not meet the demand for integrated and dynamic security solutions. In this scenario, the use of Intrusion Detection Systems has emerged as a key element in network security.In this paper we address the problem proposing a wavelet-based technique able to detect network anomalies almost in real-time. In more detail, our approach is based on the combined use of sketches and wavelet analysis to reveal the anomalies in data collected at the router level. Moreover, to improve the detection rate we propose a multi time-scale analysis. The performance analysis, presented in this paper, demonstrates the effectiveness of the proposed method.     

一种基于ARIMA模型与3 σ准则的取水异常检测方法

为提高取水预测数据的准确性,针对现有部分取水数据异常且难以进行人工判别的问题,提出一种基于ARIMA模型与3σ 准则的取水异常检测方法.分析每个取水点每年的日取水量的时间序列数据,使用时间序列的ARIMA模型和高斯分布的3σ 准则判断日取水量是否为异常值;通过时间序列分解算法分析异常值附近取水点的趋势,判断异常值附近是...     

Algorithms for anomaly detection of traces in logs of process aware information systems

This paper discusses four algorithms for detecting anomalies in logs of process aware systems. One of the algorithms only marks as potential anomalies traces that are infrequent in the log. The other three algorithms: threshold, iterative and sampling are based on mining a process model from the log, or a subset of it. The algorithms were evaluated on a set of 1500 artificial logs, with different profiles on the number of anomalous traces and the number of times each anomalous traces was present in the log. The sampling algorithm proved to be the most effective solution. We also applied the algorithm to a real log, and compared the resulting detected anomalous traces with the ones detected by a different procedure that relies on manual choices.     

基于混合式聚类算法的离群点挖掘在异常检测中的应用研究

为了提高异常检测系统的检测率,降低误警率,解决现有异常检测所存在的问题,将离群点挖掘技术应用到异常检测中,提出了一种基于混合式聚类算法的异常检测方法(NADHC)。该方法将基于距离的聚类算法与基于密度的聚类算法相结合从而形成新的混合聚类算法,通过k-中心点算法找出簇中心,进而去除隐蔽性较高的少量攻击行为样本,再将重复增加样本的方法结合基于密度的聚类算法计算出异常度,从而判断出异常行为。最后在KDD CUP 99数据集上进行实验仿真,验证了所提算法的可行性和有效性。     

A program-based anomaly intrusion detection scheme using multiple detection engines and fuzzy inference

In this paper, a hybrid anomaly intrusion detection scheme using program system calls is proposed. In this scheme, a hidden Markov model (HMM) detection engine and a normal database detection engine have been combined to utilise their respective advantages. A fuzzy-based inference mechanism is used to infer a soft boundary between anomalous and normal behaviour, which is otherwise very difficult to determine when they overlap or are very close. To address the challenging issue of high cost in HMM training, an incremental HMM training with optimal initialization of HMM parameters is suggested. Experimental results show that the proposed fuzzy-based detection scheme can reduce false positive alarms by 48%, compared to the single normal database detection scheme. Our HMM incremental training with the optimal initialization produced a significant improvement in terms of training time and storage as well. The HMM training time was reduced by four times and the memory requirement was also reduced significantly.     

Generative Adversarial Networks for anomaly detection on decentralised data

Generative Adversarial Networks (GANs) have seen great research interest in recent years, due to both their ability to represent structure in data and generate novel samples. Anomaly detection, which discerns novel samples or patterns, is a well-known problem that can be studied using GANs with a fresh perspective, especially in novel application domains such as wireless communication networks. For these models to achieve an accurate representation of the underlying data distribution, significant volumes of data are required. If this data source is not centralised (e.g. stored at multiple hosts or data centres), non-standard training methods are required to achieve comparable performance to the centralised case. This paper presents the key collaborative training methods that have emerged in recent years that draw on the GAN’s modular structure to achieve high performance while balancing computation, storage, and communication requirements and demonstrates their application to the task of anomaly detection using cognitive radios.     

Hierarchical crowd analysis and anomaly detection

ObjectiveThis work proposes a novel approach to model the spatiotemporal distribution of crowd motions and detect anomalous events.MethodsWe first learn the regions of interest (ROIs) which inform the behavioral patterns by trajectory analysis with Hierarchical Dirichlet Processes (HDP), so that the main trends of crowd motions can be modeled. Based on the ROIs, we then build a series of histograms both on global and local levels as the templates for the observed movement distribution, which statistically describes time-correlated crowd events. Once the template has been built hierarchically, we import real data containing the discrete trajectory observations from video surveillance and detect abnormal events for individuals and for crowds.ResultsExperimental results show the effectiveness of our approach, which is able to analyze and extract the crowd motion information from observed trajectory dataset, and achieve the anomaly detection at the hierarchical levels.ConclusionThe proposed hierarchical approach can learn the moving trends of crowd both in global and local area and describe the crowd behaviors in statistical way, which build a template for pedestrian movement distribution that allows for the detection of time-correlated abnormal crowd events.     

An autonomous labeling approach to support vector machines algorithms for network traffic anomaly detection

In the past years, several support vector machines (SVM) novelty detection approaches have been applied on the network intrusion detection field. The main advantage of these approaches is that they can characterize normal traffic even when trained with datasets containing not only normal traffic but also a number of attacks. Unfortunately, these algorithms seem to be accurate only when the normal traffic vastly outnumbers the number of attacks present in the dataset. A situation which can not be always hold.This work presents an approach for autonomous labeling of normal traffic as a way of dealing with situations where class distribution does not present the imbalance required for SVM algorithms. In this case, the autonomous labeling process is made by SNORT, a misuse-based intrusion detection system. Experiments conducted on the 1998 DARPA dataset show that the use of the proposed autonomous labeling approach not only outperforms existing SVM alternatives but also, under some attack distributions, obtains improvements over SNORT itself.     

Negative selection algorithm with constant detectors for anomaly detection

In the paper, two novel negative selection algorithms (NSAs) were proposed: FB-NSA and FFB-NSA. FB-NSA has two types of detectors: constant-sized detector (CFB-NSA) and variable-sized detector (VFB-NSA). The detectors of traditional NSA are generated randomly. Even for the same training samples, the position, size, and quantity of the detectors generated in each time are different. In order to eliminate the effect of training times on detectors, in the proposed approaches, detectors are generated in non-random ways. To determine the performances of the approaches, the experiments on 2-dimensional synthetic datasets, Iris dataset and ball bearing fault data were performed. Results show that FB-NSA and FFB-NSA outperforms the other anomaly detection methods in most cases. Besides, CFB-NSA can detect the abnormal degree of mechanical equipment. To determine the performances of CFB-NSA, the experiments on ball bearing fault data were performed. Results show that the abnormal degree based on the CFB-NSA can be used to diagnose the different fault types with the same fault degree, and the same fault type with the different fault degree.     

Power-aware anomaly detection in smartphones: An analysis of on-platform versus externalized operation

Many security problems in smartphones and other smart devices are approached from an anomaly detection perspective in which the main goal reduces to identifying anomalous activity patterns. Since machine learning algorithms are generally used to build such detectors, one major challenge is adapting these techniques to battery-powered devices. Many recent works simply assume that on-platform detection is prohibitive and suggest using offloaded (i.e., cloud-based) engines. Such a strategy seeks to save battery life by exchanging computation and communication costs, but it still remains unclear whether this is optimal or not in all circumstances. In this paper, we evaluate different strategies for offloading certain functional tasks in machine learning based detection systems. Our experimental results confirm the intuition that outsourced computation is clearly the best option in terms of power consumption, outweighing on-platform strategies in, essentially, all practical scenarios. Our findings also point out noticeable differences among different machine learning algorithms, and we provide separate consumption models for functional blocks (data preprocessing, training, test, and communications) that can be used to obtain power consumption estimates and compare detectors.     

MILA – multilevel immune learning algorithm and its application to anomaly detection

T-cell-dependent humoral immune response is one of the more complex immunological events in the biological immune system, involving interaction of B cells with antigen (Ag) and their proliferation, differentiation and subsequent secretion of antibody (Ab). Inspired by these immunological principles, a Multilevel Immune Learning Algorithm (MILA) is proposed for novel pattern recognition. This paper describes the detailed background of MILA, and outlines its main features in different phases: Initialization phase, Recognition phase, Evolutionary phase and Response phase. Different test problems are studied and experimented with MILA for performance evaluation. The results show MILA is flexible and efficient in detecting anomalies and novel patterns.