Scalable anomaly detection in large homogeneous populations

Anomaly detection in large populations is a challenging but highly relevant problem. It is essentially a multi-hypothesis problem, with a hypothesis for every division of the systems into normal and anomalous systems. The number of hypothesis grows rapidly with the number of systems and approximate solutions become a necessity for any problem of practical interest. In this paper we take an optimization approach to this multi-hypothesis problem. It is first shown to be equivalent to a non-convex combinatorial optimization problem and then is relaxed to a convex optimization problem that can be solved distributively on the systems and that stays computationally tractable as the number of systems increase. An interesting property of the proposed method is that it can under certain conditions be shown to give exactly the same result as the combinatorial multi-hypothesis problem and the relaxation is hence tight.     

基于孤立点检测的自适应入侵检测技术研究

传统的入侵检测技术主要是从已知攻击数据中提取出每种具体攻击的特征规则模式,然后使用这些规则模式来进行匹配。然而基于规则的入侵检测的主要问题是现有的规则模式并不能有效应对持续变化的新型入侵攻击。针对这一问题,基于数据挖掘的入侵检测方法成为了入侵检测技术新的研究热点。本文提出了一种基于孤立点挖掘的自适应入侵检测框架,首先,基于相似系数寻找孤立点,然后对孤立点集合进行聚类,并使用改进的关联规则算法来从孤立点聚类结果中提取出各类入侵活动的潜在特征模式,然后生成可使用的匹配规则模式来添加到现有的规则模式中去,进而达到自适应的目的。本文使用KDD99的UCI数据集进行孤立点挖掘,然后使用IDS Snort的作为实验平台,使用IDS Informer模拟攻击工具进行测试,这两个实验结果表明了本文所提出算法的有效性。     

Distributed anomaly detection for industrial wireless sensor networks based on fuzzy data modelling

Modern infrastructure increasingly depends on large computerized systems for their reliable operation. Supervisory Control and Data Acquisition (SCADA) systems are being deployed to monitor and control large scale distributed infrastructures (e.g. power plants, water distribution systems). A recent trend is to incorporate Wireless Sensor Networks (WSNs) to sense and gather data. However, due to the broadcast nature of the network and inherent limitations in the sensor nodes themselves, they are vulnerable to different types of security attacks. Given the critical aspects of the underlying infrastructure it is an extremely important research challenge to provide effective methods to detect malicious activities on these networks. This paper proposes a robust and scalable mechanism that aims to detect malicious anomalies accurately and efficiently using distributed in-network processing in a hierarchical framework. Unsupervised data partitioning is performed distributively adapting fuzzy c-means clustering in an incremental model. Non-parametric and non-probabilistic anomaly detection is performed through fuzzy membership evaluations and thresholds on observed inter-cluster distances. Robust thresholds are determined adaptively using second order statistical knowledge at each evaluation stage. Extensive experiments were performed and the results demonstrate that the proposed framework achieves high detection accuracy compared to existing data clustering approaches with more than 96% less communication overheads opposed to a centralized approach.     

Workload-aware anomaly detection for Web applications

The failure of Web applications often affects a large population of customers, and leads to severe economic loss. Anomaly detection is essential for improving the reliability of Web applications. Current approaches model correlations among metrics, and detect anomalies when the correlations are broken. However, dynamic workloads cause the metric correlations to change over time. Moreover, modeling various metric correlations are difficult in complex Web applications. This paper addresses these problems and proposes an online anomaly detection approach for Web applications. We present an incremental clustering algorithm for training workload patterns online, and employ the local outlier factor (LOF) in the recognized workload pattern to detect anomalies. In addition, we locate the anomalous metrics with the Student's t-test method. We evaluated our approach on a testbed running the TPC-W industry-standard benchmark. The experimental results show that our approach is able to (1) capture workload fluctuations accurately, (2) detect typical faults effectively and (3) has advantages over two contemporary ones in accuracy.     

Research on outlier detection algorithm for express logistics

To detect the problems of time delay, path error and destination error in express logistics process effectively, a novel outlier detection algorithm for express logistics is proposed in this paper. To test the detection results, the express logistics system operating model is built to test the detection results. Experiment results show that the proposed algorithm is well applied to the express logistics data with multi-attribute characteristics, and can work well in detecting the abnormal conditions of express logistics.     

Securing distributed data storage and retrieval in sensor networks

Sensor networks have been an attractive platform for pervasive computing and communication. However, they are vulnerable to attacks if deployed in hostile environments. The past research of sensor network security has focused on securing information in communication, but how to secure information in storage has been overlooked. Meanwhile, distributed data storage and retrieval have become popular for efficient data management in sensor networks, which renders the absence of schemes for securing stored information to be a more severe problem. Therefore, we propose three evolutionary schemes, namely, the simple hash-based (SHB) scheme, the enhanced hash-based (EHB) scheme, and the adaptive polynomial-based (APB) scheme, to deal with the problem. All the schemes have the properties that only authorized entities can access data stored in the sensor network, and the schemes are resilient to a large number of sensor node compromises. The EHB and the APB schemes do not involve any centralized entity except for a few initialization or renewal operations, and thus support secure, distributed data storage and retrieval. The APB scheme further provides high scalability and flexibility, and hence is the most suitable among the three schemes for real applications. The schemes were evaluated through extensive analysis and TOSSIM-based simulations.     

基于隐私保护的分布式异常检测方法

为获得鲁棒性的全局异常检测模型,需要多个组织之间的知识共享.存在的分布式异常检测技术常基于原始数据的交换或共享,侵犯了各自的隐私权,令人难以接受.基于隐私保护的分布式异常检测方法,采用本地模型共享技术,在保证数据隐私性的同时完成全局异常检测任务.通过7种异常检测模型在仿真和真实数据集上的实验说明,所提出的方法在保护数据隐私的同时,其全局异常检测效果能接近甚至超过将所有数据集中后建立的全局模型.     

Generative Adversarial Networks for anomaly detection on decentralised data

Generative Adversarial Networks (GANs) have seen great research interest in recent years, due to both their ability to represent structure in data and generate novel samples. Anomaly detection, which discerns novel samples or patterns, is a well-known problem that can be studied using GANs with a fresh perspective, especially in novel application domains such as wireless communication networks. For these models to achieve an accurate representation of the underlying data distribution, significant volumes of data are required. If this data source is not centralised (e.g. stored at multiple hosts or data centres), non-standard training methods are required to achieve comparable performance to the centralised case. This paper presents the key collaborative training methods that have emerged in recent years that draw on the GAN’s modular structure to achieve high performance while balancing computation, storage, and communication requirements and demonstrates their application to the task of anomaly detection using cognitive radios.     

An effective and efficient algorithm for high-dimensional outlier detection

The outlier detection problem has important applications in the field of fraud detection, network robustness analysis, and intrusion detection. Most such applications are most important for high-dimensional domains in which the data can contain hundreds of dimensions. Many recent algorithms have been proposed for outlier detection that use several concepts of proximity in order to find the outliers based on their relationship to the other points in the data. However, in high-dimensional space, the data are sparse and concepts using the notion of proximity fail to retain their effectiveness. In fact, the sparsity of high-dimensional data can be understood in a different way so as to imply that every point is an equally good outlier from the perspective of distance-based definitions. Consequently, for high-dimensional data, the notion of finding meaningful outliers becomes substantially more complex and nonobvious. In this paper, we discuss new techniques for outlier detection that find the outliers by studying the behavior of projections from the data set.Received: 19 November 2002, Accepted: 6 February 2004, Published online: 19 August 2004Edited by: R. Ng.     

Genetic algorithms for outlier detection and variable selection in linear regression models

This article addresses some problems in outlier detection and variable selection in linear regression models. First, in outlier detection there are problems known as smearing and masking. Smearing means that one outlier makes another, non-outlier observation appear as an outlier, and masking that one outlier prevents another one from being detected. Detecting outliers one by one may therefore give misleading results. In this article a genetic algorithm is presented which considers different possible groupings of the data into outlier and non-outlier observations. In this way all outliers are detected at the same time. Second, it is known that outlier detection and variable selection can influence each other, and that different results may be obtained, depending on the order in which these two tasks are performed. It may therefore be useful to consider these tasks simultaneously, and a genetic algorithm for a simultaneous outlier detection and variable selection is suggested. Two real data sets are used to illustrate the algorithms, which are shown to work well. In addition, the scalability of the algorithms is considered with an experiment using generated data.I would like to thank Dr Tero Aittokallio and an anonymous referee for useful comments.     

Local anomaly detection for mobile network monitoring

Huge amounts of operation data are constantly collected from various parts of communication networks. These data include measurements from the radio connections and system logs from servers. System operators and developers need robust, easy to use decision support tools based on these data. One of their key applications is to detect anomalous phenomena of the network. In this paper we present an anomaly detection method that describes the normal states of the system with a self-organizing map (SOM) identified from the data. Large deviation in the data samples from the SOM nodes is detected as anomalous behavior. Large deviation has traditionally been detected using global thresholds. If variation of the data occurs in separate parts of the data space, the global thresholds either fail to reveal anomalies or reveal false anomalies. Instead of one global threshold, we can use local thresholds, which depend on the local variation of the data. We also present a method to find an adaptive threshold using the distribution of the deviations. Our anomaly detection method can be used both in exploration of history data or comparison of unforeseen data against a data model derived from history data. It is applicable to wide range of processes that produce multivariate data. In this paper we present examples of this method applied to server log data and radio interface data from mobile networks.     

Intelligent management of distributed dynamic sensor networks

A universal solution for the management of dynamic sensor networks will be presented, covering both networking and application layers. A network of intelligent modules, overlaying the sensor network, collectively interprets mission scenarios in a special high-level language, which can start from any nodes and cover the network at runtime. The spreading scenarios are extremely compact, which may be useful for energy-saving communications. The code will be exhibited for distributed collection and fusion of sensor data, and also for tracking mobile targets by scattered and communicating sensors. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

Smart anomaly detection in sensor systems: A multi-perspective review

Anomaly detection is concerned with identifying data patterns that deviate remarkably from the expected behavior. This is an important research problem, due to its broad set of application domains, from data analysis to e-health, cybersecurity, predictive maintenance, fault prevention, and industrial automation. Herein, we review state-of-the-art methods that may be employed to detect anomalies in the specific area of sensor systems, which poses hard challenges in terms of information fusion, data volumes, data speed, and network/energy efficiency, to mention but the most pressing ones. In this context, anomaly detection is a particularly hard problem, given the need to find computing-energy-accuracy trade-offs in a constrained environment. We taxonomize methods ranging from conventional techniques (statistical methods, time-series analysis, signal processing, etc.) to data-driven techniques (supervised learning, reinforcement learning, deep learning, etc.). We also look at the impact that different architectural environments (Cloud, Fog, Edge) can have on the sensors ecosystem. The review points to the most promising intelligent-sensing methods, and pinpoints a set of interesting open issues and challenges.     

A survey of outlier detection in high dimensional data streams

The rapid evolution of technology has led to the generation of high dimensional data streams in a wide range of fields, such as genomics, signal processing, and finance. The combination of the streaming scenario and high dimensionality is particularly challenging especially for the outlier detection task. This is due to the special characteristics of the data stream such as the concept drift, the limited time and space requirements, in addition to the impact of the well-known curse of dimensionality in high dimensional space. To the best of our knowledge, few studies have addressed these challenges simultaneously, and therefore detecting anomalies in this context requires a great deal of attention. The main objective of this work is to study the main approaches existing in the literature, to identify a set of comparison criteria, such as the computational cost and the interpretation of outliers, which will help us to reveal the different challenges and additional research directions associated with this problem. At the end of this study, we will draw up a summary report which summarizes the main limits identified and we will detail the different directions of research related to this issue in order to promote research for this community.     

移动社交网络异常签到在线检测算法

随着智能手机、Pad等智能移动设备的广泛普及,移动社交网络的应用得到了快速发展。本文针对移动社交网络中用户异常签到位置检测问题,提出了一类基于用户移动行为特征的异常签到在线检测方法。首先,在基于距离的异常模型基础上,提出了基于历史位置（H-Outlier）和基于好友圈（F-Outlier）两种异常签到模型;然后,针对H-Outlier提出了一种优化的检测算法H-Opt,利用所提的签到状态模型与优化的邻居搜索机制降低检测时间;针对F-Outlier提出了一种基于触发的优化检测算法F-Opt,将连续的在线异常检测转化成了基于触发的异常检测方式;最后,在真实的移动社交网络用户签到数据集上,验证了所提算法的有效性。实验结果显示,F-Opt显著降低了H-Opt的异常检测错误率;同时,相比于LUE算法,F-Opt和H-Opt的效率分别平均提升了2.34倍和2.45倍。     

The importance of generalizability for anomaly detection

In security-related areas there is concern over novel “zero-day” attacks that penetrate system defenses and wreak havoc. The best methods for countering these threats are recognizing “nonself” as in an Artificial Immune System or recognizing “self” through clustering. For either case, the concern remains that something that appears similar to self could be missed. Given this situation, one could incorrectly assume that a preference for a tighter fit to self over generalizability is important for false positive reduction in this type of learning problem. This article confirms that in anomaly detection as in other forms of classification a tight fit, although important, does not supersede model generality. This is shown using three systems each with a different geometric bias in the decision space. The first two use spherical and ellipsoid clusters with a k-means algorithm modified to work on the one-class/blind classification problem. The third is based on wrapping the self points with a multidimensional convex hull (polytope) algorithm capable of learning disjunctive concepts via a thresholding constant. All three of these algorithms are tested using the Voting dataset from the UCI Machine Learning Repository, the MIT Lincoln Labs intrusion detection dataset, and the lossy-compressed steganalysis domain. Gilbert “Bert” Peterson is an Assistant Professor of Computer Engineering at the Air Force Institute of Technology. Dr. Peterson received a BS degree in Architecture, and an M.S. and Ph.D. in Computer Science at the University of Texas at Arlington. He teaches and conducts research in digital forensics and artificial intelligence. Brent McBride is a Communications and Information Systems officer in the United States Air Force. He received a B.S. in Computer Science from Brigham Young University and an M.S. in Computer Science from the Air Force Institute of Technology. He currently serves as Senior Software Engineer at the Air Force Wargaming Institute.     

Autoencoder-based anomaly detection for surface defect inspection

In this paper, the unsupervised autoencoder learning for automated defect detection in manufacturing is evaluated, where only the defect-free samples are required for the model training. The loss function of a Convolutional Autoencoder (CAE) model only aims at minimizing the reconstruction errors, and makes the representative features widely spread. The proposed CAE in this study incorporates a regularization that improves the feature distribution of defect-free samples within a tight range. It makes the representative feature vectors of all training samples as close as possible to the mean feature vector so that a defect sample in the evaluation stage can generate a distinct distance from the trained center of defect-free samples. The proposed CAE model with regularizations has been tested on a variety of material surfaces, including textural and patterned surfaces in images. The experimental results reveal that the proposed CAE with regularizations significantly outperforms the conventional CAE for defect detection applications in the industry.     

一种分布式CPS异常检测的无监督图模型

在分布式信息物理融合系统（CPS）中,由于各子系统间的强耦合性,常常会因为故障的传播导致整个系统的物理故障和网络异常。针对这一问题,提出了一种新的基于数据驱动的框架用于检测系统范围内的异常。该框架是用于发现和表征CPS各个子系统间相互作用的一种基于符号动力学的时空特征提取方案,并将提取的特征通过受限玻尔兹曼机（RBM）学习到一个系统级的模型。实验结果表明,该框架可以通过一个图模型捕获CPS的多模态,同时可用于异常检测。