Improving PCA‐based anomaly detection by using multiple time scale analysis and Kullback–Leibler divergence

The increasing number of network attacks causes growing problems for network operators and users. Thus, detecting anomalous traffic is of primary interest in IP networks management. In this paper, we address the problem considering a method based on PCA for detecting network anomalies. In more detail, this paper presents a new technique that extends the state of the art in PCA‐based anomaly detection. Indeed, by means of multi‐scale analysis and Kullback–Leibler divergence, we are able to obtain great improvements with respect to the performance of the ‘classical’ approach. Moreover, we also introduce a method for identifying the flows responsible for an anomaly detected at the aggregated level. The performance analysis, presented in this paper, demonstrates the effectiveness of the proposed method.Copyright © 2012 John Wiley & Sons, Ltd.     

Denial of service detection using dynamic time warping

With the rapid growth of security threats in computer networks, the need for developing efficient security-warning systems is substantially increasing. Distributed denial-of-service (DDoS) and DoS attacks are still among the most effective and dreadful attacks that require robust detection. In this work, we propose a new method to detect TCP DoS/DDoS attacks. Since analyzing network traffic is a promising approach, our proposed method utilizes network traffic by decomposing the TCP traffic into control and data planes and exploiting the dynamic time warping (DTW) algorithm for aligning these two planes with respect to the minimum Euclidean distance. By demonstrating that the distance between the control and data planes is considerably small for benign traffic, we exploit this characteristic for detecting attacks as outliers. An adaptive thresholding scheme is implemented by adjusting the value of the threshold in accordance with the local statistics of the median absolute deviation (MAD) of the distances between the two planes. We demonstrate the efficacy of the proposed method for detecting DoS/DDoS attacks by analyzing traffic data obtained from publicly available datasets.     

Adaptive and automated detection of service anomalies intransaction-oriented WANs: network analysis, algorithms, implementation,and deployment

Algorithms and software for proactive and adaptive detection of network/service anomalies (i.e., performance degradations) have been developed, implemented, deployed, and field-tested for transaction-oriented wide area networks (WANs). A real-time anomaly detection system called TRISTAN (transaction instantaneous anomaly notification) has been implemented, and is deployed in the commercially important AT&T transaction access services (TAS) network. TAS is a high volume, multiple service classes, hybrid telecom and data WAN that services transaction traffic in the U.S. and neighboring countries. TRISTAN adaptively and preactively detects network/service performance anomalies in multiple-service-class-based and transaction-oriented networks, where performances of service classes are mutually dependent and correlated, where environmental factors (e.g., nonmanaged or nonmonitored equipment within customer premises) can strongly impact network and service performances. Specifically, TRISTAN implements algorithms that: 1) sample and convert raw transaction records to service-class based performance data in which potential network anomalies are highlighted; 2) automatically construct adaptive and service-class-based performance thresholds from historical transaction records for detecting network and service anomalies; and 3) perform real-time network/service anomaly detection. TRISTAN is demonstrated to be capable of proactively detecting network/service anomalies, which easily elude detection by the traditional alarm-based network monitoring systems     

基于奇异值分解更新的多元在线异常检测方法

网络异常检测对于保证网络稳定高效运行极为重要。基于主成分分析的全网络异常检测算法虽然具有很好的检测性能,但无法满足在线检测的要求。为了解决此问题,该文引入流量矩阵模型,提出了一种基于奇异值分解更新的多元在线异常检测算法MOADA-SVDU,该算法以增量的方式构建正常子空间和异常子空间,并实现网络流量异常的在线检测。理论分析表明与主成分分析算法相比,该算法具有更低的存储和计算开销。因特网实测的流量矩阵数据集以及模拟试验数据分析表明,该算法不仅实现了网络异常的在线检测,而且取得了很好的检测性能。     

基于云计算的海量网络流量数据分析处理及关键算法研究

为了应对日益增长的网络流量数据量和对网络安全的需求,提高网络流量数据的处理效率和准确性,文中从云计算架构出发,设计并搭建了一个能承载大规模网络流量数据处理的云计算平台。基于该平台,采用了分布式存储、并行计算和机器学习等技术,对海量网络流量数据的预处理、聚类分析、异常检测等关键环节进行了研究。结果表明,基于云计算的海量网络流量数据分析处理的关键算法取得了显著成果。通过分布式存储和并行计算技术,实现了对海量网络流量数据的高效读写和处理。在预处理阶段,针对流量数据进行采样和滤波,减少了数据量,并保留了关键特征。在聚类分析方面,利用机器学习算法实现了对网络流量的分类和统计,通过构建模型、训练和优化算法,实现了对网络攻击和异常行为的准确识别和及时报警。     

Image-Based Anomaly Detection Technique: Algorithm, Implementation and Effectiveness

The frequent and large-scale network attacks have led to an increased need for developing techniques for analyzing network traffic. This paper presents NetViewer, a network measurement approach that can simultaneously detect, identify, and visualize attacks and anomalous traffic in real-time by passively monitoring packet headers. We propose to represent samples of network packet header data as frames or images. With such a formulation, a series of samples can be seen as a sequence of frames or video, revealing certain kinds of attacks to the human eye. This enables techniques from image processing and video compression to be applied to the packet header data to reveal interesting properties of traffic. We show that “scene change analysis” can reveal sudden changes in traffic behavior or anomalies. We also show that “motion prediction” techniques can be employed to understand the patterns of some of the attacks. We show that it may be feasible to represent multiple pieces of data as different colors of an image enabling a uniform treatment of multidimensional packet header data. We compare the effectiveness of NetViewer with classical detection theory-based Neyman–Pearson test.     

基于深度特征学习的网络流量异常检测方法

针对网络流量异常检测过程中提取的流量特征准确性低、鲁棒性差导致流量攻击检测率低、误报率高等问题,该文结合堆叠降噪自编码器(SDA)和softmax,提出一种基于深度特征学习的网络流量异常检测方法。首先基于粒子群优化算法设计SDA结构两阶段寻优算法：根据流量检测准确率依次对隐藏层层数及每层节点数进行寻优,确定搜索空间中的最优SDA结构,从而提高SDA提取特征的准确性。然后采用小批量梯度下降算法对优化的SDA进行训练,通过最小化含噪数据重构向量与原始输入向量间的差异,提取具有较强鲁棒性的流量特征。最后基于提取的流量特征对softmax进行训练构建异常检测分类器,从而实现对流量攻击的高性能检测。实验结果表明：该文所提方法可根据实验数据及其分类任务动态调整SDA结构,提取的流量特征具有更高的准确性和鲁棒性,流量攻击检测率高、误报率低。

     

Anomaly detection in IP networks

Network anomaly detection is a vibrant research area. Researchers have approached this problem using various techniques such as artificial intelligence, machine learning, and state machine modeling. In this paper, we first review these anomaly detection methods and then describe in detail a statistical signal processing technique based on abrupt change detection. We show that this signal processing technique is effective at detecting several network anomalies. Case studies from real network data that demonstrate the power of the signal processing approach to network anomaly detection are presented. The application of signal processing techniques to this area is still in its infancy, and we believe that it has great potential to enhance the field, and thereby improve the reliability of IP networks.     

面向PCA异常检测器的毒害攻击和防御机制

网络流量异常检测对于保证网络稳定高效运行极为重要.目前基于主成分分析(PCA)的全网络异常检测算法虽然发挥了关键作用,但它还存在着受毒害攻击而失效的问题.为此,深入分析了毒害攻击的机制并对其进行了分类,提出了量化毒害流量的两个测度,并给出了3种新的毒害攻击机制;提出了一种基于健壮PCA的异常检测算法RPCA以抵御毒害攻...     

An Efficient Network Attack Visualization Using Security Quad and Cube

Security quad and cube (SQC) is a network attack analyzer that is capable of aggregating many different events into a single significant incident and visualizing these events in order to identify suspicious or illegitimate behavior. A network administrator recognizes network anomalies by analyzing the traffic data and alert messages generated in the security devices; however, it takes a lot of time to inspect and analyze them because the security devices generate an overwhelming amount of logs and security events. In this paper, we propose SQC, an efficient method for analyzing network security through visualization. The proposed method monitors anomalies occurring in an entire network and displays detailed information of the attacks. In addition, by providing a detailed analysis of network attacks, this method can more precisely detect and distinguish them from normal events.     

Multi-step attack detection method based on network communication anomaly recognition

In view of the characteristics of internal fixed business logic,inbound and outbound network access behavior,two classes and four kinds of abnormal behaviors were defined firstly,and then a multi-step attack detection method was proposed based on network communication anomaly recognition.For abnormal sub-graphs and abnormal communication edges detection,graph-based anomaly analysis and wavelet analysis method were respectively proposed to identify abnormal behaviors in network communication,and detect multi-step attacks through anomaly correlation analysis.Experiments are carried out on the DARPA 2000 data set and LANL data set to verify the results.The experimental results show that the proposed method can effectively detect and reconstruct multi-step attack scenarios.The proposed method can effectively monitor multi-step attacks including unknown feature types.It provides a feasible idea for detecting complex multi-step attack patterns such as APT.And the network communication graph greatly reduces the data size,it is suitable for large-scale enterprise network environments.     

Real-time detection of traffic anomalies in wireless mesh networks

Anomaly detection is emerging as a necessary component as wireless networks gain popularity. Anomaly detection has been addressed broadly in wired networks and powerful methods have been developed for correct detection of a variety of known attacks and other anomalies. In this paper, we propose a real-time anomaly detection and identification scheme for wireless mesh networks (WMN) using components from previous methods developed for wired networks. Experiments over a WMN testbed show the effectiveness of the proposed scheme in isolating different types of anomalies, such as Denial-of-service attacks, port scan attacks, etc. Our scheme uses Chi-square statistics and it is based on similar ideas as the scheme presented by Lakhina et al. although it has lower computational complexity. The original method by Lakhina et al. was developed for wired networks and used Principal Component Analysis (PCA) for reducing the dimensions of observed data and Hotelling’s t ² statistics to distinguish between normal and abnormal traffic conditions. However, in our studies we found that dimension reduction is the most computationally intensive process of the scheme. In this paper we propose an alternative way of reducing dimensions using flow variances in a Chi-square test. Experimental results show that the Chi-square test performs similarly well to the PCA-based method at merely a fraction of the computations. Moreover, we propose an automatic identification scheme to pin-point the cause of the detected anomaly and its contribution in terms of additional or lack of traffic. Our results and comparison with other statistical tools show that the Chi-square test and the PCA-based method with identification scheme make powerful tools for real-time detection of various anomalies in an interference prone wireless networking environment.     

Spatio-Temporal Network Anomaly Detection by Assessing Deviations of Empirical Measures

We introduce an Internet traffic anomaly detection mechanism based on large deviations results for empirical measures. Using past traffic traces we characterize network traffic during various time-of-day intervals, assuming that it is anomaly-free. We present two different approaches to characterize traffic: (i) a model-free approach based on the method of types and Sanov's theorem, and (ii) a model-based approach modeling traffic using a Markov modulated process. Using these characterizations as a reference we continuously monitor traffic and employ large deviations and decision theory results to ldquocomparerdquo the empirical measure of the monitored traffic with the corresponding reference characterization, thus, identifying traffic anomalies in real-time. Our experimental results show that applying our methodology (even short-lived) anomalies are identified within a small number of observations. Throughout, we compare the two approaches presenting their advantages and disadvantages to identify and classify temporal network anomalies. We also demonstrate how our framework can be used to monitor traffic from multiple network elements in order to identify both spatial and temporal anomalies. We validate our techniques by analyzing real traffic traces with time-stamped anomalies.     

基于AR模型的网络异常检测

在网络流量管理中流量异常的一般检测方法是阈值监控，文章提出一种新的异常检测方法，选取适当的SNMP管理信息库变量，建立对相关变量的局部AR（自回归）模型，检测并分析一种服务器故障引起的流量异常，获得该故障的特征向量模型；该检测方法比阈值方法有更强的检测功能，并与传统GLR测试方法进行对比。     

ODC：在线检测和分类全网络流量异常的方法

提出一种从全网络的视角实时在线检测和分类流量异常的方法(简称ODC),该方法以增量方式构建以流量特征的熵为测度的流量矩阵,利用增量主成分分析算法在线地检测流量异常,然后再利用增量k-means算法实时在线地对流量异常进行分类,以便网络管理员采取相应的防御措施。理论分析和实验分析表明,ODC具有较低的时间复杂度和存储开销,能够满足在线实时处理的要求。实测数据分析和模拟实验分析的结果均证实了ODC具有很好的检测和分类性能。     

Network anomaly detection and classification via opportunistic sampling

Abstract In this article the emphasis is placed on the evaluation of the impact of intelligent flow sampling techniques on the detection and classification of network anomalies. Based on the observation that for specific-purpose applications such as anomaly detection a large fraction of information is contained in a small fraction of flows, we demonstrate that by using sampling techniques that opportunistically and preferentially sample traffic data, we achieve ?magnification? of the appearance of anomalies within the sampled data set and therefore improve their detection. Therefore, the inherently ?lossy? sampling process is transformed to an advantageous feature in the anomaly detection case, allowing the revealing of anomalies that would be otherwise untraceable, and thus becoming the vehicle for efficient anomaly detection and classification. The evaluation of the impact of intelligent sampling techniques on the anomaly detection process is based on the application of an entropy-based anomaly detection method on a packet trace with data that has been collected from a real operational university campus network.     

A novel ensembled technique for anomaly detection

Anomaly detection is a technique that works to detect those instances of data that do not comply with the data model. In this paper the problem of anomaly detection in networked traffic data is considered, and a novel ensembled technique for anomaly detection is proposed. The proposed technique uses a combination of fuzzy K‐means clustering algorithm, extended Kalman filter, and support vector machines to detect the anomalies. In the proposed technique, fuzzy membership functions are used instead of crisp clusters to compute the best set of features by fuzzy k‐means algorithm. These features are then optimized with a nonlinear Bayesian approach known as extended Kalman filter. The resultant optimized set of features is then provided as an input to the support vector machine classifier that detects the network anomalies. The proposed technique is validated by using 2 benchmark datasets, ie, DARPA 1998 and KDD CUP 1999. Experimental results indicate that the proposed technique performs quite well as compared to its traditional counterparts in accuracy, detection rate, false positive rate, and F‐score.     

基于Filter-ary-Sketch数据结构的骨干网异常检测研究

针对骨干网上异常检测的特殊要求,提出了一种基于Filter-ary-Sketch数据结构的异常检测方法。该方法通过Filter-ary-Sketch实时记录网络流量信息,然后每隔一定周期进行基于多维熵值的异常检测。如果出现异常则根据Filter-ary-Sketch记录的流量信息进行异常点定位,最后利用Bloom Filter中记录的源IP信息进行恶意流量阻断。该方法能够检测多种类型的网络攻击,且能有效地进行恶意流量阻断。利用实际骨干网流量数据,分别从效率和精度2个方法进行对比实验,取得了较好的效果。     

基于深度学习的网络异常检测和智能流量预测方法

边缘计算场景下,边缘设备时刻产生海量蜂窝流量数据,在异常检测任务中针对直接对原始数据检测异常存在的计算冗余问题,提出基于特征降维的蜂窝流量数据异常检测方法.该方法在全局范围内利用LSTM自编码器提取流量数据特征和标识异常网格,然后在存在可疑异常的网格使用K?means聚类进行局部异常确认,结果表明可以更好地检测出不同活...     

基于精细流量的网络异常行为检测研究

随着Internet的快速发展和网络应用范围的不断扩大,网络日益遭受到了黑客更多的恶意攻击,计算机网络的安全问题已成为一个国际化的问题。面对诸多的挑战与威胁,入侵的检测与防范技术必然成为当前安全审计中的核心技术之一。文章首先介绍了异常检测的发展概况和相关技术,对常用的检测算法进行了分析和评价,为基于网络精细协议流量分析的网络异常实时检测方法的研究提供理论基础。