基于尖点突变模型的联动网络流量异常检测方法

针对现有方法没有考虑联动网络流量的非线性动力学特性,以及不能有效区分正常联动业务流量和异常攻击流量的问题,提出了一种基于尖点突变模型的联动流量异常检测方法。通过对联动网络流量非线性动力学特征参数的分析与提取,建立正常流量的尖点突变模型;利用模型的平衡曲面来描述网络流量系统的行为,构造正常网络流量行为的平衡曲面;并以网络流量行为相对于正常平衡曲面的偏离程度作为异常检测的依据。实验结果表明,所提方法具有较高的检测率和较低的误报率。     

Diagnosing Traffic Anomalies Using a Two-Phase Model

Network traffic anomalies are unusual changes in a network,so diagnosing anomalies is important for network management.Feature-based anomaly detection models (ab)normal network traffic behavior by analyzing packet header features.PCA-subspace method (Principal Component Analysis) has been verified as an efficient feature-based way in network-wide anomaly detection.Despite the powerful ability of PCA-subspace method for network-wide traffic detection,it cannot be effectively used for detection on a single link.In this paper,different from most works focusing on detection on flow-level traffic,based on observations of six traffic features for packet-level traffic,we propose a new approach B6SVM to detect anomalies for packet-level traffic on a single link.The basic idea of B6-SVM is to diagnose anomalies in a multi-dimensional view of traffic features using Support Vector Machine (SVM).Through two-phase classification,B6-SVM can detect anomalies with high detection rate and low false alarm rate.The test results demonstrate the effectiveness and potential of our technique in diagnosing anomalies.Further,compared to previous feature-based anomaly detection approaches,B6-SVM provides a framework to automatically identify possible anomalous types.The framework of B6-SVM is generic and therefore,we expect the derived insights will be helpful for similar future research efforts.     

Reservoir-based network traffic stream summarization for anomaly detection

Summarization is an important intermediate step for expediting knowledge discovery tasks such as anomaly detection. In the context of anomaly detection from data stream, the summary needs to represent both anomalous and normal data. But streaming data has distinct characteristics, such as one-pass constraint, for which conducting data mining operations are difficult. Existing stream summarization techniques are unable to create summary which represent both normal and anomalous instances. To address this problem, in this paper, a number of hybrid summarization techniques are designed and developed using the concept of reservoir for anomaly detection from network traffic. Experimental results on thirteen benchmark data streams show that the summaries produced from stream using pairwise distance (PSSR) and template matching (TMSSR) techniques can retain more anomalies than existing stream summarization techniques, and anomaly detection technique can identify the anomalies with high true positive and low false positive rate.     

网络流量异常检测及分析的研究

网络流量异常检测及分析是网络异常监视及响应应用的基础,是网络及安全管理领域的重要研究内容.本文探讨了网络流量数据类型、网络流量异常种类;从流量异常检测的范围、流量异常分析的深度、在线和离线异常检测方式等方面归纳了流量异常检测的研究内容;综述了已有的研究工作针对不同应用环境和研究内容所采用的不同的研究方法和技术手段,并分析了各种研究方法的特点、局限性和适用场合等;最后本文还对现有研究工作存在的问题及有待于进一步研究的课题进行了探讨.     

Detecting Network Attacks in the Internet via Statistical Network Traffic Normality Prediction

The information technology advances that provide new capabilities to the network users and providers, also provide powerful new tools for network intruders that intend to launch attacks on critical information resources. In this paper we present a novel network attack diagnostic methodology, based on the characterization of the dynamic statistical properties of normal network traffic. The ability to detect network anomalies and attacks as unacceptable when significant deviations from the expected behavior occurs. Specifically, to provide an accurate identification of the normal network traffic behavior, we first develop an anomaly-tolerant nonstationary traffic prediction technique that is capable of removing both single pulse and continuous anomalies. Furthermore, we introduce and design dynamic thresholds, where we define adaptive anomaly violation conditions as a combined function of both magnitude and duration of the traffic deviations. Finally numerical results are presented that demonstrate the operational effectiveness and efficiency of the proposed approach under the presence of different attacks, such as mail-bombing attacks and UDP flooding attacks.     

网络流量异常检测中分类器的提取与训练方法研究

随着网络安全领域研究的不断深入,研究者提出了各种类型的流量异常检测方法,基于分类的方法是其中很重要的一类.但是因为网络环境的多样性和动态变化性,在训练数据集上具有很高精度的检测系统实际部署时可能出现大量的误报.文中针对训练模型难于获取以及部署环境的动态变化性问题,对分类器的选择、使用和训练方法进行了研究.首先把网络流量数据投影到不同维度的Hash直方图上构建检测向量,在检测向量的基础上对比了各类分类器,选用能够处理高维数据、泛化能力强的SVDD进行异常检测;采用增减式在线训练算法对分类器进行不断训练,提高异常检测系统的精度并减少训练成本;最后采用多步关联检测算法优化检测精度,并在新增样本中剔除明显的异常样本,减少训练成本提高分类精度.通过大量的真实网络流量数据验证了上述方法具有较高的检准率和较低的误报率,并能够有效减少训练成本.     

Statistical and signal‐based network traffic recognition for anomaly detection

In this paper, a framework for recognizing network traffic in order to detect anomalies is proposed. We propose to combine and correlate parameters from different layers in order to detect 0‐day attacks and reduce false positives. Moreover, we propose to combine statistical and signal‐based features. The major contribution of this paper is novel framework for network security based on the correlation approach as well as new signal‐based algorithm for intrusion detection on the basis of the Matching Pursuit (MP) algorithm. As to our best knowledge, we are the first to use MP for intrusion and anomaly detection in computer networks. In the presented experiments, we proved that our solution gives better results than intrusion detection based on discrete wavelet transform.     

Wavelet against random forest for anomaly mitigation in software-defined networking

Security and availability of computer networks remain critical issues even with the constant evolution of communication technologies. In this core, traffic anomaly detection mechanisms need to be flexible to detect the growing spectrum of anomalies that may hinder proper network operation. In this paper, we argue that Software-defined Networking (SDN) provides a suitable environment for the design and implementation of more robust and comprehensive anomaly detection approaches. Aiming towards automated management to detect and prevent potential problems, we present an anomaly identification mechanism based on Discrete Wavelet Transform (DWT) and compare it with another detection model based on Random Forest. These methods generate a normal traffic profile, which is compared with actual real network traffic to recognize abnormal events. After a threat is detected, mitigation measures are activated so that the harmful effects of the malicious event are contained. We assess the effectiveness of the proposed anomaly detection methods and mitigation schemes using Distributed Denial of Service (DDoS) and port scan attacks. Our results confirm the effectiveness of both methods as well as the mitigation routines. In particular, the correspondence between the detection rates confirms that both methods enhance the detection of anomalous behavior by maintaining a satisfactory false-alarm rate.     

ClariSense+: An enhanced traffic anomaly explanation service using social network feeds

The explosive growth in social networks that publish real-time content begs the question of whether their feeds can complement traditional sensors to achieve augmented sensing capabilities. One such capability is to explain anomalous sensor readings. In our previous conference paper, we built an automated anomaly clarification service, called ClariSense, with the ability to explain sensor anomalies using social network feeds (from Twitter). In this extended work, we present an enhanced anomaly explanation system that augments our base algorithm by considering both (i) the credibility of social feeds and (ii) the spatial locality of detected anomalies. The work is geared specifically for describing small-footprint anomalies, such as vehicular traffic accidents. The original system used information gain to select more informative microblog items to explain physical sensor anomalies. In this paper, we show that significant improvements are achieved in our ability to explain small-footprint anomalies by accounting for information credibility and further discriminating among high-information-gain items according to the size of their spatial footprint. Hence, items that lack sufficient corroboration and items whose spatial footprint in the blogosphere is not specific to the approximate location of the physical anomaly receive less consideration. We briefly demonstrate the workings of such a system by considering a variety of real-world anomalous events, and comparing their causes, as identified by ClariSense+, to ground truth for validation. A more systematic evaluation of this work is done using vehicular traffic anomalies. Specifically, we consider real-time traffic flow feeds shared by the California traffic system. When flow anomalies are detected, our system automatically diagnoses their root cause by correlating the anomaly with feeds on Twitter. For evaluation purposes, the identified cause is then retroactively compared to official traffic and incident reports that we take as ground truth. Results show a great correspondence between our automatically selected explanations and ground-truth data.     

Load characterization and anomaly detection for voice over IP traffic

We consider the problem of traffic anomaly detection in IP networks. Traffic anomalies typically arise when there is focused overload or when a network element fails and it is desired to infer these purely from the measured traffic. We derive new general formulae for the variance of the cumulative traffic over a fixed time interval and show how the derived analytical expression simplifies for the case of voice over IP traffic, the focus of this paper. To detect load anomalies, we show it is sufficient to consider cumulative traffic over relatively long intervals such as 5 min. We also propose simple anomaly detection tests including detection of over/underload. This approach substantially extends the current practice in IP network management where only the first-order statistics and fixed thresholds are used to identify abnormal behavior. We conclude with the application of the scheme to field data from an operational network.     

基于统计的网络流量异常检测模型

提出了一个基于统计的流量异常检测模型。根据网络流量的可测度集,描绘了一个正常网络流量的基线。参照该正常流量基线,使用假设检验理论进行异常检测。采用一个基于滑动窗口的流量更新策略,使异常检测能够更加高效。论述了在高速网络情况下提高检测性能的方法。     

ClariSense+: An enhanced traffic anomaly explanation service using social network feeds

The explosive growth in social networks that publish real-time content begs the question of whether their feeds can complement traditional sensors to achieve augmented sensing capabilities. One such capability is to explain anomalous sensor readings. In our previous conference paper, we built an automated anomaly clarification service, called ClariSense, with the ability to explain sensor anomalies using social network feeds (from Twitter). In this extended work, we present an enhanced anomaly explanation system that augments our base algorithm by considering both (i) the credibility of social feeds and (ii) the spatial locality of detected anomalies. The work is geared specifically for describing small-footprint anomalies, such as vehicular traffic accidents. The original system used information gain to select more informative microblog items to explain physical sensor anomalies. In this paper, we show that significant improvements are achieved in our ability to explain small-footprint anomalies by accounting for information credibility and further discriminating among high-information-gain items according to the size of their spatial footprint. Hence, items that lack sufficient corroboration and items whose spatial footprint in the blogosphere is not specific to the approximate location of the physical anomaly receive less consideration. We briefly demonstrate the workings of such a system by considering a variety of real-world anomalous events, and comparing their causes, as identified by ClariSense+, to ground truth for validation. A more systematic evaluation of this work is done using vehicular traffic anomalies. Specifically, we consider real-time traffic flow feeds shared by the California traffic system. When flow anomalies are detected, our system automatically diagnoses their root cause by correlating the anomaly with feeds on Twitter. For evaluation purposes, the identified cause is then retroactively compared to official traffic and incident reports that we take as ground truth. Results show a great correspondence between our automatically selected explanations and ground-truth data.     

基于熵的网络异常流量检测研究综述

网络流量异常检测及分析作为一种重要的网络监管控制手段,是网络及安全管理领域的重要研究内容.本文探讨了网络异常流量的种类,简述了基于传统的异常检测方法在网络异常流量检测中的应用以及存在的问题.针对基于信息熵、相对熵、活跃熵等熵值理论在网络异常流量检测中的研究,阐述了基于熵值理论的异常检测在国内外的研究进展情况.总结了当前基于熵值理论的异常检测研究工作中存在的问题及改进方向.     

基于NetFlow时间序列的网络异常检测

网络流量在正常运行的情况下是具有一定的周期性、稳定性的,异常流量会打破这种规律使流量产生异常波动。提出了一种基于NetFlow时间序列滑动窗口检测网络异常的方法,利用时间序列异常发现算法发现网络流量的异常波动从而实现了实时高效的异常流量发现及预警。已经被检测到的网络异常会持续产生预警信息并影响后续的异常检测,为此还提出了两种平抑异常的方法。实验结果表明该方法能够有效地发现网络异常。     

Optimal volume anomaly detection and isolation in large-scale IP networks using coarse-grained measurements

Recent studies from major network technology vendors forecast the advent of the Exabyte era, a massive increase in network traffic driven by high-definition video and high-speed access technology penetration. One of the most formidable difficulties that this forthcoming scenario poses for the Internet is congestion problems due to traffic volume anomalies at the core network. In the light of this challenging near future, we develop in this work different network-wide anomaly detection and isolation algorithms to deal with volume anomalies in large-scale network traffic flows, using coarse-grained measurements as a practical constraint. These algorithms present well-established optimality properties in terms of false alarm and miss detection rate, or in terms of detection/isolation delay and false detection/isolation rate, a feature absent in previous works. This represents a paramount advantage with respect to current in-house methods, as it allows to generalize results independently of particular evaluations. The detection and isolation algorithms are based on a novel linear, parsimonious, and non-data-driven spatial model for a large-scale network traffic matrix. This model allows detecting and isolating anomalies in the Origin-Destination traffic flows from aggregated measurements, reducing the overhead and avoiding the challenges of direct flow measurement. Our proposals are analyzed and validated using real traffic and network topologies from three different large-scale IP backbone networks.     

基于iForest和LOF的流量异常检测

异常检测在现代大规模分布式系统的安全管理中起着重要作用,而网络流量异常检测则是组成异常检测系统的重要工具。网络流量异常检测的目的是找到和大多数流量数据不同的流量,并将这些离群点视为异常。由于现有的基于树分离的孤立森林(iForest)检测方法存在不能检测出局部异常的缺陷,为了克服这个缺陷,提出一种基于iForest和局部离群因子(LOF)近邻集成的无监督的流量异常检测方法。首先,改进原始的iForest与LOF算法,在提升检测精度的同时控制算法时间;然后分别使用两种改进算法进行检测,并将结果进行融合以得到最终的检测结果;最后在自制数据集上对所提方法进行有效性验证。实验结果表明,所提方法能够有效地隔离出异常,获得良好的流量异常检测效果。     

基于小波的网络流量异常协同相变检测

针对网络流量表现出的非线性和非平稳性等复杂的动力学特征,提出一种基于小波的网络流量异常协同相变检测方法。该方法从网络流量时间序列的离散小波域出发,利用序参量的非线性动力学方程描述网络流量系统的复杂行为,采用势函数来刻画网络流量系统的非平稳相变过程,进一步分析了网络流量状态与各种攻击模式之间的变化关系,并通过协同学模型对网络流量序参量进行演化,当相应序参量收敛时,即可检测到相应的攻击模式或是正常流量模式。最后,采用了DARPA 1999数据集进行了实验测试,网络流量异常的平均检测率达到了90.00%,而平均误检率只有15.03%。实验结果表明,基于小波的协同相变方法可以用于网络流量异常检测。     

基于IP监控和非高斯统计的网络异常流量检测

为保障网络和信息系统安全,需要对网络实施有效的监控,确保能及时检测出网络异常(蠕虫爆发、DDoS攻击等)等流量,进而为后续的动态量化风险评估、主动防御提供有力支持。为此,本文提出了一种基于IP监控和非高斯统计的网络异常流量检测方法(IPM-NGSD)。该方法包括两个关键部分:常用IP地址库FIPD和非高斯统计建模。前者,通过利用Bloomfilter技术和FIPD,将网络流量快速分流为常见和非常见IP网络流量:S0和S1;后者,在不同聚合层次上,提取S0和S1的非高斯边缘分布的轮廓值Porfile0和Porfile1,并通过计算Porfile0和Porfile1之间的统计距离,来检测是否存在异常。通过理论分析和两组统计实验验证了该方法的有效性:在缺少有关目标流量先验知识的前提下,该方法能快速、准确地发现短期突发攻击流量和长期低密度攻击流量。     

Reactive Robust Routing: Anomaly Localization and Routing Reconfiguration for Dynamic Networks

This paper presents a novel approach to deal with dynamic and highly uncertain traffic in dynamic network scenarios. The Reactive Robust Routing (RRR) approach is introduced, a combination of proactive and reactive techniques to improve network efficiency and robustness, simplifying network operation. RRR optimizes routing for normal-operation traffic, using a time-varying extension of the already established Robust Routing technique that outperforms the stable approach. To deal with anomalous and unexpected traffic variations, RRR uses a fast anomaly detection and localization algorithm that rapidly detects and localizes abrupt changes in traffic flows, permitting an accurate routing adaptation. This algorithm presents well-established optimality properties in terms of detection/localization rates and localization delay, which allows for generalization of results, independently of particular evaluations. The algorithm is based on a novel parsimonious model for traffic demands which allows for detection of anomalies using easily available aggregated-traffic measurements, reducing the overheads of data collection.     

基于迁移学习和D-S理论的网络异常检测

对于分布不同或分布相似的未知类型的网络攻击,目前的异常检测技术往往不能达到预期的效果。针对上述问题,研究了一种基于迁移技术和D-S证据理论的网络异常检测方法,首先用迁移学习方法对已知网络攻击进行建模,此模型在构建时考虑了不同分布的异常攻击间的差异,而后用其训练得到的分类器对未知的网络行为进行分析,结合D-S证据理论,可以检测出分布不一致的未知攻击类型。实验结果表明,所提方法泛化了传统的网络异常检测技术,对未知的网络异常有着较高的检测率。