基于子空间方法的大规模网络流量异常检测

本文采用子空间方法和PCA（主成分分析或Principal Components Analysis）对大规模网络流量异常检测进行研究,并以校园网为实验环境,应用子空间方法和PCA实现了网络流量异常检测。通过实验结果与小波分析结果的对比,证明了基于子空间方法的大规模网络流量异常检测是一种既简单又高效的方法。     

基于信息熵的大规模网络流量异常检测

提出了基于信息熵的大规模网络流量异常检测方法。该方法吸收了子空间方法的思想，并结合了K-means分类方法。以校园网为实验环境，应用基于信息熵的方法实现了网络流量异常检测的全过程。通过实验结果与应用标准子空间方法对测量数据分析结果的对比，证明了基于信息熵的大规模网络流量异常检测有着更高的检测精度。     

全网异常流量簇的检测与确定机制

在网络安全管理领域,自动确定异常流量簇可为ISP分析和定位全网流量异常提供有效手段.提出了一种基于过滤的网络流数据的全网异常流量簇检测及确定机制.给出了问题的形式化描述和定义;扩展和改进了基于多维树的大流量簇检测方法,提出了灵活的"检测阈值"及"分裂值"的计算方法以改善大流量簇的检测精度;通过剪枝算法缩减了树的规模,提高了查找大流量簇的效率;给出了基于大流量簇确定异常流量簇的方法.实验表明该方法是可行的,可应用于全网异常诊断.     

Statistical and signal‐based network traffic recognition for anomaly detection

In this paper, a framework for recognizing network traffic in order to detect anomalies is proposed. We propose to combine and correlate parameters from different layers in order to detect 0‐day attacks and reduce false positives. Moreover, we propose to combine statistical and signal‐based features. The major contribution of this paper is novel framework for network security based on the correlation approach as well as new signal‐based algorithm for intrusion detection on the basis of the Matching Pursuit (MP) algorithm. As to our best knowledge, we are the first to use MP for intrusion and anomaly detection in computer networks. In the presented experiments, we proved that our solution gives better results than intrusion detection based on discrete wavelet transform.     

Optimal volume anomaly detection and isolation in large-scale IP networks using coarse-grained measurements

Recent studies from major network technology vendors forecast the advent of the Exabyte era, a massive increase in network traffic driven by high-definition video and high-speed access technology penetration. One of the most formidable difficulties that this forthcoming scenario poses for the Internet is congestion problems due to traffic volume anomalies at the core network. In the light of this challenging near future, we develop in this work different network-wide anomaly detection and isolation algorithms to deal with volume anomalies in large-scale network traffic flows, using coarse-grained measurements as a practical constraint. These algorithms present well-established optimality properties in terms of false alarm and miss detection rate, or in terms of detection/isolation delay and false detection/isolation rate, a feature absent in previous works. This represents a paramount advantage with respect to current in-house methods, as it allows to generalize results independently of particular evaluations. The detection and isolation algorithms are based on a novel linear, parsimonious, and non-data-driven spatial model for a large-scale network traffic matrix. This model allows detecting and isolating anomalies in the Origin-Destination traffic flows from aggregated measurements, reducing the overhead and avoiding the challenges of direct flow measurement. Our proposals are analyzed and validated using real traffic and network topologies from three different large-scale IP backbone networks.     

基于多尺度主成分分析的全网络异常检测方法

网络异常检测对于保证网络的可靠运行具有重要意义,而现有的异常检测方法仅仅单独利用流量的时间相关性或空间相关性.针对这一不足,同时考虑流量矩阵的时空相关性,提出了一种基于MSPCA的全网络异常检测方法.该方法综合利用小波变换具有的多尺度建模能力和PCA具有的降维能力对正常流量进行建模,然后采用Shewart控制图和EWMA控制图分析残余流量.此外,还利用滑动窗口机制对MSPCA异常检测方法进行在线扩展,提出了一种在线的MSPCA异常检测方法.因特网实测数据分析和模拟实验分析表明:MSPCA算法的检测性能优于PCA算法和近期提出的KLE算法;在线MSPCA算法的检测性能非常接近MSPCA算法,且单步执行时间很短,完全满足实时检测的需要.     

基于iForest和LOF的流量异常检测

异常检测在现代大规模分布式系统的安全管理中起着重要作用,而网络流量异常检测则是组成异常检测系统的重要工具。网络流量异常检测的目的是找到和大多数流量数据不同的流量,并将这些离群点视为异常。由于现有的基于树分离的孤立森林（iForest）检测方法存在不能检测出局部异常的缺陷,为了克服这个缺陷,提出一种基于iForest和局部离群因子（LOF）近邻集成的无监督的流量异常检测方法。首先,改进原始的iForest与LOF算法,在提升检测精度的同时控制算法时间;然后分别使用两种改进算法进行检测,并将结果进行融合以得到最终的检测结果;最后在自制数据集上对所提方法进行有效性验证。实验结果表明,所提方法能够有效地隔离出异常,获得良好的流量异常检测效果。     

Enhancing redundant network traffic elimination

Protocol-independent redundant traffic elimination (RTE) is a method to detect and remove redundant chunks of data from network-layer packets by using caching at both ends of a network link or path. In this paper, we propose a set of techniques to improve the effectiveness of packet-level RTE. In particular, we consider two bypass techniques, with one based on packet size, and the other based on content type. The bypass techniques apply at the front-end of the RTE pipeline. Within the RTE pipeline, we propose chunk overlap and oversampling as techniques to improve redundancy detection, while obviating the need for chunk expansion at the network endpoints. Finally, we propose savings-based cache management at the back-end of the RTE pipeline, as an improvement over FIFO-based cache management. We evaluate our techniques on full-payload packet-level traces from university and enterprise environments. Our results show that the proposed techniques improve detected redundancy by up to 50% for university traffic, and up to 54% for enterprise Web server traffic.     

基于层叠模型的网络流量异常检测方法*

进行网络流量异常检测,需要对正常流量行为建立准确的模型,根据异常流量与正常模型间的偏离程度作出判断。针对现有网络流量模型中自相似模型与多分形模型无法全面刻画流量特征的不足,提出了一种基于流量层叠模型分析的异常检测算法,采用层叠模型对整个时间尺度上的流量特征进行更准确的描述,并运用小波变换对流量的层叠模型进行估计,分析异常流量对模型估计的影响,提出统计累计偏离量进行异常流量检测的方法。仿真结果表明,该方法能够有效检测出基于自相似Hurst系数方法不能检测的弱异常以及未明显影响Hurst系数变化的异常流。     

网络流量异常检测中分类器的提取与训练方法研究

随着网络安全领域研究的不断深入,研究者提出了各种类型的流量异常检测方法,基于分类的方法是其中很重要的一类.但是因为网络环境的多样性和动态变化性,在训练数据集上具有很高精度的检测系统实际部署时可能出现大量的误报.文中针对训练模型难于获取以及部署环境的动态变化性问题,对分类器的选择、使用和训练方法进行了研究.首先把网络流量数据投影到不同维度的Hash直方图上构建检测向量,在检测向量的基础上对比了各类分类器,选用能够处理高维数据、泛化能力强的SVDD进行异常检测;采用增减式在线训练算法对分类器进行不断训练,提高异常检测系统的精度并减少训练成本;最后采用多步关联检测算法优化检测精度,并在新增样本中剔除明显的异常样本,减少训练成本提高分类精度.通过大量的真实网络流量数据验证了上述方法具有较高的检准率和较低的误报率,并能够有效减少训练成本.     

Load characterization and anomaly detection for voice over IP traffic

We consider the problem of traffic anomaly detection in IP networks. Traffic anomalies typically arise when there is focused overload or when a network element fails and it is desired to infer these purely from the measured traffic. We derive new general formulae for the variance of the cumulative traffic over a fixed time interval and show how the derived analytical expression simplifies for the case of voice over IP traffic, the focus of this paper. To detect load anomalies, we show it is sufficient to consider cumulative traffic over relatively long intervals such as 5 min. We also propose simple anomaly detection tests including detection of over/underload. This approach substantially extends the current practice in IP network management where only the first-order statistics and fixed thresholds are used to identify abnormal behavior. We conclude with the application of the scheme to field data from an operational network.     

基于堆叠卷积注意力的网络流量异常检测模型

入侵检测系统（IDS）在发现网络异常和攻击方面发挥着重要作用,但传统IDS误报率较高,不能准确分析和识别异常流量。目前,深度学习技术被广泛应用于网络流量异常检测,但仅仅采用简单的深度神经网络（DNN）模型难以有效提取流量数据中的重要特征。针对上述问题,提出一种基于堆叠卷积注意力的DNN网络流量异常检测模型。通过堆叠多个以残差模块连接的注意力模块增加网络模型深度,同时在注意力模块中引入卷积神经网络、池化层、批归一化层和激活函数层,防止模型过拟合并提升模型性能,最后在DNN模型中得到输出向量。基于NSL-KDD数据集对模型性能进行评估,将数据集预处理生成二进制特征,采用多分类、二分类方式验证网络流量异常检测效果。实验结果表明,该模型性能优于KNN、SVM等机器学习模型和ANN、AlertNet等深度学习模型,其在多分类任务中识别准确率为0.807 6,较对比模型提高0.034 0~0.097 5,在二分类任务中准确率和F1分数为0.860 0和0.863 8,较对比模型提高0.013 0~0.098 8和0.030 6~0.112 8。     

网络流量异常检测及分析的研究

网络流量异常检测及分析是网络异常监视及响应应用的基础,是网络及安全管理领域的重要研究内容.本文探讨了网络流量数据类型、网络流量异常种类;从流量异常检测的范围、流量异常分析的深度、在线和离线异常检测方式等方面归纳了流量异常检测的研究内容;综述了已有的研究工作针对不同应用环境和研究内容所采用的不同的研究方法和技术手段,并分析了各种研究方法的特点、局限性和适用场合等;最后本文还对现有研究工作存在的问题及有待于进一步研究的课题进行了探讨.     

基于概要数据结构可溯源的异常检测方法

提出一种基于sketch概要数据结构的异常检测方法.该方法实时记录网络数据流信息到sketch数据结构,然后每隔一定周期进行异常检测.采用EWMA(exponentially weighted moving average)预测模型预测每一周期的预测值,计算观测值与预测值之间的差异sketch,然后基于差异sketch采用均值均方差模型建立网络流量变化参考.该方法能够检测DDoS、扫描等攻击行为,并能追溯异常的IP地址.通过模拟实验验证,该方法占用很少的计算和存储资源,能够检测骨干网络流量中的异常IP地址.     

基于流量统计指纹的恶意代码检测模型

采用加密和隧道技术的恶意代码难以检测。为此,提出基于流量统计指纹的恶意代码检测模型。提取恶意代码流量中的包层特征和流层特征,对高维流层特征采用主成分分析进行降维,利用两类特征的概率密度函数建立恶意代码流量统计指纹,使用该指纹检测网络中恶意代码通信流量。实验结果表明,该模型能有效检测采用加密和隧道技术的恶意代码。     

An anomaly-based detection in ubiquitous network using the equilibrium state of the catastrophe theory

It has been increasingly important for Pervasive and Ubiquitous Applications (PUA) of the network traffic, especially anomaly detection which plays a critical role in enforcing a high protection level of the network against threats. In this paper, we present a network traffic anomaly detection method based on the catastrophe theory. In order to characterize the normal behavior of the network, we construct a profile of the normal network traffic by using an equilibrium surface of the catastrophe theory. When anomalies occur, the state of the network traffic will deviate from the normal equilibrium surface. Then, taking the normal equilibrium surface as a reference, we monitor the ongoing network traffic and we use a new index called as catastrophe distance to quantify the deviation. According to the decision theory, network traffic anomalies can be identified by the catastrophe distance. We evaluate the performance of our approach using the DARPA intrusion detection data set. Experiment results show that our approach is significantly effective on the network traffic anomaly detection.     

Intrusion detection in computer networks by a modular ensemble of one-class classifiers

Since the early days of research on intrusion detection, anomaly-based approaches have been proposed to detect intrusion attempts. Attacks are detected as anomalies when compared to a model of normal (legitimate) events. Anomaly-based approaches typically produce a relatively large number of false alarms compared to signature-based IDS. However, anomaly-based IDS are able to detect never-before-seen attacks. As new types of attacks are generated at an increasing pace and the process of signature generation is slow, it turns out that signature-based IDS can be easily evaded by new attacks. The ability of anomaly-based IDS to detect attacks never observed in the wild has stirred up a renewed interest in anomaly detection. In particular, recent work focused on unsupervised or unlabeled anomaly detection, due to the fact that it is very hard and expensive to obtain a labeled dataset containing only pure normal events.The unlabeled approaches proposed so far for network IDS focused on modeling the normal network traffic considered as a whole. As network traffic related to different protocols or services exhibits different characteristics, this paper proposes an unlabeled Network Anomaly IDS based on a modular Multiple Classifier System (MCS). Each module is designed to model a particular group of similar protocols or network services. The use of a modular MCS allows the designer to choose a different model and decision threshold for different (groups of) network services. This also allows the designer to tune the false alarm rate and detection rate produced by each module to optimize the overall performance of the ensemble. Experimental results on the KDD-Cup 1999 dataset show that the proposed anomaly IDS achieves high attack detection rate and low false alarm rate at the same time.     

基于DAE和GRU组合的流量异常检测方法

流量异常检测能够有效识别网络流量数据中的攻击行为,是一种重要的网络安全防护手段。近年来,深度学习在流量异常检测领域得到了广泛应用,现有的深度学习模型进行流量异常检测存在两个问题:一是数据受噪声影响导致检测鲁棒性差、准确率低;二是数据特征维度高以及模型参数多导致训练和检测速度慢。为了在降低流量数据噪声影响的基础上提高检测速度和准确性,本文提出了一种基于去噪自编码器(Denoising Auto Encoder,DAE)和门控循环单元(Gated Recurrent Unit,GRU)组合的流量异常检测方法。首先设计了基于DAE的流量特征提取算法,采用小批量梯度下降算法对DAE进行训练,通过最小化含噪声数据的重构向量与原始输入向量间的差异,有效提取具有较强鲁棒性的流量特征,降低特征维度。然后设计了基于GRU的异常检测算法,利用提取的低维流量特征数据训练GRU,从而构建异常流量分类器,实现对攻击流量的准确检测。最后在NSL-KDD、UNSW-NB15、CICIDS2017数据集上的实验结果表明:与其他的机器学习、深度学习方法相比,本文所提方法的检测准确率最大提升了18.71%。同时,本文方法可以实现较高的精确率、召回率和检测效率,同时具有较低的误报率。在面对数据受到噪声破坏时,具有较强的检测鲁棒性。     

基于概要数据结构的全网络持续流检测方法

持续流是隐蔽的网络攻击过程中显现的一种重要特征，它不产生大量流量且在较长周期内有规律地发生，给传统的检测方法带来极大挑战。针对网络攻击的隐蔽性、单监测点的重负荷和信息有限的问题，提出全网络持续流检测方法。首先，设计一种概要数据结构，并将其部署在每个监测点；其次，当网络流到达监测点时，提取流的概要信息并更新概要数据结构的一位；然后，在测量周期结束时，主监测点将来自其他监测点的概要信息进行综合；最后，提出流持续性的近似估计，通过一些简单计算为每个流构建一个位向量，利用概率统计方法估计流持续性，使用修正后的持续性估计检测持续流。通过真实的网络流量进行实验，结果表明，与长持续时间流检测算法（TLF）相比，所提方法的准确性提高了50%，误报率和漏报率分别降低了22%和20%，说明全网络持续流检测方法能够有效监测高速网络流量。     

Detecting zero-day attacks using context-aware anomaly detection at the application-layer

Anomaly detection allows for the identification of unknown and novel attacks in network traffic. However, current approaches for anomaly detection of network packet payloads are limited to the analysis of plain byte sequences. Experiments have shown that application-layer attacks become difficult to detect in the presence of attack obfuscation using payload customization. The ability to incorporate syntactic context into anomaly detection provides valuable information and increases detection accuracy. In this contribution, we address the issue of incorporating protocol context into payload-based anomaly detection. We present a new data representation, called \({c}_n\)-grams, that allows to integrate syntactic and sequential features of payloads in an unified feature space and provides the basis for context-aware detection of network intrusions. We conduct experiments on both text-based and binary application-layer protocols which demonstrate superior accuracy on the detection of various types of attacks over regular anomaly detection methods. Furthermore, we show how \({c}_n\)-grams can be used to interpret detected anomalies and thus, provide explainable decisions in practice.