一种面向流量异常检测的概率流抽样方法

针对基于概率抽样的网络流量异常检测数据集构造过程中无法同时兼顾大、小流抽样需求及未区分flash crowd与流量攻击等问题,该文提出一种面向流量异常检测的概率流抽样方法。在对数据流按目的、源IP地址进行分类的基础上,将每类数据流抽样率定义为其目的、源IP地址抽样率的最大值,并在抽样过程中对数据流抽样数目向上取整,保证每类数据流至少被抽样一次,使抽样得到的数据集可有效反映原始流量在大、小流和源、目的IP地址方面的分布性。采用源IP地址熵刻画异常流源IP地址分散度,并基于源IP地址熵阈值设计攻击流抽样算法,降低由flash crowd引起的非攻击异常流抽样概率。仿真结果表明,该方法能同时满足大、小流抽样需求,具有较强的异常流抽样能力,可抽样到所有与异常流相关的可疑源、目的IP地址,并能在抽样过程中过滤非攻击异常流。     

DDoS Attack Detection and Wavelets

This paper presents a systematic method for DDoS attack detection. DDoS attack can be considered a system anomaly or misuse from which abnormal behavior is imposed on network traffic. Attack detection can be performed via abnormal behavior identification. Network traffic characterization with behavior modeling could be a good indication of attack detection. Aggregated traffic has been found to be strong bursty across a wide range of time scales. Wavelet analysis is able to capture complex temporal correlation across multiple time scales with very low computational complexity. We utilize energy distribution based on wavelet analysis to detect DDoS attack traffic. Energy distribution over time will have limited variation if the traffic keeps its behavior over time (i.e. attack-free situation) while an introduction of attack traffic in the network will elicit significant energy distribution deviation in a short time period. Our experimental results with typical Internet traffic trace show that energy distribution variance markedly changes, causing a spike when traffic behaviors are affected by DDoS attack. In contrast, normal traffic exhibits a remarkably stationary energy distribution. In addition, this spike in energy distribution variance can be captured in the early stages of an attack, far ahead of congestion build-up, making it an effective detection of the attack.     

基于深度特征学习的网络流量异常检测方法

针对网络流量异常检测过程中提取的流量特征准确性低、鲁棒性差导致流量攻击检测率低、误报率高等问题,该文结合堆叠降噪自编码器(SDA)和softmax,提出一种基于深度特征学习的网络流量异常检测方法。首先基于粒子群优化算法设计SDA结构两阶段寻优算法:根据流量检测准确率依次对隐藏层层数及每层节点数进行寻优,确定搜索空间中的最优SDA结构,从而提高SDA提取特征的准确性。然后采用小批量梯度下降算法对优化的SDA进行训练,通过最小化含噪数据重构向量与原始输入向量间的差异,提取具有较强鲁棒性的流量特征。最后基于提取的流量特征对softmax进行训练构建异常检测分类器,从而实现对流量攻击的高性能检测。实验结果表明:该文所提方法可根据实验数据及其分类任务动态调整SDA结构,提取的流量特征具有更高的准确性和鲁棒性,流量攻击检测率高、误报率低。     

Accurate anomaly detection through parallelism

In this article we discuss the design and implementation of a real-time parallel anomaly detection system. The key idea is to use multiple existing anomaly detection algorithms in parallel on thousands of network traffic subclasses, which not only enables us to detect hidden anomalies but also to increase the accuracy of the system. The main challenge then is the management and aggregation of the vast amount of data generated. We propose a novel aggregation process that uses the internal continuous anomaly metrics used by the algorithms to output a single system-wide anomaly metric. The evaluation on real-world attack traces shows a lower false positive rate and false negative rate than any individual anomaly detection algorithm.     

基于端口互动模式的入侵检测模型

针对链路层的海量高速数据流、信息易被伪装、较小异常流量占比等特征,提出了一种基于端口互动模式量化模型的入侵检测模型。为提高入侵检测模型的精度和效率,提出了一种从初始流量中获取流量特征的新方法,并重点探讨如何以流量到达时间分布作为一维特征。使用相空间重构、可视化等方法证明了模型的有效性,并进一步根据长会话和短会话各自的特征设计了基于卷积层和长短时记忆层的改进神经网络,用以挖掘正常和异常流量端口互动模式之间的差异。在此基础上,设计了多模型评分机制下的改进入侵检测算法,对模型空间内的会话进行流量分类。所提出的量化模型和改进算法在提高计算效率的同时,能够有效避免伪装身份信息的情况,降低神经网络训练成本,提升小样本异常检测精度。     

RMPCM：一种基于健壮多元概率校准模型的全网络异常检测方法

提出了一种基于健壮多元概率校准模型的异常检测方法。该方法使用基于多元t分布的隐变量概率模型建立流量矩阵的常态模型,通过比较样本与常态模型之间的马氏距离进行流量异常检测。理论分析和实验表明该方法的健壮性较好,应用场景宽泛,既可以处理完整数据也可以处理数据缺失的情况,对干扰抵抗力较强,并且对模型参数的敏感性较低,性能稳定。     

一种SDN中基于熵值计算的异常流量检测方法

摘要：软件定义网络（software defined networking,SDN）是一种新型网络创新架构,其分离了控制平面与转发平面,使得网络管理更为灵活。借助SDN控制与转发分离的思想,在SDN基础上引入一个集中式安全中心,在数据平面设备上采集数据,用于对网络流量进行分析,通过熵值计算和分类算法判断异常流量行为。对于检测到的网络异常情况,安全中心通过与SDN控制器的接口通告SDN控制器上的安全处理模块,进行流表策略的下发,进而缓解网络异常行为。通过本系统可以在不影响SDN控制器性能的情况下,快速检测网络中的异常行为,并通过SDN下发流表策略对恶意攻击用户进行限制,同时对SDN控制器进行保护。     

一种基于流量聚类性和切分性的改进异常检测模型

针对链路层异常检测中，由固定反馈时间点而导致的计算量积压以及大量无意义的采样流量数据等现象，提出了一种基于流量特征值的改进异常检测模型，重点探讨如何通过反馈计算机制实现周期内计算任务的合理优化和缩减采样数据。一方面，在对流持续时间的聚类性进行了深入分析并给出其可能聚类的最优簇基础上，将统一的反馈时间分散到各个聚类时间点；另一方面，基于流时序的可切分性对流量数据进行周期划分，并设计拟合函数对周期内流量特征进行量化表达。在此基础上，设计了改进反馈机制和异常检测算法流程。仿真实验表明，所提出的模型和算法不仅通过优化反馈计算时间提高了检测精度，而且通过降低采样数据冗余提高了检测效率。     

基于神经网络的异常检测

本文针对流量异常,提出了一种使用神经网络的检测方法。在仔细分析网络流量异常的基础上,提取流量特征数据,经预处理后供优化的BP神经网络分析,可准确检测出流量异常。测试结果表明,该模型对流量异常的检测有较高的准确性。     

基于PGM-NMF的网络流量异常检测研究

通过对网络流量数据进行采样,小波空间变化过滤噪声,构建了基于信息熵的网络流量矩阵,使用PGM-NMF算法对网络流量矩阵进行分解,构建的基于非负子空间方法的残余矩阵,应用Q 图实现网络流量的异常检测。理论分析及实验结果表明,与PCA方法相比,PGM-NMF算法在网络流量的异常检测中具有较好检测性能。     

基于RCSW的数据流速度异常检测算法研究

 目前许多应用领域产生数据流的流速不断地震荡,使得面向数据流的挖掘变得困难.系统采用RCSW来完成数据流抽取,提出了实时度T、关键时点集、数据流处理率的概念,并进一步提出了数据流速度异常检测算法.系统监控、预测数据流速,当数据流速异常减速或增速时,系统智能调节环形缓冲区和数据流处理率来应对异常,为解决数据流处理能力与流速、流量与有限空间之间的矛盾提供解决方案.实验表明数据流速度异常检测算法能够保证数据流的挖掘持续正常实施,最大程度的满足系统的实时性要求.     

Space-Code Bloom Filter for Efficient Per-Flow Traffic Measurement

Per-flow traffic measurement is critical for usage accounting, traffic engineering, and anomaly detection. Previous methodologies are either based on random sampling (e.g., Cisco's NetFlow), which is inaccurate, or only account for the "elephants." We introduce a novel technique for measuring per-flow traffic approximately, for all flows regardless of their sizes, at very high-speed (say, OC768). The core of this technique is a novel data structure called Space-Code Bloom Filter (SCBF). A SCBF is an approximate representation of a multiset; each element in this multiset is a traffic flow and its multiplicity is the number of packets in the flow. The multiplicity of an element in the multiset represented by SCBF can be estimated through either of two mechanisms-maximum-likelihood estimation or mean value estimation. Through parameter tuning, SCBF allows for graceful tradeoff between measurement accuracy and computational and storage complexity. SCBF also contributes to the foundation of data streaming by introducing a new paradigm called blind streaming. We evaluate the performance of SCBF through mathematical analysis and through experiments on packet traces gathered from a tier-1 ISP backbone. Our results demonstrate that SCBF achieves reasonable measurement accuracy with very low storage and computational complexity. We also demonstrate the application of SCBF in estimating the frequency of keywords at a search engine-demonstrating the applicability of SCBF to other problems that can be reduced to multiset membership queries     

Random-Forests-Based Network Intrusion Detection Systems

Prevention of security breaches completely using the existing security technologies is unrealistic. As a result, intrusion detection is an important component in network security. However, many current intrusion detection systems (IDSs) are rule-based systems, which have limitations to detect novel intrusions. Moreover, encoding rules is time-consuming and highly depends on the knowledge of known intrusions. Therefore, we propose new systematic frameworks that apply a data mining algorithm called random forests in misuse, anomaly, and hybrid-network-based IDSs. In misuse detection, patterns of intrusions are built automatically by the random forests algorithm over training data. After that, intrusions are detected by matching network activities against the patterns. In anomaly detection, novel intrusions are detected by the outlier detection mechanism of the random forests algorithm. After building the patterns of network services by the random forests algorithm, outliers related to the patterns are determined by the outlier detection algorithm. The hybrid detection system improves the detection performance by combining the advantages of the misuse and anomaly detection. We evaluate our approaches over the Knowledge Discovery and Data Mining 1999 (KDD’99) dataset. The experimental results demonstrate that the performance provided by the proposed misuse approach is better than the best KDD’99 result; compared to other reported unsupervised anomaly detection approaches, our anomaly detection approach achieves higher detection rate when the false positive rate is low; and the presented hybrid system can improve the overall performance of the aforementioned IDSs.      

Realizations of fast 2-D/3-D image filtering and enhancement

This paper proposes a novel algorithm for multidimensional image enhancement based on a fuzzy domain enhancement method, and an implementation of a recursive and separable low-pass filter. Considering a smoothed image as a fuzzy data set, each pixel in an image is processed independently, using fuzzy domain transformation and enhancement of both the dynamic range and the local gray level variations. The algorithm has the advantages of being fast and adaptive, so it can be used in real-time image processing applications and for multidimensional data with low computational cost. It also has the ability to reduce noise and unwanted background that may affect the visualization quality of two-dimensional (2-D)/three-dimensional (3-D) data. Examples for the applications of the algorithm are given for mammograms, ultrasound 3-D images, and photographic images.     

Intelligent detection method on network malicious traffic based on sample enhancement

To address the problem that the existing methods of network traffic anomaly detection not only need a large number of training sets,but also have poor generalization ability,an intelligent detection method on network malicious traffic based on sample enhancement was proposed.The key words were extracted from the training set and the sample of the training set was enhanced based on the strategy of key word avoidance,and the ability for the method to extract the text features from the training set was improved.The experimental results show that,the accuracy of network traffic anomaly detection model and cross dataset can be significantly improved by small training set.Compared with other methods,the proposed method can reduce the computational complexity and achieve better detection ability.     

An improvement of the state-of-the-art covariance-based methods for statistical anomaly detection algorithms

This paper presents a possible improvement to one of the main statistical anomaly detection algorithms for cyber security applications, i.e., the covariance-based method. This algorithm employs covariance matrices to build a norm profile of the normal network traffic and to detect anomalous activities in the data flow. In order to improve the detection capabilities of this algorithm, we propose a modified version of the statistical decision rule based on a generalized version of the Chebyshev inequality for random vectors. The performance of the proposed algorithm is evaluated and compared, in terms of ROC (receiver operating characteristic) curves with the ones of the state-of-the-art covariance-based algorithm.     

Monitoring the Application-Layer DDoS Attacks for Popular Websites

Distributed denial of service (DDoS) attack is a continuous critical threat to the Internet. Derived from the low layers, new application-layer-based DDoS attacks utilizing legitimate HTTP requests to overwhelm victim resources are more undetectable. The case may be more serious when such attacks mimic or occur during the flash crowd event of a popular Website. Focusing on the detection for such new DDoS attacks, a scheme based on document popularity is introduced. An Access Matrix is defined to capture the spatial-temporal patterns of a normal flash crowd. Principal component analysis and independent component analysis are applied to abstract the multidimensional Access Matrix. A novel anomaly detector based on hidden semi-Markov model is proposed to describe the dynamics of Access Matrix and to detect the attacks. The entropy of document popularity fitting to the model is used to detect the potential application-layer DDoS attacks. Numerical results based on real Web traffic data are presented to demonstrate the effectiveness of the proposed method.      

基于OpenFlow流表的云网络异常行为主动监测和处理

云数据中心的网络异常行为不仅对网络设备造成严重业务负荷,同时也显著影响云用户使用体验。云计算环境中的共享资源模式和云用户迥然不同的业务形态,使得网络分析和异常行为定位变得更加困难。本文针对云数据中心的网络异常行为进行特征提取和分析,并基于sdn云数据中心的网络架构和原理进行深度剖析,总结出基于openflow流表的网络异常行为判定方法。同时采用自动化运维手段,制定了一套网络异常行为自动化检测和封堵的智能系统,实现对网络异常行为的快速处理。     

基于秩2更新的多维数据流典型相关跟踪算法

 现存的多维数据流典型相关分析(Canonical Correlation Analysis,简称CCA)算法主要是基于近似技术的求解方法,本质上并不是持续更新的精确算法.为了能在时变的环境中持续、快速而精确地跟踪数据流之间的相关性,本文提出一种多维数据流典型相关跟踪算法TCCA.该算法基于秩2更新理论,通过并行方式持续更新样本协方差矩阵的特征子空间,进而实现多维数据流典型相关的快速跟踪.理论分析及仿真实验结果表明,TCCA具有较好的稳定性、较高的计算效率和精度,可以作为基本工具应用于数据流相关性检测、特征融合、数据降维等数据流挖掘领域.     

基于奇异值分解更新的多元在线异常检测方法

网络异常检测对于保证网络稳定高效运行极为重要。基于主成分分析的全网络异常检测算法虽然具有很好的检测性能,但无法满足在线检测的要求。为了解决此问题,该文引入流量矩阵模型,提出了一种基于奇异值分解更新的多元在线异常检测算法MOADA-SVDU,该算法以增量的方式构建正常子空间和异常子空间,并实现网络流量异常的在线检测。理论分析表明与主成分分析算法相比,该算法具有更低的存储和计算开销。因特网实测的流量矩阵数据集以及模拟试验数据分析表明,该算法不仅实现了网络异常的在线检测,而且取得了很好的检测性能。