一种面向流量异常检测的概率流抽样方法

针对基于概率抽样的网络流量异常检测数据集构造过程中无法同时兼顾大、小流抽样需求及未区分flash crowd与流量攻击等问题,该文提出一种面向流量异常检测的概率流抽样方法。在对数据流按目的、源IP地址进行分类的基础上,将每类数据流抽样率定义为其目的、源IP地址抽样率的最大值,并在抽样过程中对数据流抽样数目向上取整,保证每类数据流至少被抽样一次,使抽样得到的数据集可有效反映原始流量在大、小流和源、目的IP地址方面的分布性。采用源IP地址熵刻画异常流源IP地址分散度,并基于源IP地址熵阈值设计攻击流抽样算法,降低由flash crowd引起的非攻击异常流抽样概率。仿真结果表明,该方法能同时满足大、小流抽样需求,具有较强的异常流抽样能力,可抽样到所有与异常流相关的可疑源、目的IP地址,并能在抽样过程中过滤非攻击异常流。     

基于流量负载的改进型几何抽样算法

为了解决在流量高峰时期网络节点由于超负荷工作导致报文丢失的问题,在几何抽样的基础上,设计了一种能自动适应流量负载变化的抽样算法.该算法在流量高峰时会根据负载情况动态地计算最佳的抽样概率来匹配节点的处理能力从而降低节点丢失的报文数,在流量负载轻时会自动地提高抽样概率以充分利用节点的处理性能.通过真实网络流量数据的实验分析表明,改进后的几何抽样算法不仅能有效降低节点丢弃的报文数,同时还提高了网络测量的精度,证明了该改进算法对流量负载具有良好的适应性.     

一种基于大小流区分计数的公平抽样算法

针对一种草图指导公平抽样(SGS)算法对小流估计误差大的问题,该文提出一种基于大小流区分计数的包公平抽样算法(DCMFS),并给出哈希冲突对SGS算法估计误差影响的定量分析结果。DCMFS采用大小流区分计数器,对小流采用逐流精确计数,对大流采用哈希计数。理论分析及实际的数据仿真结果均表明,DCMFS算法对小流能够实现逐流精确统计,对大流的估计标准差接近公平抽样估计标准差理论值上限。算法采用不等长位宽计数器结构,保证其空间复杂度较SGS和自适应非线性抽样方法(ANLS)没有增加;引入计数器置换使得算法时间复杂度略有提高,但仍能满足10 Gbps线速处理要求。     

基于抽样和哈希技术的长流测量算法研究

在互联网中理解网络行为最高效的途径即是对网络数据流量进行检测与分析,它是对已有互联网的组建、规范化和改造的依据,同时也是对Internet进行检测的重要环节.为了解决网络中的资源和高速IP流量之间的冲突问题,需要对网络流进行多种方式的处理与算法研究.本文首先提出了改进的数据抽样技术并综合论述了现阶段基于抽样技术的数据测量算法的研究.同时通过对重要数据参数的重新设置和分析,并结合使用多种数据取样的方法,探讨改进的数据空间映射技术与现阶段的各种取样方式在测量网络长流算法中的综合应用.     

改进的长度自适应门限组装算法

在光突发交换中,针对自相似业务流,为了降低组装算法对分组阻塞率的影响,分析了Se-Yoon等人提出的长度自适应组装算法。根据自相似业务的突发特性,算法应及时更新算法参数,因此提出了一种改进算法,它更能降低业务流的自相似性,能更合理地根据到达的网络自相似业务流的特点自适应地调整长度门限值。仿真结果表明,改进后的算法分组阻塞率影响更小。     

基于分组标识的网络流量抽样测量模型

PSAMP建议流量抽样测量模型应该简单且能够满足各种测量应用要求,为此,文章提出基于报文标识的流量抽样测量模型.对CERNET主干网络流量IP报头各字段的进行随机性分析,结果表明标识字段16比特统计上满足抽样掩码匹配字段的随机性要求.并提出基于标识字段的多掩码抽样测量算法及其修正算法,实验验证其抽样样本既能满足流量统计行为研究,又能进行网络行为研究.     

基于LRU的大流检测算法

高速网络中,检测大流是进行准确流量测量的一种重要可扩展解决方案。该文提出了一种新的基于LRU大流检测算法。它通过引入小流早期丢弃和大流预保护机制以提高测量准确性。算法分析表明:新算法具有10Gbps线速处理能力。该文基于实际互联网数据进行了实验比较,结果显示:与已有算法相比,新算法具有更高的测量准确性和实用性。     

面向带宽碎片最小化和QoS保障的数据中心网络流量调度算法

随着数据中心网络流量的迅速增长,如何提高数据中心网络性能和服务质量成为了研究热点。然而现有的流量调度算法在网络负载加大时,一方面会导致网络带宽碎片化从而使得网络吞吐量降低,另一方面忽视了流量应用需求导致网络服务质量较差。为此,该文提出一种面向带宽碎片最小化和QoS保障的动态流量调度算法,算法综合考虑了带宽敏感的大流、时延与丢包敏感的小流的不同需求,首先根据待调度流的源地址和目的地址建立最短路径集,其次从中筛选出满足待调度流的带宽需求的所有路径,然后根据路径剩余带宽信息和小流应用需求情况为每条路径建立权重函数,最后根据权重函数值利用轮盘赌算法选择转发路径。实验仿真结果显示,与其它算法相比,所提算法降低了小流的丢包率和时延,同时在网络负载较大时提升了网络吞吐量。     

基于全局纹理和抽样推断的自适应阴影检测算法

为了提高不同光线环境下阴影检测的准确度和稳定性,提出了一种自适应的阴影检测算法。设计了一种阴影检测器,利用候选前景中像素YUV分量变化比率判别阴影像素,其检测阈值由阈值估计器得到。阈值估计器利用全局纹理和抽样推断的方法统计计算出当前光线环境下所需的阈值。整个阴影检测过程不需要人工干预,适应于各种复杂动态的场景。对代表不同光线条件的标准测试视频的检测实验表明,本文算法能够自适应地检测得到各目标阴影区域,具有较好的稳定性和实时性,综合检测指标达到94%以上。     

基于非抽样Contourlet变换的自适应 阈值图像增强算法

提出了一种基于非抽样Contourlet变换的自适应阈值图像增强算法,首先对图像进行非抽样Contourlet变换得到不同尺度不同方向上的变换系数,然后由变换系数自适应地确定阈值和调整增强函数,并对变换系数做增强处理,最后对增强处理后的变换系数进行反变换,实现图像增强.实验结果表明,与其他基于变换域的算法相比,该算法可以得到更好的增强效果.     

Fast, Memory Efficient Flow Rate Estimation Using Runs

Per-flow network traffic measurements are needed for effective network traffic management, network performance assessment, and detection of anomalous network events such as incipient denial-of-service (DoS) attacks. Explicit measurement of per-flow traffic statistics is difficult in backbone networks because tracking the possibly hundreds of thousands of flows needs correspondingly large high-speed memories. To reduce the measurement overhead, many previous papers have proposed the use of random sampling and this is also used in commercial routers (Cisco's NetFlow). Our goal is to develop a new scheme that has very low memory requirements and has quick convergence to within a pre-specified accuracy. We achieve this by use of a novel approach based on sampling two-runs to estimate per-flow traffic. (A flow has a two-run when two consecutive samples belong to the same flow). Sampling two-runs automatically biases the samples towards the larger flows thereby making the estimation of these sources more accurate. This biased sampling leads to significantly smaller memory requirement compared to random sampling schemes. The scheme is very simple to implement and performs extremely well.     

基于PGM-NMF的网络流量异常检测研究

通过对网络流量数据进行采样,小波空间变化过滤噪声,构建了基于信息熵的网络流量矩阵,使用PGM-NMF算法对网络流量矩阵进行分解,构建的基于非负子空间方法的残余矩阵,应用Q 图实现网络流量的异常检测。理论分析及实验结果表明,与PCA方法相比,PGM-NMF算法在网络流量的异常检测中具有较好检测性能。     

Impact of Packet Sampling on Portscan Detection

Packet sampling is commonly deployed in high-speed backbone routers to minimize resources used for network monitoring. It is known that packet sampling distorts traffic statistics and its impact has been extensively studied for traffic engineering metrics such as flow size and mean rate. However, it is unclear how packet sampling impacts anomaly detection, which has become increasingly critical to network providers. This paper is the first attempt to address this question by focusing on one common class of nonvolume-based anomalies, portscans, which are associated with worm/virus propagation. Existing portscan detection algorithms fall into two general approaches: target-specific and traffic profiling. We evaluated representative algorithms for each class, namely: 1) TRWSYN that performs stateful traffic analysis; 2) TAPS that tracks connection pattern of scanners; and 3) entropy-based traffic profiling. We applied these algorithms to detect portscans in both the original and sampled packet traces from a Tier-1 provider's backbone network. Our results demonstrate that sampling introduces fundamental bias that degrades the effectiveness of these detection algorithms and dramatically increases false positives. Through both experiments and analysis, we identify the traffic features critical for anomaly detection that are affected by sampling. Finally, using insight gained from this study, we show how portscan algorithms can be enhanced to be more robust to sampling     

一种新的基于事件触发的网络流量采样测量方法及其应用研究

该文主要对采样测量方法进行了研究,提出了基于事件触发的分层随机双采样方法(ESRDS)。通过与传统采样方法的仿真实验比较,证明该方法在分析包长、包到达时间间隔时性能都比较高,尤其该方法采样结果在估计包到达时间间隔时方面和实际比较接近。同时作者还对采样测量方法在计算Hurst参数方面进行了研究,应用ESRDS对网络流量的Hurst参数进行了估计,估计值和实际值之间的误差非常小。     

Network anomaly detection and classification via opportunistic sampling

Abstract In this article the emphasis is placed on the evaluation of the impact of intelligent flow sampling techniques on the detection and classification of network anomalies. Based on the observation that for specific-purpose applications such as anomaly detection a large fraction of information is contained in a small fraction of flows, we demonstrate that by using sampling techniques that opportunistically and preferentially sample traffic data, we achieve ?magnification? of the appearance of anomalies within the sampled data set and therefore improve their detection. Therefore, the inherently ?lossy? sampling process is transformed to an advantageous feature in the anomaly detection case, allowing the revealing of anomalies that would be otherwise untraceable, and thus becoming the vehicle for efficient anomaly detection and classification. The evaluation of the impact of intelligent sampling techniques on the anomaly detection process is based on the application of an entropy-based anomaly detection method on a packet trace with data that has been collected from a real operational university campus network.     

基于奇异值分解更新的多元在线异常检测方法

网络异常检测对于保证网络稳定高效运行极为重要。基于主成分分析的全网络异常检测算法虽然具有很好的检测性能,但无法满足在线检测的要求。为了解决此问题,该文引入流量矩阵模型,提出了一种基于奇异值分解更新的多元在线异常检测算法MOADA-SVDU,该算法以增量的方式构建正常子空间和异常子空间,并实现网络流量异常的在线检测。理论分析表明与主成分分析算法相比,该算法具有更低的存储和计算开销。因特网实测的流量矩阵数据集以及模拟试验数据分析表明,该算法不仅实现了网络异常的在线检测,而且取得了很好的检测性能。     

Trajectory Sampling With Unreliable Reporting

We define and evaluate methods to perform robust network monitoring using trajectory sampling in the presence of report loss. The first challenge is to reconstruct an unambiguous set of packet trajectories from the reports on sampled packets received at a collector. In this paper we extend the reporting paradigm of trajectory sampling to enable the elimination of ambiguous groups of reports, but without introducing bias into any characterization of traffic based on the surviving reports. Even after the elimination, a proportion of trajectories are incomplete due to report loss. A second challenge is to adapt measurement based applications (including network engineering, path tracing, and passive performance measurement) to incomplete trajectories. To achieve this, we propose a method to join multiple incomplete trajectories for inference, and analyze its performance. We also show how applications can distinguish between packet and report loss at the statistical level.     

一种基于流量聚类性和切分性的改进异常检测模型

针对链路层异常检测中，由固定反馈时间点而导致的计算量积压以及大量无意义的采样流量数据等现象，提出了一种基于流量特征值的改进异常检测模型，重点探讨如何通过反馈计算机制实现周期内计算任务的合理优化和缩减采样数据。一方面，在对流持续时间的聚类性进行了深入分析并给出其可能聚类的最优簇基础上，将统一的反馈时间分散到各个聚类时间点；另一方面，基于流时序的可切分性对流量数据进行周期划分，并设计拟合函数对周期内流量特征进行量化表达。在此基础上，设计了改进反馈机制和异常检测算法流程。仿真实验表明，所提出的模型和算法不仅通过优化反馈计算时间提高了检测精度，而且通过降低采样数据冗余提高了检测效率。