Impact of Packet Sampling on Portscan Detection

Packet sampling is commonly deployed in high-speed backbone routers to minimize resources used for network monitoring. It is known that packet sampling distorts traffic statistics and its impact has been extensively studied for traffic engineering metrics such as flow size and mean rate. However, it is unclear how packet sampling impacts anomaly detection, which has become increasingly critical to network providers. This paper is the first attempt to address this question by focusing on one common class of nonvolume-based anomalies, portscans, which are associated with worm/virus propagation. Existing portscan detection algorithms fall into two general approaches: target-specific and traffic profiling. We evaluated representative algorithms for each class, namely: 1) TRWSYN that performs stateful traffic analysis; 2) TAPS that tracks connection pattern of scanners; and 3) entropy-based traffic profiling. We applied these algorithms to detect portscans in both the original and sampled packet traces from a Tier-1 provider's backbone network. Our results demonstrate that sampling introduces fundamental bias that degrades the effectiveness of these detection algorithms and dramatically increases false positives. Through both experiments and analysis, we identify the traffic features critical for anomaly detection that are affected by sampling. Finally, using insight gained from this study, we show how portscan algorithms can be enhanced to be more robust to sampling     

Steganalysis of Network Packet Length Based Data Hiding

Recently, data hiding by modifying network parameters like packet header, payload, and packet length has become popular among researchers. Different algorithms have been proposed during the last few years which have altered the network packets in different ways to embed the data bits. Some of these algorithms modify the network packet length for embedding. Although most of the packet length based embedding schemes try to imitate the normal network traffic distribution, they have altered the statistical distribution of network packet lengths during embedding. These statistical anomalies can be exploited to detect such schemes. In this paper, a second order detection scheme for packet length based steganography has been proposed. A comprehensive set of experiments have been carried out to show that the proposed detection scheme can detect network packet length based steganography with a considerably high accuracy.     

Image-Based Anomaly Detection Technique: Algorithm, Implementation and Effectiveness

The frequent and large-scale network attacks have led to an increased need for developing techniques for analyzing network traffic. This paper presents NetViewer, a network measurement approach that can simultaneously detect, identify, and visualize attacks and anomalous traffic in real-time by passively monitoring packet headers. We propose to represent samples of network packet header data as frames or images. With such a formulation, a series of samples can be seen as a sequence of frames or video, revealing certain kinds of attacks to the human eye. This enables techniques from image processing and video compression to be applied to the packet header data to reveal interesting properties of traffic. We show that “scene change analysis” can reveal sudden changes in traffic behavior or anomalies. We also show that “motion prediction” techniques can be employed to understand the patterns of some of the attacks. We show that it may be feasible to represent multiple pieces of data as different colors of an image enabling a uniform treatment of multidimensional packet header data. We compare the effectiveness of NetViewer with classical detection theory-based Neyman–Pearson test.     

Statistical Techniques for Detecting Traffic Anomalies Through Packet Header Data

This paper proposes a traffic anomaly detector, operated in postmortem and in real-time, by passively monitoring packet headers of traffic. The frequent attacks on network infrastructure, using various forms of denial of service attacks, have led to an increased need for developing techniques for analyzing network traffic. If efficient analysis tools were available, it could become possible to detect the attacks, anomalies and to take action to contain the attacks appropriately before they have had time to propagate across the network. In this paper, we suggest a technique for traffic anomaly detection based on analyzing correlation of destination IP addresses in outgoing traffic at an egress router. This address correlation data are transformed using discrete wavelet transform for effective detection of anomalies through statistical analysis. Results from trace-driven evaluation suggest that proposed approach could provide an effective means of detecting anomalies close to the source. We also present a multidimensional indicator using the correlation of port numbers and the number of flows as a means of detecting anomalies.     

Inverting sampled traffic

Routers have the ability to output statistics about packets and flows of packets that traverse them. Since, however, the generation of detailed traffic statistics does not scale well with link speed, increasingly routers and measurement boxes implement sampling strategies at the packet level. In this paper, we study both theoretically and practically what information about the original traffic can be inferred when sampling, or "thinning", is performed at the packet level. While basic packet level characteristics such as first order statistics can be fairly directly recovered, other aspects require more attention. We focus mainly on the spectral density, a second-order statistic, and the distribution of the number of packets per flow, showing how both can be exactly recovered, in theory. We then show in detail why in practice this cannot be done using the traditional packet based sampling, even for high sampling rate. We introduce an alternative flow-based thinning, where practical inversion is possible even at arbitrarily low sampling rate. We also investigate the theory and practice of fitting the parameters of a Poisson cluster process, modeling the full packet traffic, from sampled data.     

Network anomaly detection and classification via opportunistic sampling

Abstract In this article the emphasis is placed on the evaluation of the impact of intelligent flow sampling techniques on the detection and classification of network anomalies. Based on the observation that for specific-purpose applications such as anomaly detection a large fraction of information is contained in a small fraction of flows, we demonstrate that by using sampling techniques that opportunistically and preferentially sample traffic data, we achieve ?magnification? of the appearance of anomalies within the sampled data set and therefore improve their detection. Therefore, the inherently ?lossy? sampling process is transformed to an advantageous feature in the anomaly detection case, allowing the revealing of anomalies that would be otherwise untraceable, and thus becoming the vehicle for efficient anomaly detection and classification. The evaluation of the impact of intelligent sampling techniques on the anomaly detection process is based on the application of an entropy-based anomaly detection method on a packet trace with data that has been collected from a real operational university campus network.     

Exploiting packet‐sampling measurements for traffic characterization and classification

The use of packet sampling for traffic measurement has become mandatory for network operators to cope with the huge amount of data transmitted in today's networks, powered by increasingly faster transmission technologies. Therefore, many networking tasks must already deal with such reduced data, more available but less rich in information. In this work we assess the impact of packet sampling on various network monitoring‐activities, with a particular focus on traffic characterization and classification. We process an extremely heterogeneous dataset composed of four packet‐level traces (representative of different access technologies and operational environments) with a traffic monitor able to apply different sampling policies and rates to the traffic and extract several features both in aggregated and per‐flow fashion, providing empirical evidences of the impact of packet sampling on both traffic measurement and traffic classification. First, we analyze feature distortion, quantified by means of two statistical metrics: most features appear already deteriorated under low sampling step, no matter the sampling policy, while only a few remain consistent under harsh sampling conditions, which may even cause some artifacts, undermining the correctness of measurements. Second, we evaluate the performance of traffic classification under sampling. The information content of features, even though deteriorated, still allows a good classification accuracy, provided that the classifier is trained with data obtained at the same sampling rate of the target data. The accuracy is also due to a thoughtful choice of a smart sampling policy which biases the sampling towards packets carrying the most useful information. Copyright © 2012 John Wiley & Sons, Ltd.     

Evaluation and characterization of available bandwidth probing techniques

The packet pair mechanism has been shown to be a reliable method to measure the bottleneck link capacity on a network path, but its use for measuring available bandwidth is more challenging. In this paper, we use modeling, measurements, and simulations to better characterize the interaction between probing packets and the competing network traffic. We first construct a simple model to understand how competing traffic changes the probing packet gap for a single-hop network. The gap model shows that the initial probing gap is a critical parameter when using packet pairs to estimate available bandwidth. Based on this insight, we present two available bandwidth measurement techniques, the initial gap increasing (IGI) method and the packet transmission rate (PTR) method. We use extensive Internet measurements to show that these techniques estimate available bandwidth faster than existing techniques such as Pathload, with comparable accuracy. Finally, using both Internet measurements and ns simulations, we explore how the measurement accuracy of active probing is affected by factors such as the probing packet size, the length of probing packet train, and the competing traffic on links other than the tight link.     

Adaptive traffic sampling for P2P botnet detection

Peer‐to‐peer (P2P) botnets have become one of the major threats to network security. Most existing botnet detection systems detect bots by examining network traffic. Unfortunately, the traffic volumes typical of current high‐speed Internet Service Provider and enterprise networks are challenging for these network‐based systems, which perform computationally complex analyses. In this paper, we propose an adaptive traffic sampling system that aims to effectively reduce the volume of traffic that P2P botnet detectors need to process while not degrading their detection accuracy. Our system first identifies a small number of potential P2P bots in high‐speed networks as soon as possible, and then samples as many botnet‐related packets as possible with a predefined target sampling rate. The sampled traffic then can be delivered to fine‐grained detectors for further in‐depth analysis. We evaluate our system using traffic datasets of real‐world and popular P2P botnets. The experiments demonstrate that our system can identify potential P2P bots quickly and accurately with few false positives and greatly increase the proportion of botnet‐related packets in the sampled packets while maintain the high detection accuracy of the fine‐grained detectors.     

Biologically inspired artificial intrusion detection system for detecting wormhole attack in MANET

A mobile ad hoc network (MANET) does not have traffic concentration points such as gateway or access points which perform behaviour monitoring of individual nodes. Therefore, maintaining the network function for the normal nodes when other nodes do not forward and route properly is a big challenge. One of the significant attacks in ad hoc network is wormhole attack. In this wormhole attack, the adversary disrupts ad hoc routing protocols using higher bandwidth and lower-latency links. Wormhole attack is more hidden in character and tougher to detect. So, it is necessary to use mechanisms to avoid attacking nodes which can disclose communication among unauthorized nodes in ad hoc networks. Mechanisms to detect and punish such attacking nodes are the only solution to solve this problem. Those mechanisms are known as intrusion detection systems (IDS). In this paper, the suggested biological based artificial intrusion detection system (BAIDS) include hybrid negative selection algorithm (HNSA) detectors in the local and broad detection subsection to detect anomalies in ad hoc network. In addition to that, response will be issued to take action over the misbehaving nodes. These detectors employed in BAIDS are capable of discriminating well behaving nodes from attacking nodes with a good level of accuracy in a MANET environment. The performance of BAIDS in detecting wormhole attacks in the background of DSR, AODV and DSDV routing protocols is also evaluated using Qualnet v 5.2 network simulator. Detection rate, false alarm rate, packet delivery ratio, routing overhead are used as metrics to compare the performance of HNSA and the BAIDS technique.     

Accurate anomaly detection through parallelism

In this article we discuss the design and implementation of a real-time parallel anomaly detection system. The key idea is to use multiple existing anomaly detection algorithms in parallel on thousands of network traffic subclasses, which not only enables us to detect hidden anomalies but also to increase the accuracy of the system. The main challenge then is the management and aggregation of the vast amount of data generated. We propose a novel aggregation process that uses the internal continuous anomaly metrics used by the algorithms to output a single system-wide anomaly metric. The evaluation on real-world attack traces shows a lower false positive rate and false negative rate than any individual anomaly detection algorithm.     

Spatio-Temporal Network Anomaly Detection by Assessing Deviations of Empirical Measures

We introduce an Internet traffic anomaly detection mechanism based on large deviations results for empirical measures. Using past traffic traces we characterize network traffic during various time-of-day intervals, assuming that it is anomaly-free. We present two different approaches to characterize traffic: (i) a model-free approach based on the method of types and Sanov's theorem, and (ii) a model-based approach modeling traffic using a Markov modulated process. Using these characterizations as a reference we continuously monitor traffic and employ large deviations and decision theory results to ldquocomparerdquo the empirical measure of the monitored traffic with the corresponding reference characterization, thus, identifying traffic anomalies in real-time. Our experimental results show that applying our methodology (even short-lived) anomalies are identified within a small number of observations. Throughout, we compare the two approaches presenting their advantages and disadvantages to identify and classify temporal network anomalies. We also demonstrate how our framework can be used to monitor traffic from multiple network elements in order to identify both spatial and temporal anomalies. We validate our techniques by analyzing real traffic traces with time-stamped anomalies.     

Effective forward-channel capacity of non-dedicated CDPD networks

Cellular digital packet data (CDPD) is a mobile packet data technology developed to operate on the spectrum assigned to a telephone cellular network. Since it operates on non-dedicated RF channels, it is subject to channel hopping and its performance is affected by the traffic profile of the underlying telephone system. We develop formulas expressing the normalized forward channel capacity in terms of telephone traffic conditions and channel hopping parameters. We show that forward capacity is very near to maximum for telephone network utilization less than one, while it degrades rapidly thereafter. Moreover, we develop an expression for the average hopping rate and we demonstrate the sensitivity of capacity with the number of RF channels     

一种互联网宏观流量异常检测方法

网络流量异常指网络中流量不规则地显著变化。网络短暂拥塞、分布式拒绝服务攻击、大范围扫描等本地事件或者网络路由异常等全局事件都能够引起网络的异常。网络异常的检测和分析对于网络安全应急响应部门非常重要,但是宏观流量异常检测需要从大量高维的富含噪声的数据中提取和解释异常模式,因此变得很困难。文章提出一种分析网络异常的通用方法,该方法运用主成分分析手段将高维空间划分为对应正常和异常网络行为的子空间,并将流量向量影射在正常子空间中,使用基于距离的度量来检测宏观网络流量异常事件。     

一种新的端到端采样网络流量测量方法

本文主要对端到端采样测量方法进行了研究,在大量实验的基础上提出了IP包头标识字段取模采样方法,并利用该采样方法进行了模拟实验,结果证明该方法具有效率高,采样比率易控制等特点。     

Early recognition of internet traffic based on signature inspection

The accurate and efficient classification of Internet traffic is the first and key step to accurate traffic management, network security and traffic analysis. The classic ways to identify flows is either inaccu-rate or inefficient, which are not suitable to be applied to real-time online classification. In this paper, we originally presented an early recognition method named Early Recognition Based on Deep Packet Inspec-tion (ERBDPI) based on deep packet inspection, after analyzing the distribution of payload signature be-tween packets of a flow in detail. The basic concept of ERBDPI is classifying flows based on the payload signature of their first some packets, so that we can identify traffic at the beginning of a flow connection. We compared the performance of ERBDPI with that of traditional sampling methods both synthetically and using real-world traffic traces. The result shows that ERBDPI can get a higher classification accuracy with a lower packet sampling rate, which makes it suitable to be applied to accurate real-time classification in high-speed links.     

Effective discovery of attacks using entropy of packet dynamics

Network-based attacks are so devastating that they have become major threats to network security. Early yet accurate warning of these attacks is critical for both operators and end users. However, neither speed nor accuracy is easy to achieve because both require effective extraction and interpretation of anomalous patterns from overwhelmingly massive, noisy network traffic. The intrusion detection system presented here is designed to assist in diagnosing and identifying network attacks. This IDS is based on the notion of packet dynamics, rather than packet content, as a way to cope with the increasing complexity of attacks. We employ a concept of entropy to measure time-variant packet dynamics and, further, to extrapolate this entropy to detect network attacks. The entropy of network traffic should vary abruptly once the distinct patterns of packet dynamics embedded in attacks appear. The proposed classifier is evaluated by comparing independent statistics derived from five well-known attacks. Our classifier detects those five attacks with high accuracy and does so in a timely manner.     

A survey on high-precision time synchronization techniques for optical datacenter networks and a zero-overhead microsecond-accuracy solution

A datacenter, which is a highly distributed multiprocessing system, needs to keep accurate track of time across a large number of machines. Precise time synchronization has become a critical component due to stringent requirements of several time critical applications such as real-time big data analytics, high-performance computing, and financial trading. Our study starts with a survey on the most relevant time synchronization techniques for datacenter networks. Then, we propose a zero-overhead microsecond-accuracy solution to synchronize a packet-switched optical network for datacenters. To achieve the desired time accuracy, we consider precision time protocol to synchronize the server clocks with a central controller clock. Zero-overhead is maintained by using data traffic to carry the time messages instead of a separate control channel. Through simulation, we show that microsecond level of time accuracy can be achieved. We also discuss the dependency of the accuracy on different traffic loads, traffic distributions, and packet lengths.     

All-Optical Wavelength-Bypassing Ring Communications Networks

We introduce an all-optical WDM packet communication network that performs wavelength bypassing at the routers. Packets that arrive at a wavelength (optical cross-connect) router at designated wavelengths are switched by the router without having their headers examined. Thus, the processing element of the router is bypassed by such packets. For packet traffic that uses wavelengths that do not bypass a switch, the headers of such packets are examined to determine if this switch is the destination for the flow. If latter is the case, the packet is removed. Otherwise, the packet is switched to a pre-determined output without incurring (network internal) queueing delays. We study a ring network with routers that employ such a WDM bypassing scheme. We present methods to construct wavelength graphs that define the bypassing pattern employed by the routers to guide the traffic flows distributed at each given wavelength. Performance is measured in terms of the network throughput and the average processing path length (i.e., the average number of switches not being bypassed). For a fixed total processing capacity, we show that a WDM bypassing ring network provides a higher throughput level than that exhibited by a non-bypassing ring network, using the same value of total link capacity. By using WDM bypassing, the average processing path length (and thus the packet latency) is reduced. We study a multitude of network loading configurations, corresponding to distinct traffic matrices and client-server scenarios. Higher throughput levels are obtained for network configurations driven by non-uniform traffic matrices. The demonstrated advantages of WDM bypassing methods shown here for WDM ring networks are also applicable to more general network topological layouts.     

一种基于数据平面可编程的软件定义网络报文转发验证机制

针对软件定义网络(SDN)中OpenFlow协议匹配字段固定且数量有限,数据流转发缺少有效的转发验证机制等问题,该文提出一种基于数据平面可编程的软件定义网络报文转发验证机制。通过为数据报文添加自定义密码标识,将P4转发设备加入基于OpenFlow的软件定义网络,在不影响数据流正常转发的基础上,对网络业务流精确控制和采样。控制器验证采样业务报文完整性,并针对异常报文下发流规则至OpenFlow转发设备,对恶意篡改、伪造等异常数据流进行转发控制。最后,构建基于开源BMv2的P4转发设备和基于OpenFlow的Open vSwitch转发设备的转发验证原型,并构建仿真网络进行实验。实验结果表明,该机制能够有效检测业务报文篡改、伪造等转发异常行为,与同类验证机制相比,在安全验证处理开销保持不变的情况下,能够实现更细粒度的业务流精确控制采样和更低的转发时延。