Inverting sampled traffic

Routers have the ability to output statistics about packets and flows of packets that traverse them. Since, however, the generation of detailed traffic statistics does not scale well with link speed, increasingly routers and measurement boxes implement sampling strategies at the packet level. In this paper, we study both theoretically and practically what information about the original traffic can be inferred when sampling, or "thinning", is performed at the packet level. While basic packet level characteristics such as first order statistics can be fairly directly recovered, other aspects require more attention. We focus mainly on the spectral density, a second-order statistic, and the distribution of the number of packets per flow, showing how both can be exactly recovered, in theory. We then show in detail why in practice this cannot be done using the traditional packet based sampling, even for high sampling rate. We introduce an alternative flow-based thinning, where practical inversion is possible even at arbitrarily low sampling rate. We also investigate the theory and practice of fitting the parameters of a Poisson cluster process, modeling the full packet traffic, from sampled data.     

Detection accuracy of network anomalies using sampled flow statistics

We investigated the detection accuracy of network anomalies when using flow statistics obtained through packet sampling. Through a case study based on measurement data, we showed that network anomalies generating a large number of small flows, such as network scans or SYN flooding, become difficult to detect during packet sampling. We then developed an analytical model that enables us to quantitatively evaluate the effect of packet sampling and traffic conditions, such as anomalous traffic volume, on detection accuracy. We also investigated how the detection accuracy worsens when the packet sampling rate decreases. In addition, we show that, even with a low sampling rate, spatially partitioning monitored traffic into groups makes it possible to increase detection accuracy. We also developed a method of determining an appropriate number of partitioned groups, and we show its effectiveness. Copyright © 2011 John Wiley & Sons, Ltd.     

On wavelength-converted optical routers employing flow routing

Optical networks have been extensively investigated in recent years to provide high capacity for the Internet traffic. Among them the optical packet-switching network deploying buffering, wavelength conversion and multipath routing could be the most suitable one. It cannot only provide high capacity transport for Internet traffic but also achieve high utilization of the network resources. However due to the packet-oriented routing and switching, such a network can result in a large amount of packets out-of-order, packet loss and/or with various delays upon arriving at end systems, causing TCP flows that comprise those packets corrupted. Large amount of corrupted flows can increase the burstiness of the Internet traffic and cause higher-layer protocol to malfunction. This paper presents a novel routing and switching method for optical IP networks-flow routing. Without using a complicate control mechanism flow routing deals with packet-flows to reduce the amount of corrupted flows. The performance of the wavelength-converted optical flow router is investigated, based on a novel analytical model. A performance metric, i.e., good-throughput, is used, measuring the ratio of the amount of packets comprised in the noncorrupted flows to total amount of packets. Comparing with optical packet-switching routers, a remarkable improvement of good-throughput can be obtained by using optical flow routers. More important, using wavelength conversion can greatly improve the good-throughput of optical flow routers.     

Early recognition of internet traffic based on signature inspection

The accurate and efficient classification of Internet traffic is the first and key step to accurate traffic management, network security and traffic analysis. The classic ways to identify flows is either inaccu-rate or inefficient, which are not suitable to be applied to real-time online classification. In this paper, we originally presented an early recognition method named Early Recognition Based on Deep Packet Inspec-tion (ERBDPI) based on deep packet inspection, after analyzing the distribution of payload signature be-tween packets of a flow in detail. The basic concept of ERBDPI is classifying flows based on the payload signature of their first some packets, so that we can identify traffic at the beginning of a flow connection. We compared the performance of ERBDPI with that of traditional sampling methods both synthetically and using real-world traffic traces. The result shows that ERBDPI can get a higher classification accuracy with a lower packet sampling rate, which makes it suitable to be applied to accurate real-time classification in high-speed links.     

A Factor Analytic Approach to Inferring Congestion Sharing Based on Flow Level Measurements

Internet traffic primarily consists of packets from elastic flows, i.e., Web transfers, file transfers, and e-mail, whose transmissions are mediated via the Transmission Control Protocol (TCP). In this paper, we develop a methodology to process TCP flow measurements in order to analyze throughput correlations among TCP flow classes that can be used to infer congestion sharing in the Internet. The primary contributions of this paper are: 1) development of a technique for processing flow records suitable for inferring congested resource sharing; 2) evaluation of the use of factor analysis on processed flow records to explore which TCP flow classes might share congested resources; and 3)validation of our inference methodology using bootstrap methods and nonintrusive, flow level measurements collected at a single network site. Our proposal for using flow level measurements to infer congestion sharing differs significantly from previous research that has employed packet level measurements for making inferences. Possible applications of our method include network monitoring and root cause analysis of poor performance     

Trajectory sampling for direct traffic observation

Traffic measurement is a critical component for the control and engineering of communication networks. We argue that traffic measurement should make it possible to obtain the spatial flow of traffic through the domain, i.e., the paths followed by packets between any ingress and egress point of the domain. Most resource allocation and capacity planning tasks can benefit from such information. Also, traffic measurements should be obtained without a routing model and without knowledge of network state. This allows the traffic measurement process to he resilient to network failures and state uncertainty. We propose a method that allows the direct inference of traffic flows through a domain by observing the trajectories of a subset of all packets traversing the network. The key advantages of the method are that (1) it does not rely on routing state; (2) its implementation cost is small; and (3) the measurement reporting traffic is modest and can be controlled precisely. The key idea of the method is to sample packets based on a hash function computed over the packet content. Using the same hash function will yield the same sample set of packets in the entire domain, and enables us to reconstruct packet trajectories     

Inference of flow statistics via packet sampling in the internet

We show in this note that by deterministic packet sampling, the tail of the distribution of the original flow size can be obtained by rescaling that of the sampled flow size. To recover information on the flow size distribution lost through packet sampling, we propose a parametric method based on measurements from different backbone IP networks. This method allows us to recover the complete flow size distribution and has successfully been tested by using a real ADSL traffic trace.     

基于抽样流长与完全抽样阈值的异常流自适应抽样算法

高速IP网络的流量测量与异常检测是网络测量领域研究的热点。针对目前网络流量测量算法对小流估计精度偏低,对异常流量筛选能力较差的缺陷,该文提出一种基于业务流已抽样长度与完全抽样阈值S的自适应流抽样算法(AFPT)。AFPT算法根据完全抽样阈值S筛选对异常流量敏感相关的小流,同时根据业务流已抽样长度自适应调整抽样概率。仿真和实验结果表明,AFPT算法的估计误差与理论上界相符,具有较强的异常流量筛选能力,能够有效提高异常检测算法的准确率。     

Impact of Packet Sampling on Portscan Detection

Packet sampling is commonly deployed in high-speed backbone routers to minimize resources used for network monitoring. It is known that packet sampling distorts traffic statistics and its impact has been extensively studied for traffic engineering metrics such as flow size and mean rate. However, it is unclear how packet sampling impacts anomaly detection, which has become increasingly critical to network providers. This paper is the first attempt to address this question by focusing on one common class of nonvolume-based anomalies, portscans, which are associated with worm/virus propagation. Existing portscan detection algorithms fall into two general approaches: target-specific and traffic profiling. We evaluated representative algorithms for each class, namely: 1) TRWSYN that performs stateful traffic analysis; 2) TAPS that tracks connection pattern of scanners; and 3) entropy-based traffic profiling. We applied these algorithms to detect portscans in both the original and sampled packet traces from a Tier-1 provider's backbone network. Our results demonstrate that sampling introduces fundamental bias that degrades the effectiveness of these detection algorithms and dramatically increases false positives. Through both experiments and analysis, we identify the traffic features critical for anomaly detection that are affected by sampling. Finally, using insight gained from this study, we show how portscan algorithms can be enhanced to be more robust to sampling     

Trajectory Sampling With Unreliable Reporting

We define and evaluate methods to perform robust network monitoring using trajectory sampling in the presence of report loss. The first challenge is to reconstruct an unambiguous set of packet trajectories from the reports on sampled packets received at a collector. In this paper we extend the reporting paradigm of trajectory sampling to enable the elimination of ambiguous groups of reports, but without introducing bias into any characterization of traffic based on the surviving reports. Even after the elimination, a proportion of trajectories are incomplete due to report loss. A second challenge is to adapt measurement based applications (including network engineering, path tracing, and passive performance measurement) to incomplete trajectories. To achieve this, we propose a method to join multiple incomplete trajectories for inference, and analyze its performance. We also show how applications can distinguish between packet and report loss at the statistical level.     

一种新的IPv6网络带宽测量方法

该文提出一种新的应用于IPv6网络的带宽测量方法PTTS(Packet Train Time Stamp)。源端主动向网络发送报文序列(Mh-L-Mt packets train),序列中负载报文反应网络流量特征,测试报文带有IPv6时间戳扩展报文头,逐跳记录路由器的当前时间,获得报文序列通过链路所花费时间,得到可用带宽。同时利用IPv6基本报文头中流标签字段定义测试流,确保报文列中测试报文和负载报文路径一致;利用流量类型字段,增设用于网络测量的测试级消除其与背景业务的相互影响。仿真证明,报文设计合理,测量方法可行。     

On‐line detection of persistently high packet‐rate flows via a sliding window scheme with random packet sampling

We consider on‐line detection of persistently high packet‐rate flows. We assume that flow information is collected via a time‐based sliding window scheme with random packet sampling. In this framework, we propose a method of determining the threshold of the number of sampled packets, which guarantees the false negative ratio. We also formulate and solve the design problem of our scheme, where we aim to minimize the false positive ratio. We then conduct sampling experiments with public trace data and confirm that our method works well as designed. Copyright © 2013 John Wiley & Sons, Ltd.     

Learn more, sample less: control of volume and variance in network measurement

This paper deals with sampling objects from a large stream. Each object possesses a size, and the aim is to be able to estimate the total size of an arbitrary subset of objects whose composition is not known at the time of sampling. This problem is motivated from network measurements in which the objects are flow records exported by routers and the sizes are the number of packet or bytes reported in the record. Subsets of interest could be flows from a certain customer or flows from a worm attack. This paper introduces threshold sampling as a sampling scheme that optimally controls the expected volume of samples and the variance of estimators over any classification of flows. It provides algorithms for dynamic control of sample volumes and evaluates them on flow data gathered from a commercial Internet Protocol (IP) network. The algorithms are simple to implement and robust to variation in network conditions. The work reported here has been applied in the measurement infrastructure of the commercial IP network. To not have employed sampling would have entailed an order of magnitude greater capital expenditure to accommodate the measurement traffic and its processing.     

A random early demotion and promotion marker for assured services

The differentiated services (DiffServ) model, proposed to evolve the current best-effort Internet to a quality-of-service-aware Internet, provides packet level service differentiation on a per-hop basis. The end-to-end service differentiation may be provided by extending the per-hop behavior over multiple network domains through service level agreements between domains. The edge routers of each of the domains monitor the aggregate flow of the incoming packets and demote packets when the aggregate incoming traffic exceeds the negotiated interdomain service agreement. A demoted packet may encounter other edge routers on its path that have sufficient resources to route the packet with its original marking. In this paper, we propose a random early demotion and promotion (REDP) technique that works at the aggregate traffic level and allows (1) fair demotion of packets belonging to different flows, and (2) easy and fair detection and promotion of the demoted packets. Using early and random decisions on packets REDP ensures fairness in promotion and demotion. It uses a three color marking mechanism, reserving one color fur differentiating between a demoted packet and a packet with the original out-of-profile marking. We experiment with the proposed REDP scheme using the ns2 simulator for both TCP and UDP streams. The results demonstrate the fairness of REDP scheme in demoting and promoting packets. Furthermore, we show a variety of results that demonstrates that REDP provides better assured services compared to the previously proposed RIO scheme with or without the provision of promotion     

A guaranteed minimum throughput service for TCP flows using measurement‐based admission control

We propose a new scheme for a network service that guarantees a minimum throughput to flows accepted by admission control (AC). The whole scheme only uses a small set of packet classes in a core‐stateless network. At the ingress of the network each flow packet is marked into one of the sets of classes, and within the network, each class is assigned a different discarding priority. The AC method is based on edge‐to‐edge per‐flow throughput measurements using the first packets of the flow, and it requires flows to send with a minimum rate. We evaluate the scheme through simulations in a simple bottleneck topology with different traffic loads consisting of TCP flows that carry files of varying sizes. We use a modified TCP source with a new algorithm that forces the source to send with a minimum rate. We compare our scheme with the best‐effort service and we study the influence of the measurement duration on the scheme's performance. The results prove that the scheme guarantees the requested throughput to accepted flows and achieves a high utilization of network resources. Copyright © 2006 John Wiley & Sons, Ltd.     

Space-Code Bloom Filter for Efficient Per-Flow Traffic Measurement

Per-flow traffic measurement is critical for usage accounting, traffic engineering, and anomaly detection. Previous methodologies are either based on random sampling (e.g., Cisco's NetFlow), which is inaccurate, or only account for the "elephants." We introduce a novel technique for measuring per-flow traffic approximately, for all flows regardless of their sizes, at very high-speed (say, OC768). The core of this technique is a novel data structure called Space-Code Bloom Filter (SCBF). A SCBF is an approximate representation of a multiset; each element in this multiset is a traffic flow and its multiplicity is the number of packets in the flow. The multiplicity of an element in the multiset represented by SCBF can be estimated through either of two mechanisms-maximum-likelihood estimation or mean value estimation. Through parameter tuning, SCBF allows for graceful tradeoff between measurement accuracy and computational and storage complexity. SCBF also contributes to the foundation of data streaming by introducing a new paradigm called blind streaming. We evaluate the performance of SCBF through mathematical analysis and through experiments on packet traces gathered from a tier-1 ISP backbone. Our results demonstrate that SCBF achieves reasonable measurement accuracy with very low storage and computational complexity. We also demonstrate the application of SCBF in estimating the frequency of keywords at a search engine-demonstrating the applicability of SCBF to other problems that can be reduced to multiset membership queries     

CL-MAC: A Cross-Layer MAC protocol for heterogeneous Wireless Sensor Networks

In duty cycled MAC protocols, multi-packet, multi-flow and multi-hop traffic patterns experience significant latencies, which are partially due to duty cycling. Several cross-layer routing/MAC schemes have been proposed to mitigate this latency. However, they utilize routing information from a single flow and/or a single packet perspective, thus limiting their adaptation to varying traffic loads and patterns. In this paper, we propose a novel Cross-Layer MAC protocol (CL-MAC) for WSNs, to efficiently handle multi-packet, multi-hop and multi-flow traffic patterns while adapting to a wide range of traffic loads. CL-MAC’s scheduling is based on a unique structure of flow setup packets that efficiently utilize routing information to transmit multiple data packets over multiple multi-hop flows. Unlike other MAC protocols, supporting construction of multi-hop flows, CL-MAC considers all pending packets in the routing layer buffer and all flow setup requests from neighbors, when setting up a flow. This allows CL-MAC to make more informed scheduling decisions, reflecting the current network status, and dynamically optimize its scheduling mechanism accordingly. We evaluate CL-MAC through extensive ns-2 simulations and compare its performance to the state of the art, over various networks and for a wide variety of traffic loads and patterns. In all our experiments, CL-MAC substantially reduces end-to-end latency, increases delivery ratio while reducing the average energy consumed per packet delivered.     

Ballot theorems applied to the transient analysis of nD/D/1 queues

The problem of transporting constant-bit-rate (CBR) traffic through a packet network is analyzed. In the system considered, CBR traffic is packetized and packets from several similar sources are multiplexed on a transmission link. The bit streams are recreated at the receiving end by demultiplexing the packets and then playing out the packets of each CBR stream. Traffic fluctuations may cause gaps to appear in the playout process. Their frequency can be reduced by adding a smoothing delay to each stream. The queueing system analyzed has periodic arrivals and deterministic service times. A method of analysis, based on the ballot theorems of Takacs (1967), is presented to provide steady-state delay distributions as well as a transient analysis of the system to predict the statistics of the time for a gap to develop in the CBR stream as a function of the smoothing delay     

Wide-area IP multicast traffic characterization

IP multicast is gaining acceptance among service providers as the protocols and infrastructure mature. However, characteristics of multicast traffic remain poorly understood. Using passive OC-12 monitors, we observed multicast traffic on links connecting aggregated customers and peer networks to our native multicast backbone network. We first refined existing traffic flow profiling methodologies via an exploration of temporal differences in multicast packet trains. Based on this framework, we collected multicast flow traces from four geographically dispersed nodes in the Worldcom vBNS network over a one-month period. We present multicast-specific traffic characteristics including packet and flow sizes, fragmentation, sources per group, and address space distribution. Analysis reveals results contrary to prevailing wisdom, including a preponderance of single-packet flows; a highly variable packet size distribution, with many large packets and strong modes; the existence of fragmented multicast traffic; and an insignificant number of simultaneous multiple-source groups. Based on our analysis, we recommend policies for deployment and improvements to protocol implementations.     

Fast, Memory Efficient Flow Rate Estimation Using Runs

Per-flow network traffic measurements are needed for effective network traffic management, network performance assessment, and detection of anomalous network events such as incipient denial-of-service (DoS) attacks. Explicit measurement of per-flow traffic statistics is difficult in backbone networks because tracking the possibly hundreds of thousands of flows needs correspondingly large high-speed memories. To reduce the measurement overhead, many previous papers have proposed the use of random sampling and this is also used in commercial routers (Cisco's NetFlow). Our goal is to develop a new scheme that has very low memory requirements and has quick convergence to within a pre-specified accuracy. We achieve this by use of a novel approach based on sampling two-runs to estimate per-flow traffic. (A flow has a two-run when two consecutive samples belong to the same flow). Sampling two-runs automatically biases the samples towards the larger flows thereby making the estimation of these sources more accurate. This biased sampling leads to significantly smaller memory requirement compared to random sampling schemes. The scheme is very simple to implement and performs extremely well.