IDGraphs: intrusion detection and analysis using stream compositing

IDGraphs is an interactive visualization system, supporting intrusion detection over massive network traffic streams. It features a novel time-versus-failed-connections mapping that aids in discovery of attack patterns. The number of failed connections (SYN-SYN/ACK) is a strong indicator of suspicious network flows. IDGraphs offers several flow aggregation methods that help reveal different attack patterns. The system also offers high visual scalability through the use of Histographs. The IDGraphs intrusion detection system detects and analyzes a variety of attacks and anomalies, including port scanning, worm outbreaks, stealthy TCP SYN flooding, and some distributed attacks. In this article, we demonstrate IDGraphs using a single day of NetFlow network traffic traces collected at edge routers at Northwestern University which has several OC-3 links.     

An Architectural Framework for Accurate Characterization of Network Traffic

In networks carrying large volume of traffic, accurate traffic characterization is necessary for understanding the dynamics and patterns of network resource usage. Previous approaches to flow characterization are based on random sampling of the packets (e.g., Cisco's NetFlow) or inferring characteristics solely based on long lived flows (LLFs) or on lossy data structures (e.g., bloom filters, hash tables). However, none of these approaches takes into account the heavy-tailed nature of the Internet traffic and separates the estimation algorithm from the flow measurement architecture.In this paper, we propose an alternate approach to traffic characterization by closely linking the flow measurement architecture with the estimation algorithm. Our measurement framework stores complete information related to short lived flows (SLFs) while collecting partial information related to LLFs. For real-time separation of LLFs and SLFs, we propose a novel algorithm based on typical sequences from Information theory. The distribution (pdf) and sample space of the underlying traffic is estimated using the non-parametric Parzen window technique and likelihood function defined over the Coupon collector problem. We validate the accuracy and performance of our estimation technique using traffic traces from the internal LAN in our laboratory and from National Library for Applied Network Research (NLANR).     

NetFlow流量采集与存储技术的研究实现*

为解决高速大流量网络的流量监测与分析问题,提出了一套基于NetFlow的流量采集与存储方案,设计了多线程与双链表的NetFlow数据采集机制,有效提高了数据采集效率和可靠性.此外,在存储原始NetFlow数据的基础上,设计了一套NetFlow流量的三级聚合和存储方案.基于此方法可对纷繁复杂的原始流量信息进行有效整理,为前端静态、动态流量分析提供合理高效的数据支持.     

A flow measurement architecture to preserve application structure

The Internet has significantly evolved in the number and variety of applications. Network operators need mechanisms to constantly monitor and study these applications. Modern routers employ passive measurement solution called Sampled NetFlow to collect basic statistics on a per-flow basis (for a small subset of flows), that could provide valuable information for application monitoring. Given modern applications routinely consist of several flows, potentially to many different destinations, only a few flows are sampled per application session using Sampled NetFlow. To address this issue, in this paper, we introduce related sampling that allows network operators to give a higher probability to flows that are part of the same application session. Given the lack of application semantics in the middle of the network, our architecture, RelSamp, treats flows that share the same source IP address as related. Our heuristic works well in practice as hosts typically run few applications at any given instant, as observed using a measurement study on real traces. In our evaluation using real traces, we show that RelSamp achieves 5–10× more flows per application session compared to Sampled NetFlow for the same effective number of sampled packets. We also show that behavioral and statistical classification approaches such as BLINC, SVM and C4.5 achieve up to 50% better classification accuracy compared to Sampled NetFlow, while not impairing existing management tasks such as volume estimation too much.     

Semisupervised learning of hierarchical latent trait models for data visualization

Recently, we have developed the hierarchical generative topographic mapping (HGTM), an interactive method for visualization of large high-dimensional real-valued data sets. We propose a more general visualization system by extending HGTM in three ways, which allows the user to visualize a wider range of data sets and better support the model development process. 1) We integrate HGTM with noise models from the exponential family of distributions. The basic building block is the latent trait model (LTM). This enables us to visualize data of inherently discrete nature, e.g., collections of documents, in a hierarchical manner. 2) We give the user a choice of initializing the child plots of the current plot in either interactive, or automatic mode. In the interactive mode, the user selects "regions of interest", whereas in the automatic mode, an unsupervised minimum message length (MML)-inspired construction of a mixture of LTMs is employed. The unsupervised construction is particularly useful when high-level plots are covered with dense clusters of highly overlapping data projections, making it difficult to use the interactive mode. Such a situation often arises when visualizing large data sets. 3) We derive general formulas for magnification factors in latent trait models. Magnification factors are a useful tool to improve our understanding of the visualization plots, since they can highlight the boundaries between data clusters. We illustrate our approach on a toy example and evaluate it on three more complex real data sets.     

Flow Mapping and Multivariate Visualization of Large Spatial Interaction Data

Spatial interactions (or flows), such as population migration and disease spread, naturally form a weighted location-to-location network (graph). Such geographically embedded networks (graphs) are usually very large. For example, the county-to-county migration data in the U.S. has thousands of counties and about a million migration paths. Moreover, many variables are associated with each flow, such as the number of migrants for different age groups, income levels, and occupations. It is a challenging task to visualize such data and discover network structures, multivariate relations, and their geographic patterns simultaneously. This paper addresses these challenges by developing an integrated interactive visualization framework that consists three coupled components: (1) a spatially constrained graph partitioning method that can construct a hierarchy of geographical regions (communities), where there are more flows or connections within regions than across regions; (2) a multivariate clustering and visualization method to detect and present multivariate patterns in the aggregated region-to-region flows; and (3) a highly interactive flow mapping component to map both flow and multivariate patterns in the geographic space, at different hierarchical levels. The proposed approach can process relatively large data sets and effectively discover and visualize major flow structures and multivariate relations at the same time. User interactions are supported to facilitate the understanding of both an overview and detailed patterns.     

Visualizing Multidimensional Lifelogging Data: A Case Study on MyMovieHistory Project

ABSTRACT

A large amount of time-series data has been frequently used to extract the useful patterns and trends and to visualize them for better understanding. This work is focusing on visualizing personal lifelogging data for tracking back to personal histories. Thereby, we present several similarity measures between multidimensional data at two different time points. For human evaluation, the method has been applied to MyMovieHistory (which a social recommendation system by storing personal movie logs) and tested with many users. Experimental results shown that the proposed visualization method and interfaces can help to understand user history.     

Flow-inspector: a framework for visualizing network flow data using current web technologies

New web technologies led to the development of browser applications for data analysis. Modern browser engines allow for building interactive real-time visualization applications that enable efficient ways to understand complex data. We present Flow-Inspector, a highly interactive open-source web framework for visualizing network flow data using latest web technologies. Flow-Inspector includes a backend for processing and storing large-scale network flow data, as well as a JavaScript-based web application capable to display and manipulate traffic information in real-time. This work provides operators with a toolkit to analyze their networks and enables the scientific community to create new and innovative visualizations of traffic data with an extensible framework. We demonstrate the applicability of our approach by implementing several different visualization components that help to identify topological characteristics in network flows.     

Interactive exploration of data traffic with Hierarchical Network Maps

Network communication has become indispensable in business, education and government. With the pervasive role of the Internet as a means of sharing information across networks, its misuse for destructive purposes, such as spreading malicious code, compromising remote hosts, or damaging data through unauthorized access, has grown immensely in the recent years. The classical way of monitoring the operation of large network systems is by analyzing the system logs for detecting anomalies. In this work, we introduce hierarchical network map, an interactive visualization technique for gaining a deeper insight into network flow behavior by means of user-driven visual exploration. Our approach is meant as an enhancement to conventional analysis methods based on statistics or machine learning. We use multidimensional modeling combined with position and display awareness to view source and target data of the hosts in a hierarchical fashion with the ability to interactively change the level of aggregation or apply filtering. The interdisciplinary approach integrating data warehouse technology, information visualization and decision support brings about the benefit of efficiently collecting the input data and aggregating over very large data sets, visualizing the results and providing interactivity to facilitate analytical reasoning     

基于业务流数量自适应的资源限制分组抽样

高速网络中,流量抽样测量技术是一种重要可扩展的解决方案,其中NetFlow在流量测量中有着广泛的应用。针对NetFlow的缺陷提出了一种基于业务流数量自适应的资源限制分组抽样算法,该算法结合 “分层抽样”的思想,把 “累积业务流数量”作为重要的参数,来自适应地调节抽样概率,该抽样方法简单、易于实现,平衡了资源的消耗量和准确性。并基于实际互联网数据进行了实验比较,结果显示：该方法具有简单性、自适应性、资源可控性的同时不会失去准确性。     

Network visualization with Nam, the VINT network animator

Network protocol designers face many difficult tasks, including simultaneously monitoring state in a potentially large number of nodes, understanding and analyzing complex message exchanges, and characterizing dynamic interactions with competing traffic. Traditionally they have used packet traces to accomplish these tasks, but traces have two major drawbacks: they present an incredible amount of detail, which challenges the designer's ability to comprehend the data; and they are static, which hides an important dimension of protocol behavior. As a result, detailed analysis frequently becomes tedious and error-prone. Although network simulators such as the VINT project's ns can easily generate numerous detailed traces, they provide limited help for analyzing and understanding the data. Nam, the network animator that we developed in our work at the VINT project, provides packet-level animation, protocol graphs, traditional time-event plots of protocol actions, and scenario editing capabilities. Nam benefits from a close relationship with ns, which can collect detailed protocol information from a simulation. With some preprocessing. Nam can visualize data taken directly from real network traces     

Visualizing temporal patterns in large multivariate data using textual pattern matching

Extracting and visualizing temporal patterns in large scientific data is an open problem in visualization research. First, there are few proven methods to flexibly and concisely define general temporal patterns for visualization. Second, with large time-dependent data sets, as typical with today's large-scale simulations, scalable and general solutions for handling the data are still not widely available. In this work, we have developed a textual pattern matching approach for specifying and identifying general temporal patterns. Besides defining the formalism of the language, we also provide a working implementation with sufficient efficiency and scalability to handle large data sets. Using recent large-scale simulation data from multiple application domains, we demonstrate that our visualization approach is one of the first to empower a concept driven exploration of large-scale time-varying multivariate data.     

MViewer: mobile phone spatiotemporal data viewer

Nowadays movement patterns and people’s behavioral models are needed for traffic engineers and city planners. These observations could be used to reason about mobility and its sustainability and to support decision makers with reliable information. The very same knowledge about human diaspora and behavior extracted from these data is also valuable to the urban planner, so as to localize new services, organize logistics systems and to detect changes as they occur in the movement behavior. Moreover, it is interesting to investigate movement in places like a shopping area or a working district either for commercial purposes or for improving the service quality. These kinds of tracking data are made available by wireless and mobile communication technologies. It is now possible to record and collect a large amount of mobile phone calls in a city. Technologies for object tracking have recently become affordable and reliable and hence we were able to collect mobile phone data from a city in China from January 1, 2008 to December 31, 2008. The large amount of phone call records from mobile operators can be considered as life mates and sensors of persons to inform howmany people are present in any given area and how many are entering or leaving. Each phone call record usually contains the caller and callee IDs, date and time, and the base station where the phone calls are made. As mobile phones are widely used in our daily life, many human behaviors can be revealed by analyzing mobile phone data. Through mobile phones, we can learn the information about locations, communications between mobile phone users during their daily lives. In this work, we propose a comprehensive visual analysis system named as MViewer, Mobile phone spatiotemporal data Viewer, which is the first system to visualize and analyze the population’smobility patterns from millions of phone call records. Our system consists of three major components: 1) visual analysis of user groups in a base station; 2) visual analysis of the mobility patterns on different user groups making phone calls in certain base stations; 3) visual analysis of handoff phone call records. Some well-established visualization techniques such as parallel coordinates and pixelbased representations have been integrated into our system. We also develop a novel visualization schemes, Voronoidiagram-based visual encoding to reveal the unique features of mobile phone data. We have applied our system to real mobile phone datasets that are kindly provided by our project partners and obtained some interesting findings regarding people’s mobility patterns.     

基于NetFlow的网络异常流量检测

NetFlow可以提供网络中IP流的信息。这些流的信息有多种用途，包括网管、网络规划、ISP计费等。在网络安全领域，NetFlow提供的IP流信息可以用来分析网络中的异常流量，这是对现有的基于特征的NIDS的很好的补充。本文介绍了Net—Flow—based Anomaly Traffic Analyzer，一个基于NetFlow的网络异常流量检测系统，并通过一些实验证明了该系统的有效性。     

Sequential hashing: A flexible approach for unveiling significant patterns in high speed networks

Identification of significant patterns in network traffic, such as IPs or flows that contribute large volume (heavy hitters) or those that introduce large changes of volume (heavy changers), has many applications in accounting and network anomaly detection. As network speed and the number of flows grow rapidly, identifying heavy hitters/changers by tracking per-IP or per-flow statistics becomes infeasible due to both the computational overhead and memory requirements. In this paper, we propose SeqHash, a novel sequential hashing scheme that supports fast and accurate recovery of heavy hitters/changers, while requiring memory just slightly higher than the theoretical lower bound. SeqHash monitors data traffic using a sketch data structure that can flexibly trade-off between the memory usage and the computational overhead in a large range that can be utilized by different computer architectures for optimizing the overall performance. In addition, we propose statistically efficient algorithms for estimating the values of heavy hitters/changers. Using both mathematical analysis and experimental studies of Internet traces, we demonstrate that SeqHash can achieve the same accuracy as the existing methods do but using much less memory and computational overhead.     

A coherent grid traversal approach to visualizing particle-based simulation data

We present an approach to visualizing particle-based simulation data using interactive ray tracing and describe an algorithmic enhancement that exploits the properties of these data sets to provide highly interactive performance and reduced storage requirements. This algorithm for fast packet-based ray tracing of multilevel grids enables the interactive visualization of large time-varying data sets with millions of particles and incorporates advanced features like soft shadows. We compare the performance of our approach with two recent particle visualization systems: one based on an optimized single ray grid traversal algorithm and the other on programmable graphics hardware. This comparison demonstrates that the new algorithm offers an attractive alternative for interactive particle visualization.     

Resource allocation for multimedia traffic flows using rate variance envelopes

In order for networks to support the delay and loss requirements of interactive multimedia applications, resource management algorithms are needed that efficiently allocate network resources. In this paper, we introduce a new resource allocation scheme based on rate variance envelopes. Such envelopes capture a flow's burstiness properties and autocorrelation structure by characterizing the variance of its rate distribution over intervals of different length. From this traffic characterization, we develop a simple and efficient resource allocation algorithm for static priority schedulers by employing a Gaussian approximation over intervals and considering a maximal busy period. Our approach supports heterogeneous quality-of-service requirements via our consideration of prioritized service disciplines, and supports heterogeneous and bursty traffic flows via our general framework of traffic envelopes. To evaluate the scheme, we perform trace-driven simulation experiments with long traces of compressed video and show that our approach is accurate enough to capture most of the available statistical multiplexing gain, achieving average network utilizations of up to 90% for these traces and substantially outperforming alternate schemes.     

Spatio-temporal facility utilization analysis from exhaustive WiFi monitoring

The optimization of logistics in large building complexes with many resources, such as hospitals, require realistic facility management and planning. Current planning practices rely foremost on manual observations or coarse unverified assumptions and therefore do not properly scale or provide realistic data to inform facility planning. In this paper, we propose analysis methods to extract knowledge from large sets of network collected WiFi traces to better inform facility management and planning in large building complexes. The analysis methods, which build on a rich set of temporal and spatial features, include methods for quantification of area densities, as well as flows between specified locations, buildings or departments, classified according to the feature set. Spatio-temporal visualization tools built on top of these methods enable planners to inspect and explore extracted information to inform facility-planning activities. To evaluate the proposed methods and visualization tools, we present facility utilization analysis results for a large hospital complex covering more than 10 hectares. The evaluation is based on WiFi traces collected in the hospital’s WiFi infrastructure over two weeks observing around 18000 different devices recording more than a billion individual WiFi measurements. We highlight the tools’ ability to deduce people’s presences and movements and how they can provide respective insights into the test-bed hospital by investigating utilization patterns globally as well as selectively, e.g. for different user roles, daytimes, spatial granularities or focus areas.     

Vortex visualization for practical engineering applications

In order to understand complex vortical flows in large data sets, we must be able to detect and visualize vortices in an automated fashion. In this paper, we present a feature-based vortex detection and visualization technique that is appropriate for large computational fluid dynamics data sets computed on unstructured meshes. In particular, we focus on the application of this technique to visualization of the flow over a serrated wing and the flow field around a spinning missile with dithering canards. We have developed a core line extraction technique based on the observation that vortex cores coincide with local extrema in certain scalar fields. We also have developed a novel technique to handle complex vortex topology that is based on k-means clustering. These techniques facilitate visualization of vortices in simulation data that may not be optimally resolved or sampled. Results are included that highlight the strengths and weaknesses of our approach. We conclude by describing how our approach can be improved to enhance robustness and expand its range of applicability.     

基于多数据源的网络流量矩阵估计

针对目前多数网络流量矩阵估计方法都采用单一的SNMP链路负载或抽样的NetFlow数据的问题,提出一种综合使用多个数据源进行流量矩阵估计的方法,将SNMP链路负载与抽样的NetFlow数据结合作为相互纠错码,设计过滤这2种数据源中脏数据的算法。以校园网为实验环境,与通用重力模型方法的比较结果表明,利用该算法去除脏数据后进行流量矩阵估计具有更高的准确性。