一种基于Hadoop平台CloudSVM的网络流量分类方法

大规模的netflow训练数据集是构建高质量、高稳定网络流量分类器的必然要求。但随着网络流特征维数的提高和数据集规模的扩大,无论是网络流的分析处理还是基于支持向量机(SVM)的分类器模型的训练,都无法在有效的时间内得到有效的处理结果。本文基于Hadoop云计算平台,采用MapReduce技术对SVM网络流量分类器进行分布式学习和训练,构建CloudSVM网络流量分类器。通过对来自校园网出口镜像的近2 T的大规模网络流量的跟踪文件的分布式存储和处理,对抽取的样本数据集进行分类,实验验证了基于Hadoop平台分布式存储和并行处理大规模网络数据集的高效率性,也验证了CloudSVM分类器在不降低分类准确度的情况下可以快速收敛到最佳,并随着大规模网络流样本的增加,SVM分类器训练的时间趋近平稳。     

基于云计算的海量网络流量数据分析处理研究

随着互联网技术的迅速发展,在我们的生活中网络已经成为我们不可以缺少的重要的组成部分.网络流量监控技术是对网络中海量的流量数据进行分析的重要工具和技术.基于云计算的网络流量监控技术可以对网络流量数据和用户的特征进行更好的分析,可以对用户的上网行为进行深入的挖掘,更好的为用户推荐喜欢的网络内容.本文结合互联网的发展现状对海量网络流量数据分析技术进行了分析,在基于云计算的基础上提出了海量网络流量数据分析的几个关键性技术,对基于云计算的海量网络流量数据分析技术进行了分析和研究.     

基于Hadoop的云计算平台安全机制研究

Hadoop作为一种开源的基础云计算框架,在企业界逐步得到了有效的应用,但其安全机制的薄弱已成为阻碍其发展的主要问题之一。文中首先归纳了传统网络的安全问题以及安全机制,分析了云计算的安全问题,然后通过对Hadoop工作模式的描述,提出了Hadoop的安全需求,最后通过对Hadoop当前安全机制的研究,分析了其中的安全隐患,并提出了相应的解决办法,为Hadoop新安全需求的企业应用提供了技术支持。     

LTE-Advanced网络Uu接口用户行为分析系统的研究与实现

大数据时代,面对海量且复杂的结构化、半结构化和非结构化数据,传统的信令监测分析系统无法快速准确地分类、处理以及存储海量数据中包含的信息.针对这些问题,提出了一种基于Hadoop系统技术平台和支持向量机(Support Vector Machine,SVM)分类算法的LTE-Advanced网络Uu接口用户行为分析系统.对用户行为分析系统的系统架构、在Hadoop平台下数据挖掘分类算法SVM的实现进行了详细阐述,并通过Uu接口进行了现网测试,测试结果表明,提出的用户行为分析系统达到了预期的效果,对用户偏好分析以及精准营销具有推广意义.     

Hadoop云平台下的并行化图像处理实现

近年来,云计算在IT行业掀起了新一轮技术革新浪潮。云计算是一种新兴的计算模型,它是并行计算、分布式计算、网格计算的综合发展,以简单、透明服务的形式提供无限制的计算资源。Hadoop实现的开源云平台提供了并行计算模型MapReduce、分布式文件系统HDFS和分布式数据库HBase等。随着数字图像数据量不断增长,单机模式的图像处理已逐渐不能满足用户需求。文中提出了利用Hadoop云平台实现海量图像的并行     

Quality of Experience from user and network perspectives

The impact of network performance on user experience is important to know, as it determines the success or failure of a service. Unfortunately, it is very difficult to assess it in real-time on an operational network. Monitoring of network-level performance criteria is easier and more usual. But the problem is then to correlate these network-level Quality of Service (QoS) to the Quality of Experience (QoE) perceived by the users. Efforts have been done in the previous years to map user behaviour to traffic characteristics on the network to QoS. However, being able to successfully relate these traffic characteristics to user satisfaction is not a simple task and still requires further investigations. In this work, we try to associate on one side the correlations between various traffic characteristics measured on an operational network and on the other side the user experience tested on an experimental platform. Our aim is to observe some pronounced trends regarding relationships between both types of results. More precisely, we want to validate how and to what extent the volumes of user sessions represent the level of user satisfaction. Along this way, we need to revise classical relationships between some of the network performance indicators such as loss, download time and throughput in order to strengthen the understanding of this impact on each other and on user satisfaction. This preliminary study is based on the application web.     

采用云计算技术的网络流量检测

为了实现对现网大数据网络流量的实时、有效检测,提出了一种基于云计算的网络流量检测方案。该方案充分利用Hadoop平台Map/Reduce编程模型在海量数据处理方面的优势,采用分层化的设计思想,克服了传统检测方案在海量数据应用环境中效率低下、可扩展性与安全性不足的缺点。重庆移动DPI平台应用表明,该方案较为有效,流量检测效果良好,在大数据处理时效率较普通分布式处理有明显提高。     

基于OpenFlow流表的云网络异常行为主动监测和处理

云数据中心的网络异常行为不仅对网络设备造成严重业务负荷,同时也显著影响云用户使用体验。云计算环境中的共享资源模式和云用户迥然不同的业务形态,使得网络分析和异常行为定位变得更加困难。本文针对云数据中心的网络异常行为进行特征提取和分析,并基于sdn云数据中心的网络架构和原理进行深度剖析,总结出基于openflow流表的网络异常行为判定方法。同时采用自动化运维手段,制定了一套网络异常行为自动化检测和封堵的智能系统,实现对网络异常行为的快速处理。     

Hadoop‐based analytic framework for cyber forensics

With an exponential increase in the data size and complexity of various documents to be investigated, existing methods of network forensics are found not much efficient with respect to accuracy and detection ratio. The existing techniques for network forensic analysis exhibit inherent limitations while processing a huge volume, variety, and velocity of data. It makes network forensic a time‐consuming and resource‐consuming task. To balance time taken and output delivered, these existing techniques put a limit on the amount of data under analysis, which results in a polynomial time complexity of these solutions. So to mitigate these issues, in this paper, we propose an effective framework to overcome the limitation to handle large volume, variety, and velocity of data. An architectural setup that consists of MapReduce framework on top of Hadoop Distributed File System environment is proposed in this paper. The proposed framework demonstrates its capability to handle issues of storage and processing of big data using cloud computing. Also, in the proposed framework, supervised machine learning (random forest‐based decision tree) algorithm has been implemented to demonstrate better sensitivity. To train and validate the model, online available data set from CAIDA is taken and university network traffic samples, with increasing size, has been taken for experiment. Results thus obtained confirm the superiority of the proposed framework in network forensics, with an average accuracy of 99.34% (malicious and nonmalicious traffic).     

基于Hadoop的海量电信数据云计算平台研究

传统的数据分析方法面对海量电信数据存在管理和分析难的问题.Hadoop是一个可实现大规模分布式计算的开源框架,具有高效、可靠、可伸缩的优点,被广泛应用于云计算领域.本文在对云计算和Hadoop进行分析和研究的基础上,提出了一种针对海量电信数据的分布式云计算方法,建立了基于Hadoop的海量电信数据云计算平台.实验证明,该平台能够有效完成海量数据的管理和分析任务,提高海量数据分析的速度和效率.     

Hodoop集群计算在电信业BI系统中的应用

传统的数据分析方法和相关BI数据分析软件面对海量电信数据已显得力不从心。Hadoop是一个可实现大规模分布式计算的开源框架,具有高效、可靠、可扩展的优点,被广泛应用于云计算领域。文中在对云计算和Hadoop进行分析和研究的基础上。介绍了Hadoop技术及业界的一些典型应用案例,提出了Hadoop未来可以应用在电信业BI系统中的一些场景,并进行了探讨和研究。     

Classifier Learning and Decision Making for a Connection Manager on a Heterogeneous Network

An attractive feature of the Connection Manager Intelligence Agent is its use of network traffic and multi-attribute behavior to locate the best network devices. This study has integrated this agent with a user interface; a network connection handoff; wired and wireless network device drivers; network management applications of the (plug-in) play interface; module-to-module communication authentication; and a DBus for added versatility. To reduce the time that developers of embedded systems spend on the software engineering of this module and to achieve rapid operational efficiency, an Open Source platform, such as MeeGo or Android, must be used. This study has implemented an interactive interface through the function (based on Fuzzy-AHP) of acquisition user behavior and machine designers, boosting iterations for User-Case. The algorithm maintains a set of weights as a distribution class table of cases, as in the parameter learning by user-case; it is quite possible that the expectation–maximization of maximum probability model can be classified by user behavior. In this study, user interaction showed that the agent satisfactorily matched user intent.     

Proactive content caching by exploiting transfer learning for mobile edge computing

To address the vast multimedia traffic volume and requirements of user quality of experience in the next‐generation mobile communication system (5G), it is imperative to develop efficient content caching strategy at mobile network edges, which is deemed as a key technique for 5G. Recent advances in edge/cloud computing and machine learning facilitate efficient content caching for 5G, where mobile edge computing can be exploited to reduce service latency by equipping computation and storage capacity at the edge network. In this paper, we propose a proactive caching mechanism named learning‐based cooperative caching (LECC) strategy based on mobile edge computing architecture to reduce transmission cost while improving user quality of experience for future mobile networks. In LECC, we exploit a transfer learning‐based approach for estimating content popularity and then formulate the proactive caching optimization model. As the optimization problem is NP‐hard, we resort to a greedy algorithm for solving the cache content placement problem. Performance evaluation reveals that LECC can apparently improve content cache hit rate and decrease content delivery latency and transmission cost in comparison with known existing caching strategies.     

基于云计算的海量网络流量数据分析处理及关键算法研究

为了应对日益增长的网络流量数据量和对网络安全的需求,提高网络流量数据的处理效率和准确性,文中从云计算架构出发,设计并搭建了一个能承载大规模网络流量数据处理的云计算平台。基于该平台,采用了分布式存储、并行计算和机器学习等技术,对海量网络流量数据的预处理、聚类分析、异常检测等关键环节进行了研究。结果表明,基于云计算的海量网络流量数据分析处理的关键算法取得了显著成果。通过分布式存储和并行计算技术,实现了对海量网络流量数据的高效读写和处理。在预处理阶段,针对流量数据进行采样和滤波,减少了数据量,并保留了关键特征。在聚类分析方面,利用机器学习算法实现了对网络流量的分类和统计,通过构建模型、训练和优化算法,实现了对网络攻击和异常行为的准确识别和及时报警。     

Impact of Interference on Multi-Hop Wireless Network Performance

In this paper, we address the following question: given a specific placement of wireless nodes in physical space and a specific traffic workload, what is the maximum throughput that can be supported by the resulting network? Unlike previous work that has focused on computing asymptotic performance bounds under assumptions of homogeneity or randomness in the network topology and/or workload, we work with any given network and workload specified as inputs.A key issue impacting performance is wireless interference between neighboring nodes. We model such interference using a conflict graph, and present methods for computing upper and lower bounds on the optimal throughput for the given network and workload. To compute these bounds, we assume that packet transmissions at the individual nodes can be finely controlled and carefully scheduled by an omniscient and omnipotent central entity, which is unrealistic. Nevertheless, using ns-2 simulations, we show that the routes derived from our analysis often yield noticeably better throughput than the default shortest path routes even in the presence of uncoordinated packet transmissions and MAC contention. This suggests that there is opportunity for achieving throughput gains by employing an interference-aware routing protocol.     

Performance analysis of path rerouting algorithms for handoffcontrol in mobile ATM networks

This paper studies the effects of user mobility and handoff path rerouting on the traffic distributions in a mobile network environment. In mobile ATM networks, extra traffic load may be added to network links due to user mobility and handoff path rerouting. This requires higher network link capacity and possible topology reengineering in order to support the same quality of service (QoS) for mobile services. To capture the dynamic variations in mobile ATM networks, we propose to use a flow model. The model represents the mobile-generated traffic as a set of stochastic flows over a set of origin-destination (OD) pairs. The user mobility is defined by transfer probabilities of the flows and the handoff path rerouting algorithm is modeled by a transformation between the routing functions for traffic flows. The analysis shows that user mobility may cause temporal variations as well as smoothing effects on the network traffic. Using the flow network model, typical handoff path rerouting algorithms are evaluated through both analytical and experimental approaches. The evaluation methodology can be used for either redesigning the network topology for a given path rerouting algorithm or selecting a path rerouting algorithm for a given network topology under a specific mobile service scenario     

大数据技术在网络安全分析中的应用研究

针对误用检测方法和异常检测方法缺少对网络风险因素分析,导致网络漏洞检出率较低的问题,提出了基于大数据H adoop技术检测网络安全漏洞研究.构建标准化直接关系矩阵、生成总关系矩阵确定网络风险元素属性,由此分析网络风险.构建Hadoop框架,采集入侵行为漏洞信息,搭建核心装置,处理网络漏洞.避免外力干扰情况下,评估网络漏...     

Providing sustainable QoS in next-generation networks

In this article we discuss QoS from a somewhat unconventional point of view. First of all, we argue that high availability is a key ingredient in the QoS perceived by the user. High availability with extremely short interruptions in case of failure is needed for acceptable QoS in real-time dialog services such as telephony or video conferencing. Second, an even distribution of the traffic load over the network is essential to ensure efficient network utilization given that some kind of admission control for QoS traffic has to be in place for overload avoidance. Unfortunately, both these features are not adequately provided in current IP networks. We therefore introduce a new routing scheme offering a massive improvement in availability as well as efficient traffic distribution, while causing only minor computational effort. Our proposals are then evaluated for a high-speed core network. We claim that our proposed approach would be a useful basis for QoS services in an NGN.     

An online cost-efficient protection scheme for quick recovery in all-optical WDM mesh networks

Since a single fiber carries a huge amount of data in optical WDM networks, a fiber cut even for a brief period is a very serious event. Designing schemes to prevent disruption of user traffic and recovery techniques from failures is thus an important area of research. Since a single fiber cut is the most common type of fault, in this paper we address the problem of protecting all-optical WDM mesh networks from single link failure. Our proposed online protection scheme is an improvement over an existing approach and is not only cost-efficient in terms of network resource consumption but can also provide quick recovery from a link failure. We first provide an ILP formulation for the problem and then propose a heuristic solution iStreams that can provide near-optimal solution in polynomial time. Performance comparisons with some well-known schemes of protection show that our heuristic algorithm can be a better choice for conserving resource while providing quick recovery from a link failure.     

Self-similarity in World Wide Web traffic: evidence and possiblecauses

The notion of self-similarity has been shown to apply to wide-area and local-area network traffic. We show evidence that the subset of network traffic that is due to World Wide Web (WWW) transfers can show characteristics that are consistent with self-similarity, and we present a hypothesized explanation for that self-similarity. Using a set of traces of actual user executions of NCSA Mosaic, we examine the dependence structure of WWW traffic. First, we show evidence that WWW traffic exhibits behavior that is consistent with self-similar traffic models. Then we show that the self-similarity in such traffic can be explained based on the underlying distributions of WWW document sizes, the effects of caching and user preference in file transfer, the effect of user “think time”, and the superimposition of many such transfers in a local-area network. To do this, we rely on empirically measured distributions both from client traces and from data independently collected at WWW servers