Lightweight and Informative Traffic Metrics for Data Center Monitoring

In recent years, thousands of commodity servers have been deployed in Internet data centers to run large scale Internet applications or cloud computing services. Given the sheer volume of data communications between servers and millions of end users, it becomes a daunting task to continuously monitor the availability, performance and security of data centers in real-time operational environments. In this paper, we propose and evaluate a lightweight and informative traffic metric, streaming frequency, for network monitoring in Internet data centers. The power-series based metric that is extracted from the aggregated IP traffic streams, not only carries temporal characteristics of data center servers, but also helps uncover traffic patterns of these servers. We show the convergence and reconstructability properties of this metric through theoretical proof and algorithm analysis. Using real data-sets collected from multiple data centers of a large Internet content provider, we demonstrate its applications in detecting unwanted traffic towards data center servers. To the best of our knowledge, this paper is the first to introduce a streaming metric with a unique reconstruction capability that could aid data center operators in network management and security monitoring.     

Reducing the network overhead of user mobility–induced virtual machine migration in mobile edge computing

With the popularity of mobile devices (such as smartphones and tablets) and the development of the Internet of Things, mobile edge computing is envisioned as a promising approach to improving the computation capabilities and energy efficiencies of mobile devices. It deploys cloud data centers at the edge of the network to lower service latency. To satisfy the high latency requirement of mobile applications, virtual machines (VMs) have to be correspondingly migrated between edge cloud data centers because of user mobility. In this paper, we try to minimize the network overhead resulting from constantly migrating a VM to cater for the movement of its user. First, we elaborate on two simple migration algorithms (M-All and M-Edge), and then, two optimized algorithms are designed by classifying user mobilities into two categories (certain and uncertain moving trajectories). Specifically, a weight-based algorithm (M-Weight) and a mobility prediction–based heuristic algorithm (M-Predict) are proposed for the two types of user mobilities, respectively. Numerical results demonstrate that the two optimized algorithms can significantly lower the network overhead of user mobility–induced VM migration in mobile edge computing environments.     

Implementation and performance evaluation of a scheduling algorithm for divisible load parallel applications in a cloud computing environment

Cloud computing is an emerging technology in which information technology resources are virtualized to users in a set of computing resources on a pay‐per‐use basis. It is seen as an effective infrastructure for high performance applications. Divisible load applications occur in many scientific and engineering applications. However, dividing an application and deploying it in a cloud computing environment face challenges to obtain an optimal performance due to the overheads introduced by the cloud virtualization and the supporting cloud middleware. Therefore, we provide results of series of extensive experiments in scheduling divisible load application in a Cloud environment to decrease the overall application execution time considering the cloud networking and computing capacities presented to the application's user. We experiment with real applications within the Amazon cloud computing environment. Our extensive experiments analyze the reasons of the discrepancies between a theoretical model and the reality and propose adequate solutions. These discrepancies are due to three factors: the network behavior, the application behavior and the cloud computing virtualization. Our results show that applying the algorithm result in a maximum ratio of 1.41 of the measured normalized makespan versus the ideal makespan for application in which the communication to computation ratio is big. They show that the algorithm is effective for those applications in a heterogeneous setting reaching a ratio of 1.28 for large data sets. For application following the ensemble clustering model in which the computation to communication ratio is big and variable, we obtained a maximum ratio of 4.7 for large data set and a ratio of 2.11 for small data set. Applying the algorithm also results in an important speedup. These results are revealing for the type of applications we consider under experiments. The experiments also reveal the impact of the choice of the platforms provided by Amazon on the performance of the applications under study. Considering the emergence of cloud computing for high performance applications, the results in this paper can be widely adopted by cloud computing developers. Copyright © 2014 John Wiley & Sons, Ltd.     

Load balancing aware scheduling algorithms for fog networks

Fog networks have attracted the attention of researchers recently. The idea is that a part of the computation of a job/application can be performed by fog devices that are located at the network edge, close to the users. Executing latency sensitive applications on the cloud may not be feasible, owing to the significant communication delay involved between the user and the cloud data center (cdc). By the time the application traverses the network and reaches the cloud data center, it might already be too late. However, fog devices, also known as mobile data centers (mdcs), are capable of executing such latency sensitive applications. In this paper, we study the problem of balancing the application load while taking account of security constraints of jobs, across various mdcs in a fog network. In case a particular mdc does not have sufficient capacity to execute a job, the job needs to be migrated to some other mdc. To this end, we propose three heuristic algorithms: minimum distance, minimum load, and minimum hop distance and load (MHDL). In addition, we also propose an ILP-based algorithm called load balancing aware scheduling ILP (LASILP) for solving the task mapping and scheduling problem. The performance of the proposed algorithms have been compared with the cloud only algorithm and another heuristic algorithm called fog-cloud-placement (FCP). Simulation results performed on real-life workload traces reveal that the MHDL heuristic performs better as compared to other scheduling policies in the fog computing environment while meeting application privacy requirements.     

Value of information based scheduling of cloud computing resources

Traditionally, heavy computational tasks were performed on a dedicated infrastructure requiring a heavy initial investment, such as a supercomputer or a data center. Grid computing relaxed the assumptions of the fixed infrastructure, allowing the sharing of remote computational resources. Cloud computing brought these ideas into the commercial realm and allows users to request on demand an essentially unlimited amount of computing power. However, in contrast to previous assumptions, this computing power is metered and billed on an hour-by-hour basis.In this paper, we are considering applications where the output quality increases with the deployed computational power, a large class including applications ranging from weather prediction to financial modeling. We are proposing a computation scheduling that considers both the financial cost of the computation and the predicted financial benefit of the output, that is, its value of information (VoI). We model the proposed approach for an example of analyzing real-estate investment opportunities in a competitive environment. We show that by using the VoI-based scheduling algorithm, we can outperform minimalistic computing approaches, large but fixedly allocated data centers and cloud computing approaches that do not consider the VoI.     

面向边缘计算应用的宽度孪生网络

边缘计算是将计算、存储、通信等任务分配到网络边缘的计算模式. 它强调在用户终端附近执行数据处理过程, 以达到降低延迟, 减少能耗, 保护用户隐私等目的. 然而网络边缘的计算、存储、能源资源有限, 这给边缘计算应用的推广带来了新的挑战. 随着边缘智能的兴起, 人们更希望将边缘计算应用与人工智能技术结合起来, 为我们的生活带来更多的便利. 许多人工智能方法, 如传统的深度学习方法, 需要消耗大量的计算、存储资源, 并且伴随着巨大的时间开销. 这不利于强调低延迟的边缘计算应用的推广. 为了解决这个问题, 我们提出将宽度学习系统(Broad learning system, BLS)等浅层网络方法应用到边缘计算应用领域, 并且设计了一种宽度孪生网络算法. 我们将宽度学习系统与孪生网络结合起来用于解决分类问题. 实验结果表明我们的方法能够在取得与传统深度学习方法相似精度的情况下降低时间和资源开销, 从而更好地提高边缘计算应用的性能.     

Sliding window top-k dominating query processing over distributed data streams

Preference query processing is important for a wide range of applications involving distributed databases, such as network monitoring, web-based systems, and market analysis. In such applications, data objects are generated frequently and massively, which presents an important and challenging problem of continuous query processing over distributed data stream environments. A top-k dominating query, which has been receiving much research attention recently, returns the k data objects that dominate the highest number of data objects in a given dataset, and due to its dominance-based ranking function, we can easily obtain superior data objects. An emerging requirement in distributed stream environments is an efficient technique for continuously monitoring top-k dominating data objects. Despite of this fact, no study has addressed this problem. In this paper, therefore, we address the problem of continuous top-k dominating query processing over distributed data stream environments. We present two algorithms that monitor the exact top-k dominating data and efficiently eliminate unqualified data objects for the result, which reduces both communication and computation costs. In addition to these algorithms, we present an approximate algorithm that further reduces both communication and computation costs. Extensive experiments on both synthetic and real data have demonstrated the efficiency and scalability of our algorithms.     

Mining@home: toward a public‐resource computing framework for distributed data mining

Several classes of scientific and commercial applications require the execution of a large number of independent tasks. One highly successful and low‐cost mechanism for acquiring the necessary computing power for these applications is the ‘public‐resource computing’, or ‘desktop Grid’ paradigm, which exploits the computational power of private computers. So far, this paradigm has not been applied to data mining applications for two main reasons. First, it is not straightforward to decompose a data mining algorithm into truly independent sub‐tasks. Second, the large volume of the involved data makes it difficult to handle the communication costs of a parallel paradigm. This paper introduces a general framework for distributed data mining applications called Mining@home. In particular, we focus on one of the main data mining problems: the extraction of closed frequent itemsets from transactional databases. We show that it is possible to decompose this problem into independent tasks, which however need to share a large volume of the data. We thus introduce a data‐intensive computing network, which adopts a P2P topology based on super peers with caching capabilities, aiming to support the dissemination of large amounts of information. Finally, we evaluate the execution of a pattern extraction task on such network. Copyright © 2009 John Wiley & Sons, Ltd.     

Morphic computing

In this paper, we introduce a new type of computation called “morphic computing”. Morphic computing is based on field theory and more specifically morphic fields. Morphic fields were first introduced by Rupert Sheldrake [R. Sheldrake, A New Science of Life: The Hypothesis of Morphic Resonance (second edition, 1985), Park Street Press; Reprint edition (March 1, 1995), 1981] from his hypothesis of formative causation that made use of the older notion of morphogenetic fields. Rupert Sheldrake [R. Sheldrake, A New Science of Life: The Hypothesis of Morphic Resonance (second edition, 1985), Park Street Press; Reprint edition (March 1, 1995), 1981] developed his famous theory, morphic resonance, on the basis of the work by French philosopher Henri Bergson. Morphic fields and its subset morphogenetic fields have been at the center of controversy for many years in mainstream science and the hypothesis is not accepted by some scientists who consider it a pseudoscience. We claim that morphic computing is a natural extension of holographic computation, quantum computation, soft computing, and DNA computing. All natural computations bonded by the turing machine can be formalised and extended by our new type of computation model—morphic computing. In this paper, we introduce the basis for this new computing paradigm and its extensions such as quantum logic and entanglement in morphic computing, morphic systems and morphic system of systems (M-SOS). Its applications to the field of computation by words as an example of the morphic computing, morphogenetic fields in neural network and morphic computing, morphic fields - concepts and web search, and agents and fuzzy in morphic computing will also be discussed.     

基于小波概要的并行数据流聚类

许多应用中会连续不断产生大量随时间演变的序列型数据,构成时间序列数据流,如传感器网络、实时股票行情、网络及通信监控等场合.聚类是分析这类并行多数据流的一种有力工具.但数据流长度无限、随时间演变和大数据量的特点,使得传统的聚类方法无法直接应用.利用数据流的遗忘特性,应用离散小波变换,分层、动态地维护每个数据流的概要结构.基于该概要结构,快速计算数据流与聚类中心之间的近似距离,实现了一种适合并行多数据流的K-means聚类方法.所进行的实验验证了该聚类方法的有效性.     

Fog computing for next-generation Internet of Things: Fundamental,state-of-the-art and research challenges

In recent times, the Internet of Things (IoT) applications, including smart transportation, smart healthcare, smart grid, smart city, etc. generate a large volume of real-time data for decision making. In the past decades, real-time sensory data have been offloaded to centralized cloud servers for data analysis through a reliable communication channel. However, due to the long communication distance between end-users and centralized cloud servers, the chances of increasing network congestion, data loss, latency, and energy consumption are getting significantly higher. To address the challenges mentioned above, fog computing emerges in a distributed environment that extends the computation and storage facilities at the edge of the network. Compared to centralized cloud infrastructure, a distributed fog framework can support delay-sensitive IoT applications with minimum latency and energy consumption while analyzing the data using a set of resource-constraint fog/edge devices. Thus our survey covers the layered IoT architecture, evaluation metrics, and applications aspects of fog computing and its progress in the last four years. Furthermore, the layered architecture of the standard fog framework and different state-of-the-art techniques for utilizing computing resources of fog networks have been covered in this study. Moreover, we included an IoT use case scenario to demonstrate the fog data offloading and resource provisioning example in heterogeneous vehicular fog networks. Finally, we examine various challenges and potential solutions to establish interoperable communication and computation for next-generation IoT applications in fog networks.     

Threshold-based probabilistic top-k dominating queries

Recently, due to intrinsic characteristics in many underlying data sets, a number of probabilistic queries on uncertain data have been investigated. Top-k dominating queries are very important in many applications including decision making in a multidimensional space. In this paper, we study the problem of efficiently computing top-k dominating queries on uncertain data. We first formally define the problem. Then, we develop an efficient, threshold-based algorithm to compute the exact solution. To overcome some inherent computational deficiency in an exact computation, we develop an efficient randomized algorithm with an accuracy guarantee. Our extensive experiments demonstrate that both algorithms are quite efficient, while the randomized algorithm is quite scalable against data set sizes, object areas, k values, etc. The randomized algorithm is also highly accurate in practice.     

Scalable and topology-aware reconciliation on P2P networks

Collaborative applications are characterized by high levels of data sharing. Optimistic replication has been suggested as a mechanism to enable highly concurrent access to the shared data, whilst providing full application-defined consistency guarantees. Nowadays, there are a growing number of emerging cooperative applications adequate for Peer-to-Peer (P2P) networks. However, to enable the deployment of such applications in P2P networks, it is required a mechanism to deal with their high data sharing in dynamic, scalable and available way. Previous work on optimistic replication has mainly concentrated on centralized systems. Centralized approaches are inappropriate for a P2P setting due to their limited availability and vulnerability to failures and partitions from the network. In this paper, we focus on the design of a reconciliation algorithm designed to be deployed in large scale cooperative applications, such as P2P Wiki. The main contribution of this paper is a distributed reconciliation algorithm designed for P2P networks (P2P-reconciler). Other important contributions are: a basic cost model for computing communication costs in a DHT overlay network; a strategy for computing the cost of each reconciliation step taking into account the cost model; and an algorithm that dynamically selects the best nodes for each reconciliation step. Furthermore, since P2P networks are built independently of the underlying topology, which may cause high latencies and large overheads degrading performance, we also propose a topology-aware variant of our P2P-reconciler algorithm and show the important gains on using it. Our P2P-reconciler solution enables high levels of concurrency thanks to semantic reconciliation and yields high availability, excellent scalability, with acceptable performance and limited overhead.     

Exploiting the untapped potential of mobile distributed computing via approximation

Mobile computing is one of the largest untapped reservoirs in today’s pervasive computing world as it has the potential to enable a variety of in-situ, real-time applications. Yet, this computing paradigm suffers when the available resources–such as energy in the network, CPU cycles, memory, I/O data rate–are limited. In this article, the new paradigm of approximate computing is proposed to harness such potential and to enable real-time computation-intensive mobile applications in resource-limited and uncertain environments. A reduction in time and energy consumed by an application is obtained via approximate computing by decreasing the amount of computation needed; such improvement, however, comes with the potential loss in accuracy. Hence, a Mobile Distributed Computing framework, is introduced to determine offline the ‘approximable’ tasks in an application and a light-weight online algorithm is devised to select the approximate version of the tasks in an application during run time. The effectiveness of the proposed approach is validated through extensive simulation and testbed experiments by comparing approximate versus exact-computation performance.     

SDN/NFV security framework for fog-to-things computing infrastructure

Currently, core networking architectures are facing disruptive developments, due to emergence of paradigms such as Software-Defined-Networking (SDN) for control, Network Function Virtualization (NFV) for services, and so on. These are the key enabling technologies for future applications in 5G and locality-based Internet of things (IoT)/wireless sensor network services. The proliferation of IoT devices at the Edge networks is driving the growth of all-connected world of Internet traffic. In the Cloud-to-Things continuum, processing of information and data at the Edge mandates development of security best practices to arise within a fog computing environment. Service providers are transforming their business using NFV-based services and SDN-enabled networks. The SDN paradigm offers an easily programmable model, global view, and control for modern networks, which demand faster response to security incidents and dynamically enforce countermeasures to intrusions and cyberattacks. This article proposes an autonomic multilayer security framework called Distributed Threat Analytics and Response System (DTARS) for a converged architecture of Fog/Edge computing and SDN infrastructures, for emerging applications in IoT and 5G networks. The major detection scheme is deployed within the data plane, consisting of a coarse-grained behavioral, anti-spoofing, flow monitoring and fine-grained traffic multi-feature entropy-based algorithms. We developed exemplary defense applications under DTARS framework, on a malware testbed imitating the real-life DDoS/botnets such as Mirai. The experiments and analysis show that DTARS is capable of detecting attacks in real-time with accuracy more than 95% under attack intensities up to 50 000 packets/s. The benign traffic forwarding rate remains unaffected with DTARS, while it drops down to 65% with traditional NIDS for advanced DDoS attacks. Further, DTARS achieves this performance without incurring additional latency due to data plane overhead.     

Parallel multi-dimensional range query processing with R-trees on GPU

The general purpose computing on graphics processing unit (GP-GPU) has emerged as a new cost effective parallel computing paradigm in high performance computing research that enables large amount of data to be processed in parallel. Large scale scientific data intensive applications have been playing an important role in modern high performance computing research. A common access pattern into such scientific data analysis applications is multi-dimensional range query, but not much research has been conducted on multi-dimensional range query on the GPU. Inherently multi-dimensional indexing trees such as R-Trees are not well suited for GPU environment because of its irregular tree traversal. Traversing irregular tree search path makes it hard to maximize the utilization of massively parallel architectures. In this paper, we propose a novel MPTS (Massively Parallel Three-phase Scanning) R-tree traversal algorithm for multi-dimensional range query, that converts recursive access to tree nodes into sequential access. Our extensive experimental study shows that MPTS R-tree traversal algorithm on NVIDIA Tesla M2090 GPU consistently outperforms traditional recursive R-trees search algorithm on Intel Xeon E5506 processors.     

Mobiiscape: Middleware support for scalable mobility pattern monitoring of moving objects in a large-scale city

With the explosive proliferation of mobile devices such as smartphones, tablets, and sensor nodes, location-based services are getting even more attention than before, considered as one of the killer applications in the upcoming mobile computing era. Developing location-based services necessarily requires an effective and scalable location data processing technology. In this paper, we present Mobiiscape, a novel location monitoring system that collectively monitors mobility patterns of a large number of moving objects in a large-scale city to support city-wide mobility-aware applications. Mobiiscape provides an SQL-like query language named Moving Object Monitoring Query Language (MQL) that allows applications to intuitively specify Mobility Pattern Monitoring Queries (MPQs). Further, Mobiiscape provides a set of scalable location monitoring techniques to efficiently process a large number of MPQs over a large number of location streams. The scalable processing techniques include a (1) Place Border Index, a spatial index for quickly searching for relevant queries upon receiving location streams, (2) Place-Based Window, a spatial-purpose window for efficiently detecting primitive mobility patterns, (3) Shared NFA, a shared query processing technique for efficiently matching complex mobility patterns, and (4) Attribute Pre-matching Bitmap, an in-memory data structure for efficiently filtering out moving objects based on their attributes. We have implemented a Mobiiscape prototype system. Then, we show the usefulness of the system by implementing promising location-based applications based on it such as a ubiquitous taxicab service and a location-based advertising. Also, we demonstrate the performance benefit of the system through extensive evaluation and comparison.     

A fuzzy data reduction cluster method based on boundary information for large datasets

The fuzzy c-means algorithm (FCM) is aimed at computing the membership degree of each data point to its corresponding cluster center. This computation needs to calculate the distance matrix between the cluster center and the data point. The main bottleneck of the FCM algorithm is the computing of the membership matrix for all data points. This work presents a new clustering method, the bdrFCM (boundary data reduction fuzzy c-means). Our algorithm is based on the original FCM proposal, adapted to detect and remove the boundary regions of clusters. Our implementation efforts are directed in two aspects: processing large datasets in less time and reducing the data volume, maintaining the quality of the clusters. A significant volume of real data application (> 10⁶ records) was used, and we identified that bdrFCM implementation has good scalability to handle datasets with millions of data points.

     

Application of density-based outlier detection to database activity monitoring

To prevent internal data leakage, database activity monitoring uses software agents to analyze protocol traffic over networks and to observe local database activities. However, the large size of data obtained from database activity monitoring has presented a significant barrier to effective monitoring and analysis of database activities. In this paper, we present database activity monitoring by means of a density-based outlier detection method and a commercial database activity monitoring solution. In order to provide efficient computing of outlier detection, we exploited a kd-tree index and an Approximated k-nearest neighbors (ANN) search method. By these means, the outlier computation time could be significantly reduced. The proposed methodology was successfully applied to a very large log dataset collected from the Korea Atomic Energy Research Institute (KAERI). The results showed that the proposed method can effectively detect outliers of database activities in a shorter computation time.     

Methods for mining frequent items in data streams: an overview

In many real-world applications, information such as web click data, stock ticker data, sensor network data, phone call records, and traffic monitoring data appear in the form of data streams. Online monitoring of data streams has emerged as an important research undertaking. Estimating the frequency of the items on these streams is an important aggregation and summary technique for both stream mining and data management systems with a broad range of applications. This paper reviews the state-of-the-art progress on methods of identifying frequent items from data streams. It describes different kinds of models for frequent items mining task. For general models such as cash register and Turnstile, we classify existing algorithms into sampling-based, counting-based, and hashing-based categories. The processing techniques and data synopsis structure of each algorithm are described and compared by evaluation measures. Accordingly, as an extension of the general data stream model, four more specific models including time-sensitive model, distributed model, hierarchical and multi-dimensional model, and skewed data model are introduced. The characteristics and limitations of the algorithms of each model are presented, and open issues waiting for study and improvement are discussed.