GAMESH: A grid architecture for scalable monitoring and enhanced dependable job scheduling

Grid computing is a largely adopted paradigm to federate geographically distributed data centers. Due to their size and complexity, grid systems are often affected by failures that may hinder the correct and timely execution of jobs, thus causing a non-negligible waste of computing resources. Despite the relevance of the problem, state-of-the-art management solutions for grid systems usually neglect the identification and handling of failures at runtime. Among the primary goals to be considered, we claim the need for novel approaches capable to achieve the objectives of scalable integration with efficient monitoring solutions and of fitting large and geographically distributed systems, where dynamic and configurable tradeoffs between overhead and targeted granularity are necessary. This paper proposes GAMESH, a Grid Architecture for scalable Monitoring and Enhanced dependable job ScHeduling. GAMESH is conceived as a completely distributed and highly efficient management infrastructure, concentrating on two crucial aspects for large-scale and multi-domain grid environments: (i) the scalable dissemination of monitoring data and (ii) the troubleshooting of job execution failures. GAMESH has been implemented and tested in a real deployment encompassing geographically distributed data centers across Europe. Experimental results show that GAMESH (i) enables the collection of measurements of both computing resources and conditions of task scheduling at geographically sparse sites, while imposing a limited overhead on the entire infrastructure, and (ii) provides a failure-aware scheduler able to improve the overall system performance, even in the presence of failures, by coordinating local job schedulers at multiple domains.     

Autonomous decentralized community communication for information dissemination

To meet the increasing demand for real-time content delivery, the proposed autonomous decentralized community communication system offers an efficient information dissemination infrastructure with a decentralized architecture. ADCC's aim is to help end-user communities communicate and exchange information efficiently; to meet this goal, the system uses an application-level multicast technique that arbitrarily scales to large groups. The ADCC system also features a scalable community construction and maintenance scheme that eases the burden of organizing an online community network.     

A scalable interest management scheme for distributed virtual environments

With the expansion of the internet and its bandwidth, distributed virtual environment (DVE) applications have become more prevalent. In DVE applications, users frequently crowd in a specific place, and a key aspect to consider is how to provide interactive performance for users. However, existing approaches using multicast require users to receive uninteresting messages. Even though recent works have addressed fine‐grained filtering, they still incur other drawbacks in terms of assigning lots of multicast addresses or handling overhead of multicast groups. This makes the system less scalable as the number of users increases. In this paper, we propose a new scalable filtering scheme that reduces not only the number of messages during interaction in a region and among neighboring regions, but also the number of multicast addresses without significant computational overhead. Interest management in a region dynamically creates groups of users with the same interests. While members communicate with each other with high fidelity, a representative sends information to non‐members with low frequency. For interaction among neighboring regions, we propose a sub‐region concept to select only a subset of users from the neighboring regions based on proximity, the distribution of the users' locations, and the viewing direction of a user. Copyright © 2007 John Wiley & Sons, Ltd.     

一种基于调度的VOD系统的研究与实现

传统的视频点播(Video-On-Demand,VOD)系统中,服务器为每个用户请求分配一个独立的信道。这样过多的用户经常会造成系统的服务器I/O带宽或网络带宽的瓶颈。对服务器信道调度方案的设计是缓解这一瓶颈问题的一种有效方法。介绍了一种采用了可控多播(Controlled Multieast,CM)信道调度方案的系统的实现。测试结果表明,该系统可以节约带宽,增大视频点播服务的用户数目,较好地缓解传统VOD系统中的服务器I/O或网络带宽瓶颈问题。     

A technique for extracting behavioral sequence patterns from GPS recorded data

The mobile wireless market has been attracting many customers. Technically, the paradigm of anytime-anywhere connectivity raises previously unthinkable challenges, including the management of million of mobile customers, their profiles, the profiles-based selective information dissemination, and server-side computing infrastructure design issues to support such a large pool of users automatically and intelligently. In this paper, we propose a data mining technique for discovering frequent behavioral patterns from a collection of trajectories gathered by Global Positioning System. Although the search space for spatiotemporal knowledge is extremely challenging, imposing spatial and temporal constraints on spatiotemporal sequences makes the computation feasible. Specifically, the mined patterns are incorporated with synthetic constraints, namely spatiotemporal sequence length restriction, minimum and maximum timing gap between events, time window of occurrence of the whole pattern, inclusion or exclusion event constraints, and frequent movement patterns predictive of one ore more classes. The algorithm for mining all frequent constrained patterns is named cAllMOP. Moreover, to control the density of pattern regions a clustering algorithm is exploited. The proposed method is efficient and scalable. Its efficiency is better than that of the previous algorithms AllMOP and GSP with respect to the compactness of discovered knowledge, execution time, and memory requirement.     

Peer-to-peer service provisioning in cloud computing environments

This paper aims to advance the management and delivery of services in large, heterogeneous, uncertain, and evolving cloud computing environments. The goal is important because such systems are becoming increasingly popular, yet existing service management methods do not scale well, and nor do they perform well under highly unpredictable conditions. If these problems can be solved, then Information Technology (IT) services can be made to operate in more scalable and reliable manner. In this paper, we present a peer-to-peer approach for managing services in large scale, dynamic, and evolving cloud computing environments. The system components such as virtualized services, computing servers, storage, and databases self-organize themselves using a peer-to-peer networking overlay. Inter-networking system components through peer-to-peer routing and information dissemination structure is essential to avoid the problems of management bottleneck and single point of failure that is predominantly associated with traditional centralized and hierarchical distributed (grids/clouds) system design approaches. We have validated our approach by conducting a set of rigorous performance evaluation study using the Amazon EC2 cloud computing environment. The results prove that managing services based on peer-to-peer routing and information dissemination structure is feasible and offers significant performance benefits as regards to overall system reliability, scalability, and self-management.     

A Distributed Approach for Monitoring Multicast Service Availability

With the deployment of native multicast in commercial networks, multicast is getting closer to becoming a ubiquitous service in the Internet. The success of this deployment largely depends on the availability of good management tools and systems. One of the most important management tasks for multicast is to verify the availability of the service to its users. This task is usually referred to as reachability monitoring. Reachability monitoring requires a number of monitoring stations to work together to collect this information in a distributed manner in the interdomain scale. In this paper we present a general architecture for multicast reachability monitoring systems and focus on three critical functions: agent configuration, monitoring, and feedback collection. For each component, we provide a number of alternative approaches to implement the required functionality and discuss their advantages and disadvantages. Then, we focus on the feedback collection component. To a large extent, it determines the complexity and the overhead of a monitoring system. Using simulations, we compare a number of alternative approaches for feedback collection and make suggestions on when to use each. We believe our work provides insight into the issues and considerations in designing and developing multicast reachability monitoring systems.     

OverSim:可扩展的应用层组播网络仿真框架

应用层组播作为IP组播的一个可行替代方案日益成为Internet应用研究的一个热点。如何仿真大规模、动态的应用层组播覆盖网络已经成为应用层组播方案研究人员面临的一个共同问题。本文在分析当前几款主流的覆盖网络仿真软件的基础上,提出了采用基于OMNet++网络仿真环境的Over-Sim覆盖网络仿真框架来模拟大规模、动态的应用层组播覆盖网络,并在此基础上研究分析可扩展的应用层组播方案。最后我们采用OverSim实现了对应用层组播协议Scribe的仿真并对结果进行了分析。仿真结果表明,OverSim可以在有限的硬件环境下,稳定地仿真高达上万个节点的大规模动态覆盖网络,仿真过程中内存消耗较低,且与覆盖网络规模呈线性关系。     

Middleware infrastructure for parallel and distributed programming models in heterogeneous systems

We introduce a middleware infrastructure that provides software services for developing and deploying high-performance parallel programming models and distributed applications on clusters and networked heterogeneous systems. This middleware infrastructure utilizes distributed agents residing on the participating machines and communicating with one another to perform the required functions. An intensive study of the parallel programming models in Java has helped identify the common requirements for a runtime support environment, which we used to define the middleware functionality. A Java-based prototype, based on this architecture, has been developed along with a Java object-passing interface (JOPI) class library. Since this system is written completely in Java, it is portable and allows executing programs in parallel across multiple heterogeneous platforms. With the middleware infrastructure, users need not deal with the mechanisms of deploying and loading user classes on the heterogeneous system. Moreover, details of scheduling, controlling, monitoring, and executing user jobs are hidden, while the management of system resources is made transparent to the user. Such uniform services are essential for facilitating the development and deployment of scalable high-performance Java applications on clusters and heterogeneous systems. An initial deployment of a parallel Java programming model over a heterogeneous, distributed system shows good performance results. In addition, a framework for the agents' startup mechanism and organization is introduced to provide scalable deployment and communication among the agents.     

云计算系统可靠性研究综述

云计算作为一种新型计算模式,已经受到了学术界和工业界的广泛关注.基于资源虚拟化技术,云计算能够以按需使用、按使用量付费的方式为用户提供基础设施、平台、软件等服务.因此,越来越多的企业和组织选择云计算来部署他们的科学或商业应用.然而,随着用户数量的不断增加,数据中心的规模在迅速扩大、架构变得日益复杂,导致云计算系统的运行故障频繁发生,造成了巨大的损失.因此在规模巨大、架构复杂的云计算系统中,如何保障系统的可靠性已经成为一个极具挑战性的问题.针对云计算可靠性问题,概述了云计算系统中常见的各种故障,并详细描述了目前云计算中提高可靠性关键的故障管理技术;由于故障管理技术的应用会不可避免地增加系统的能耗,因此介绍了云计算中可靠性与能耗权衡问题的研究现状;最后列举了当前云计算可靠性研究中存在的主要挑战.     

EScribe:一种动态路由的Topic发布订阅系统

Scribe是一种经典的基于Topic的发布订阅系统,它通过分布式组播树将Event分发给订阅者.Scribe需要定期维护组播树,因此造成了大量冗余的Event传递和高昂的维护代价.提出一种基于Scribe的增强型Topic发布订阅系统,简称EScribe.EScribe利用布隆过滤器存储Pastry叶子节点的订阅信息,动态地调整下一跳路由.节点维护子树的间隔时间随着节点在组播树中的层次增大而加长.实验结果表明,EScribe大幅减小了冗余Event传递的数量和组播树规模,也明显减小了Event分发和组播树维护的代价.     

An agent-based approach to solve dynamic meeting scheduling problems with preferences

Multi-agent systems are widely used to address large-scale distributed combinatorial applications in the real world. One such application is meeting scheduling (MS), which is defined by a variety of features. The MS problem is naturally distributed and especially subject to many alterations. In addition, this problem is characterized by the presence of users’ preferences that turn it into a search for an optimal rather than a feasible solution. However, in real-world applications users usually have conflicting preferences, which makes the solving process an NP-hard problem. Most research efforts in the literature, adopting agent-based technologies, tackle the MS problem as a static problem. They often share some common properties: allowing the relaxation of any user's time restriction, not dealing with achieving any level of consistency among meetings to enhance the efficiency of the solving process, not tackling the consequences of the dynamic environment, and especially not addressing the real difficulty of distributed systems which is the complexity of message passing operations.In an attempt to facilitate and streamline the process of scheduling meetings in any organization, the main contribution of this work is a new scalable agent-based approach for any dynamic MS problem (that we called MSRAC, for Meeting Scheduling with Reinforcement of Arc Consistency). In this approach we authorize only the relaxation of users’ preferences while maintaining arc-consistency on the problem. The underlying protocol can efficiently reach the optimal solution (satisfying some predefined optimality criteria) whenever possible, using only minimum localized asynchronous communications. This purpose is achieved with minimal message passing while trying to preserve at most the privacy of involved users. Detailed experimental results on randomly generated MS problems show that MSRAC is scalable and it leads to speed up over other approaches, especially for large problems with strong constraints.     

Adaptive low-latency peer-to-peer streaming and its application

Peer-to-peer (P2P) streaming is emerging as a viable communications paradigm. Recent research has focused on building efficient and optimal overlay multicast trees at the application level. A few commercial products are being implemented to provide voice services through P2P streaming platforms. However, even though many P2P protocols from the research community claim to be able to support large scale low-latency streaming, none of them have been adopted by a commercial voice system so far. This gap between advanced research prototypes and commercial implementations shows that there is a lack of a practical and scalable P2P system design that can provide low-latency service in a real implementation. After analyzing existing P2P system designs, we found two important issues that could lead to improvements. First, many existing designs that aim to build a low-latency streaming platform often make the unreasonable assumption that the processing time involved at each node is zero. However in a real implementation, these delays can add up to a significant amount of time after just a few overlay hops and make interactive applications difficult. Second, scant attention has been paid to the fact that even in a conversation involving a large number of users, only a few of the users are actually actively speaking at a given time. We term these users, who have more critical demands for low-latency, active users. In this paper, we detail the design of a novel peer-to-peer streaming architecture called ACTIVE. We then present a complete commercial scale voice chat system called AudioPeer that is powered by the ACTIVE protocol. The ACTIVE system significantly reduces the end-to-end delay experienced among active users while at the same time being capable of providing streaming services to very large multicast groups. ACTIVE uses realistic processing assumptions at each node and dynamically optimizes the streaming structure while the group of active users changes over time. Consequently, it provides virtually all users with the low-latency service that before was only possible with a centralized approach. We present results from both simulations and our real implementation, which clearly show that our ACTIVE system is a feasible approach to scalable, low-latency P2P streaming.     

A network infrastructure for IP mobility support in metropolitan areas

The original design of the Internet and its underlying protocols did not anticipate users to be mobile. With the growing interest in supporting mobile users and mobile computing, a great deal of work is taking place to solve this problem. For a solution to be practical, it has to integrate easily with existing Internet infrastructure and protocols, and offer an adequate migration path toward what might represent the ultimate solution. In that respect, the solution has to be incrementally scalable to handle a large number of mobile users and wide geographical scopes, and well performing so as to support all application requirements including voice and video communications and a wide range of mobility speeds. In this paper, we present a survey of the state-of-the-art and propose a scalable infrastructure to support mobility in Internet protocol networks. In that respect, we exploit local area network (LAN) technologies to create the network infrastructure necessary to offer connectivity to mobile users across any geographical area (building, campus and metropolis). The intrinsic properties of LAN technologies and their underlying protocols, namely flat address space, transparent learning and low complexity renders this solution particularly cost effective for supporting user mobility. In particular, we propose a network topology and a set of protocols that render the infrastructure scalable to a large geographical area and many users.     

Live Distributed Objects: Enabling the Active Web

Distributed computing has lagged behind the productivity revolution that has transformed the desktop in recent years. Programmers still generally treat the Web as a separate technology space and develop network applications using low-level message-passing primitives or unreliable Web services method invocations. Live distributed objects are designed to offer developers a scalable multicast infrastructure that's tightly integrated with a runtime environment.     

大数据存储技术进展

在大数据时代,我们面临着多种数据类型,数据规模以前所未有的速度增长,这给数据存储、管理以及分析带来了很大的挑战。传统的单机存储引擎显然不能满足数据爆炸性增长的需求,需要构建高性能、高可扩展、低成本、易用的分布式存储系统基础设施。本文对不同的分布式存储系统和其中的关键技术的研究进行全面的阐述和分析。     

Efficient techniques for time-constrained information dissemination using location-based social networks

Social networks have undergone an explosive growth in recent years. They constitute a central part of users׳ everyday lives as they are used as major tools for the spread of information, ideas and notifications among the members of the network. In this work we investigate the use of location-based social networks as a medium of emergency notification, for efficient dissemination of emergency information among members of the social network under time constraints. Our objective is the following: given a location-based social network comprising a number of mobile users, the social relationships among the users, the set of recipients, and the corresponding timeliness requirements, our goal is to select an appropriate subset of users so that the spread of information is maximized, time constraints are satisfied and costs are considered. We propose LATITuDE, our system that investigates the interactions among the members of the social network to infer their social relationships, and develop scalable dissemination mechanisms that select the most efficient set of users to initiate the dissemination process in order to maximize the information reach among the appropriate receivers within a time window. Our detailed experimental results illustrate that our approach is practical, effectively addresses the problem of informing the appropriate set of users within a deadline when an emergency event occurs, uses a small number of messages, and consistently outperforms its competitors.     

Capacity Gain of Mixed Multicast/Unicast Transport Schemes in a TV Distribution Network

This paper presents three approaches to estimate the required resources in an infrastructure where digital TV channels can be delivered in unicast or multicast (broadcast) mode. Such situations arise for example in Cable TV, IPTV distribution networks or in (future) hybrid mobile TV networks. The three approaches presented are an exact calculation, a Gaussian approximation and a simulation tool. We investigate two scenarios that allow saving bandwidth resources. In a static scenario, the most popular channels are multicast and the less popular channels rely on unicast. In a dynamic scenario, the list of multicast channels is dynamic and governed by the users' behavior. We prove that the dynamic scenario always outperforms the static scenario. We demonstrate the robustness, versatility and the limits of our three approaches. The exact calculation application is limited because it is computationally expensive for cases with large numbers of users and channels, while the Gaussian approximation is good exactly for such systems. The simulation tool takes long to yield results for small blocking probabilities. We explore the capacity gain regions under varying model parameters. Finally, we illustrate our methods by discussing some realistic network scenarios using channel popularities based on measurement data as much as possible.