Computational and data Grids in large-scale science and engineering

As the practice of science moves beyond the single investigator due to the complexity of the problems that now dominate science, large collaborative and multi-institutional teams are needed to address these problems. In order to support this shift in science, the computing and data-handling infrastructure that is essential to most of modern science must also change in order to support this increased complexity. This is the goal of computing and data Grids: software infrastructure that facilitates solving large-scale problems by providing the mechanisms to access, aggregate, and manage the computer network-based infrastructure of science. This infrastructure includes computing systems, data archive systems, scientific instruments, and computer-mediated human collaborations. This paper examines several large-scale science problems, their requirements for computing and data Grid infrastructure, and the current approaches to providing the necessary functionality.     

Characterizing Grids: Attributes, Definitions, and Formalisms

Grid systems and technologies have evolved over nearly a decade; yet, there is still no widely accepted definition for Grids. In particular, the essential attributes that distinguish Grids from other distributed computing environments have not been articulated. Most approaches to definition adopt a static view and consider only the properties and components of, or the applications supported by, Grids. The definition proposed in this paper is based on the runtime semantics of distributed systems. Rather than attempt to simply compare static characteristics of Grids and other distributed computing environments, this paper analyzes operational differences, from the viewpoint of an application executing in both environments. Our definition is expressed formally as an Abstract State Machine that facilitates the analysis of existing Grid systems or the design of new ones with rigor and precision. This new, semantical approach proposes an alternative to the currently accepted models for determining whether or not a distributed system is a Grid.     

A Launch-time Scheduling Heuristics for Parallel Applications on Wide Area Grids

Large and dynamic computational Grids, generally known as wide-area Grids, are characterized by a large availability, heterogene- ity on computational resources, and high vari- ability on their status during the time. Such Grid infrastructures require appropriate schedule mechanisms in order to satisfy the application performance requirements (QoS). In this paper we propose a launch-time heuristics to schedule component-based parallel applications on such kind of Grid. The goal of the proposed heuristics is threefold: to meet the minimal task computation- al requirement, to maximize the throughput between communicating tasks, and to evaluate on-the-fly the resource availability to minimize the aging effect on the resources state. We evaluate the proposed heuristics by simulations applying it to a suite of task graphs and Grid platforms randomly generated. Moreover, a further test was conducted to schedule a real application on a real Grid. Experimental results shown that the proposed solution can be a viable one.     

Enhancing Computational Grids with Peer-to-Peer Technology for Large Scale Service Discovery

Within computational Grids, some services (typically software components, e.g., linear algebra libraries) are made available by some servers to some clients. In spite of the growing popularity of such Grids, the service discovery, although efficient in many cases, does not reach several requirements. Among them, the flexibility of the discovery and its efficiency on wide-area dynamic platforms are two major issues. Therefore, it becomes crucial to propose new tools coping with such platforms. Emerging peer-to-peer technologies provide algorithms allowing the distribution and the retrieval of data items while addressing the dynamicity of the underlying network. Whereas merging peer-to-peer technology and Grid infrastructures has been widely suggested, very few implementations are available. The contribution of this paper is twofold. First, we present the design, the implementation and the experimentation of the first architecture, to our knowledge, extending traditional Network-Enabled Servers (NES) systems with an unstructured peer-to-peer network. This extension allows to dynamically connect distributed agents thus providing to clients an entry point to servers geographically distributed. Our implementation is based on the Diet middleware and the JXTA toolbox and experimentation have been conducted on a high speed network. Then, we study the service discovery in a pure peer-to-peer environment. We describe a new trie-based approach for the peer-to-peer service discovery service, supporting range queries while providing fault-tolerance and taking into account the topology of the underlying network. We validate this approach both by analysis and simulation. This work has been supported in part by the ANR project LEGO (ANR-05-CIGC-11).     

具有自主计算特征的计算网格资源备份服务系统

设计了具有自主计算特征的计算网格资源备份服务系统,该系统采用了先进的自主计算思想,具有一定程度的自识别、自感知、自管理、自组织、自优化、自稳定和自愈合的能力。     

ROAM: An Authorization Manager for Grids

The Resource Oriented Authorization Manager (ROAM) was created to provide a simple but flexible authorization system for the FusionGrid computational Grid. ROAM builds on and extends previous community efforts by both responding to access authorization requests and by providing a Web interface for resource management. ROAM works with the Globus Resource Allocation Manager (GRAM), and is general enough to be used by other virtual organizations that use Globus middleware or X.509/TLS authentication schemes to secure a Grid of distributed resources. In addition to describing ROAM, this paper discusses the basic design parameters of a Grid authorization system and the reasons for the choices made in the ROAM design.     

基于模糊粒子群优化的计算网格工作调度算法

网格计算是利用网络把分散的计算资源组织起来解决复杂问题的计算模式,工作调度是待解决的主要问题之一。本文提出一种基于模糊粒子群优化的网格计算工作调度算法,该算法利用模糊粒子群优化动态地产生网格计算工作调度的优化方案,使现有计算资源完成所有工作的时间最小化。实验结果表明,与基于遗传算法、模拟退火、蚁群算法的工作调度方法相比,所提出的算法在时间和精度上具有一定的优势。     

知识图谱划分算法研究综述

知识图谱是人工智能的重要基石,因其包含丰富的图结构和属性信息而受到广泛关注.知识图谱可以精确语义描述现实世界中的各种实体及其联系,其中顶点表示实体,边表示实体间的联系.知识图谱划分是大规模知识图谱分布式处理的首要工作,对知识图谱分布式存储、查询、推理和挖掘起基础支撑作用.随着知识图谱数据规模及分布式处理需求的不断增长,...     

Market-oriented Grids and Utility Computing: The State-of-the-art and Future Directions

Traditional resource management techniques (resource allocation, admission control and scheduling) have been found to be inadequate for many shared Grid and distributed systems, that consist of autonomous and dynamic distributed resources contributed by multiple organisations. They provide no incentive for users to request resources judiciously and appropriately, and do not accurately capture the true value, importance and deadline (the utility) of a user’s job. Furthermore, they provide no compensation for resource providers to contribute their computing resources to shared Grids, as traditional approaches have a user-centric focus on maximising throughput and minimising waiting time rather than maximising a providers own benefit. Consequently, researchers and practitioners have been examining the appropriateness of ‘market-inspired’ resource management techniques to address these limitations. Such techniques aim to smooth out access patterns and reduce the chance of transient overload, by providing a framework for users to be truthful about their resource requirements and job deadlines, and offering incentives for service providers to prioritise urgent, high utility jobs over low utility jobs. We examine the recent innovations in these systems (from 2000–2007), looking at the state-of-the-art in price setting and negotiation, Grid economy management and utility-driven scheduling and resource allocation, and identify the advantages and limitations of these systems. We then look to the future of these systems, examining the emerging ‘Catallaxy’ market paradigm. Finally we consider the future directions that need to be pursued to address the limitations of the current generation of market oriented Grids and Utility Computing systems.     

P2P-MPI: A Peer-to-Peer Framework for Robust Execution of Message Passing Parallel Programs on Grids

This paper presents P2P-MPI, a middleware aimed at computational Grids. From the programmer point of view, P2P-MPI provides a message-passing programming model which enables the development of MPI applications for Grids. Its originality lies in its adaptation to unstable environments. First, the peer-to-peer design of P2P-MPI allows for a dynamic discovery of collaborating resources. Second, it gives the user the possibility to adjust the robustness of an execution thanks to an internal process replication mechanism. Finally, we measure the performance of the integrated message passing library on several benchmarks and on different hardware platforms.     

面向异质图的在线图划分算法

图划分算法是分布式图计算系统里的重要组成部分, 它将一个图划分为若干子图以便在分布式系统中运行, 并将子图上的点和边数据及子图上的计算任务分配到各分区. 异质图是现实世界中广泛存在的一种图, 它是指具有多种节点类型或边类型的图, 在针对异质图的计算过程中, 现有的图划分算法对于异质图的处理没有考虑到以下问题: 在图计算过程中, 不同类型的节点和边携带的数据量可能不同; 不同的节点和边类型, 可能会采用不同的处理算法, 其计算时间也会不同. 针对现有图划分方法的不足, 本文提出一种面向异质图的在线图划分算法OGP-HG算法, 并对现有的GraphX图计算引擎进行改进, 将OGP-HG算法在改进后的图计算引擎中实现. 本文提出的OGP-HG算法通过计算节点划分到不同分区上的负载均衡得分和边划分到不同分区上的数据均衡得分, 得到使异质图负载和内存占用均衡的划分结果. 实验表明, 与传统图划分算法相比, 该算法提高异质图计算效率1.05–1.4倍.     

Neuroscience instrumentation and distributed analysis of brain activity data: a case for eScience on global Grids

The distribution of knowledge (by scientists) and data sources (advanced scientific instruments), and the need for large‐scale computational resources for analyzing massive scientific data are two major problems commonly observed in scientific disciplines. Two popular scientific disciplines of this nature are brain science and high‐energy physics. The analysis of brain‐activity data gathered from the MEG (magnetoencephalography) instrument is an important research topic in medical science since it helps doctors in identifying symptoms of diseases. The data needs to be analyzed exhaustively to efficiently diagnose and analyze brain functions and requires access to large‐scale computational resources. The potential platform for solving such resource intensive applications is the Grid. This paper presents the design and development of MEG data analysis system by leveraging Grid technologies, primarily Nimrod‐G, Gridbus, and Globus. It describes the composition of the neuroscience (brain‐activity analysis) application as parameter‐sweep application and its on‐demand deployment on global Grids for distributed execution. The results of economic‐based scheduling of analysis jobs for three different optimizations scenarios on the world‐wide Grid testbed resources are presented along with their graphical visualization. Copyright © 2005 John Wiley & Sons, Ltd.     

Packet Spacing: An Enabling Mechanism for Delivering Multimedia Content in Computational Grids

Streaming multimedia with UDP has become increasingly popular over distributed systems like the Internet. Scientific applications that stream multimedia include remote computational steering of visualization data and video-on-demand teleconferencing over the Access Grid. However, UDP does not possess a self-regulating, congestion-control mechanism; and most best-effort traffic is served by congestion-controlled TCP. Consequently, UDP steals bandwidth from TCP such that TCP flows starve for network resources. With the volume of Internet traffic continuing to increase, the perpetuation of UDP-based streaming will cause the Internet to collapse as it did in the mid-1980's due to the use of non-congestion-controlled TCP.To address this problem, we introduce the counter-intuitive notion of inter-packet spacing with control feedback to enable UDP-based applications to perform well in the next-generation Internet and computational grids. When compared with traditional UDP-based streaming, we illustrate that our approach can reduce packet loss over 50% without adversely affecting delivered throughput.     

Hierarchical Partitioning Techniques for Structured Adaptive Mesh Refinement Applications

This paper presents the design and preliminary evaluation of hierarchical partitioning and load-balancing techniques for distributed structured adaptive mesh refinement (SAMR) applications. The overall goal of these techniques is to enable the load distribution to reflect the state of the adaptive grid hierarchy and exploit it to reduce synchronization requirements, improve load-balance, and enable concurrent communications and incremental redistribution. The hierarchical partitioning algorithm (HPA) partitions the computational domain into subdomains and assigns them to hierarchical processor groups. Two variants of HPA are presented in this paper. The static hierarchical partitioning algorithm (SHPA) assigns portions of overall load to processor groups. In SHPA, the group size and the number of processors in each group is setup during initialization and remains unchanged during application execution. It is experimentally shown that SHPA reduces communication costs as compared to the Non-HPA scheme, and reduces overall application execution time by up to 59%. The adaptive hierarchical partitioning algorithm (AHPA) dynamically partitions the processor pool into hierarchical groups that match the structure of the adaptive grid hierarchy. Initial evaluations of AHPA show that it can reduce communication costs by up to 70%.     

GMarte: Grid middleware to abstract remote task execution

Grid computing technologies are now being largely deployed with the widespread adoption of the Globus Toolkit as the industrial standard Grid middleware. However, its inherent steep learning curve discourages the use of these technologies for non‐experts. Therefore, to increase the use of Grid computing, it is important to have high‐level tools that simplify the process of remote task execution. In this paper we introduce a middleware, developed on top of the Java Commodity Grid, which offers an object‐oriented, user‐friendly application programming interface, from the Java language, which eases remote task execution for computationally intensive applications. Copyright © 2006 John Wiley & Sons, Ltd.     

网格环境下工作流的费用-时间调度算法

开放网格服务架构（OGSA）和计算经济模型的提出,使得动态的、不同QoS的服务支持下的资源调度成为一个复杂且具有挑战性的问题。该文提出了网格环境下基于费用-时间的工作流调度算法,该算法采用动态资源选择策略适应网格计算环境下的动态性和自治性。在追求较小的工作流完成时间的同时,对费用进行了优化。模拟结果显示该调度算法符合计算网格的复杂环境,能够更好地满足不同用户的实际需要。     

一种基于双向拍卖机制的计算网格资源分配方法

针对计算网格资源的特点以及运用经济机制进行网格资源管理所具有的灵活性及有效性,提出一种改进的基于双向拍卖机制的网格资源分配方法.首先,描述了基于双向拍卖机制的资源分配框架,整个系统由买方、卖方和计算资源经纪人组成.然后,针对网格中的CPU资源,提出一种改进的双向拍卖机制,采用统一拍卖方式,可以灵活调节交易双方的付费.进而,分析了该双向拍卖机制满足优势策略激励相容、预算平衡以及个人理性的特点,并定义了拍卖机制的效率.最后,通过实验分析了双向拍卖分配机制的效率.     

VCPPS:一种面向并行Verilog模拟的代码分割器

本文介绍了一种面向并行模拟的Verilog代码分割器VCPPS。VCPPS通过图形化的方式进行人机交互,并采取用户启发式的方法指导分割。文章介绍了VCPPS设计与实现中的一些关键技术,通过分析可以看出,VCPPS可以正确地完成繁琐的Verilog代码分割工作,为并行Verilog模拟提供支持。     

An optimization-based adaptive resource management framework for economic Grids: A switching mechanism

The application of Grid computing has been broadening day by day. An increasing number of users has led to the requirement of a job scheduling process, which can benefit them through optimizing their utility functions. On the other hand, resource providers are exploring strategies suitable for economically efficient resource allocation so that they can maximize their profit through satisfying more users. In such a scenario, economic-based resource management strategies (economic models) have been found to be compelling to satisfy both communities. However, existing research has identified that different economic models are suitable for different scenarios in Grid computing. The Grid application and resource models are typically very dynamic, making it challenging for a particular model for delivering stable performance all the time. In this work, our focus is to develop an adaptive resource management architecture capable of dealing with multiple models based on the models’ domains of strengths (DOS). Our preliminary results show promising outcomes if we consider multiple models rather than relying on a single model throughout the life cycle of a Grid.