Achieving Interoperation of Grid Data Resources via Workflow Level Integration

Production Grids are becoming widely utilized by the e-Science community to run computation and data intensive experiments more efficiently. Unfortunately, different production Grid infrastructures are based on different middleware technologies, both for computation and for data access. Although there is significant effort from the Grid community to standardize the underlying middleware, solutions that allow existing non-standard tools to interoperate are one of the major concerns of Grid users today. This paper describes the generic requirements towards the interoperation of Grid data resources within computational workflows, and suggests integration techniques that allow workflow engines to access various heterogeneous data resources during workflow execution. Reference implementations of these techniques are presented and recommendations on their applicability and suitability are made.     

The Organization and Management of Grid Infrastructures

Grid computing technology has become fundamental to e-Science. As the virtual organizations established by scientific communities progress from testing their applications to more routine usage, maintaining reliable and adaptive grid infrastructures becomes increasingly important.     

Grid Interoperability Based on a Formal Design

We can distinguish two different Grid concepts: desktop and service Grids. Both Grid concepts have their advantages and disadvantages, however these are different. For example desktop Grids are a cost-effective platform, but sometimes unreliable. On the other hand service Grids are highly reliable, but need remarkable funding. The aim of Grid interoperability is to combine the advantages of the different Grid concepts, so the integrated infrastructure offers the best of both concepts. Within the paper we define the Grid interoperability problem, and approximate to the generic architecture through a formal model. We prove formally that the resulting architecture solves the Grid interoperability problem, and is generic enough to interconnect different Grid infrastructures with minor work. We also show in the paper that the formal concept can be applied for creating a gLite to BOINC bridge, and the performance of the core bridge implementation is satisfactory.     

Merging Grid Technologies

This paper reports the integration of the astronomical Grid solution realised in the Astro-WISE information system with the EGEE Grid and the porting of Astro-WISE applications on EGEE. We review the architecture of the Astro-WISE Grid, define the problems for the integration of the Grid infrastructures and our solution to these problems. We give examples of applications running on Astro-WISE and EGEE and review future development of the merged system.     

Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal

Computational Grids connect resources and users in a complex way in order to deliver nontrivial qualities of services. According to the current trend various communities build their own Grids and due to the lack of generally accepted standards these Grids are usually not interoperable. As a result, large scale sharing of resources is prevented by the isolation of Grid systems. Similarly, people are isolated, because the collaborative work of Grid users is not supported by current environments. Each user accesses Grids as an individual person without having the possibility of organizing teams that could overcome the difficulties of application development and execution more easily. The paper describes a new workflow-oriented portal concept that solves both problems. It enables the interoperability of various Grids during the execution of workflow applications, and supports users to develop and run their Grid workflows in a collaborative way. The paper also introduces a classification model that can be used to identify workflow-oriented Grid portals based on two general features: Ability to access multiple Grids, and support for collaborative problem solving. Using the approach the different potential portal types are introduced, their unique features are discussed and the portals and Problem Solving Environments (PSE) of our days are classified. The P-GRADE Portal as a Globus-based implementation for the classification model is also presented. The work described in this paper is supported by the Hungarian Grid project (IHM 4671/1/2003), by the Hungarian OTKA project (No. T042459) and a collaboration project with the University of Reading.     

Regional Climate Simulations on EU-INDIA Grid Infrastructures: Methodologies and Performance

In this paper, we present a possible approach to use different Grid infrastructures across Europe and India for regional climate simulations and discuss in details the advantages and limitations in using them. The application taken into consideration is the Regional Climate Model RegCM4, which has been recently re-written for making it more efficient and easier to be fully exploited on any kind of computational infrastructure. We describe here the methods applied to port this package on the Grid infrastructures made available by the EU-IndiaGrid project. We also discuss different approaches, the way to run the model on both European and Indian infrastructures and our promising approach to deal with data management issues. Use of RegCM on Grid infrastructure has further been compared with that on HPC resources. The domain worked upon for these tests is the South-Asia CORDEX domain, which is of great importance for the Indian Summer Monsoon. Our final conclusion is that, for certain class of experiments, RegCM4 model can be efficiently and easily integrated on Grid infrastructures, by means of the procedures described in this paper.     

网格环境下费用约束的科学工作流可靠调度算法

网格基础设施是目前科学工作流应用规划、部署和执行的主要支撑环境.然而由于网格资源的自治、动态及异构性,如何在保障用户QoS约束下有效调度科学工作流是一个研究热点.针对费用约束下的科学工作流调度问题,为了提高其执行的可靠性,本文使用随机服务模型描述资源节点的动态服务能力并考虑本地任务负载对资源执行性能的影响,给出一种资源可靠性的评估方法,在此基础上提出一种费用约束下的科学工作流可靠调度算法RSASW.仿真实验结果表明RSASW算法相对于GAIN3,GreedyTime-CD及PFAS算法,对工作流的执行具有很好的可靠性保障.     

A Grid resource brokering strategy based on resource and network performance in Grid

To achieve high performance distributed data access and computing in Grid environment, monitoring of resource and network performance is vital. Our proposed Grid network monitoring architecture is modeled by the Grid scheduler. The proposed Grid network monitoring retrieves network metrics using sensors as network monitoring tools. The mobile agents are migrated to start the sensors to measure the network metrics in all Grid Resources from the Resource Broker. The raw data provided by the monitoring tools is used to produce a high level view of the Grid through the set of internal cost functions. The network cost function is formed by combining various network metrics such as bandwidth, Round Trip Time, jitter and packet loss to measure the network performance. This paper presents the Grid Resource Brokering strategy which analyzes the network metrics along with the resource metrics for the selection of the Grid resource to submit the job and the proposed approach is integrated with CARE Resource Broker (CRB) for job submission. The experimental results are evident for the minimization of job completion time for the submitted job. The simulation results also prove that the more number of jobs are completed with the proposed strategy which influences the better utilization of the Grid resources.     

Searching for Software on the EGEE Infrastructure

Several large-scale Grid infrastructures are currently in operation around the world, federating an impressive collection of computational resources, a wide variety of application software, and hundreds of user communities. To better serve the current and prospective users of Grid infrastructures, it is important to develop advanced software retrieval services that could help users locate software components suitable to their needs. In this paper, we present the design and implementation of Minersoft, a distributed, multi-threaded harvester for application software located in large-scale Grid infrastructures. Minersoft crawls the sites of a Grid infrastructure, discovers installed software resources, annotates them with keyword-rich metadata, and creates inverted indexes that can be used to support full-text software retrieval. We present insights derived from the implementation and deployment of Minersoft on EGEE, one of the largest Grid production services currently in operation. Experimental results show that Minersoft achieves a high performance in crawling EGEE sites and discovering software-related files, and a high efficiency in supporting software retrieval.     

Writing programs that run EveryWare on the Computational Grid

The Computational Grid has been proposed, for the implementation of high-performance applications using widely dispersed computational resources. The goal of a Computational Grid is to aggregate ensembles of shared, heterogeneous, and distributed resources (potentially controlled by separate organizations) to provide computational, "power" to an application program. We provide a toolkit for the development of globally deployable Grid applications. The toolkit, called EveryWare, enables an application to draw computational power transparently from the Grid. It consists of a portable set of processes and libraries that can be incorporated into an application so that a wide variety of dynamically changing distributed infrastructures and resources can be used together to achieve supercomputer-like performance. We provide our experiences gained while building the EveryWare toolkit prototype and an explanation of its use in implementing a large-scale Grid application     

An Interoperable Grid Information System for Integrated Resource Monitoring Based on Virtual Organizations

In many Grid infrastructures different kinds of information services are in use, which utilize different incompatible data structures and interfaces to encode and provide their data. Homogeneous monitoring of these infrastructures with the monitoring data being accessible everywhere independently of the middleware which provided it, is the basis for a consistent status reporting on the Grids’ resources and services. Thus, interoperability or interoperation between the different information services in a heterogeneous Grid infrastructure is required. Monitoring data must contain the identity of the affected Virtual Organization (VO) so that it can be related to the resources and services the VO has allocated to enable VO-specific information provision. This paper describes a distributed architecture for an interoperable information service, which combines data unification and categorization with policies for VO membership, VO resource management and data transformations. This service builds the basis for an integrated and interoperating monitoring of Grids, which provide their data to more than one VO and utilize heterogeneous information services.     

A standards-based interoperable single sign-on framework in ARC Grid middleware

Security infrastructure is one of the most challenging tasks in the development, integration and deployment of Grid middlewares. Even though the Grid community addresses the security issue through public key infrastructures (PKI) to support mutual authentication using X.509 certificates, maintaining X.509 credentials is not that easy for non-IT-experts, and has proved to be an obstacle for a more wide deployment of Grid technologies. The identity federation is an increasingly popular technology that can facilitate cross-domain single sign-on without requiring the users to maintain any credentials additional to their own institutional accounts. We believe that utilizing identity federation for Grid middlewares is a promising path for the Grid technology to get more widely used. This paper describes a single sign-on infrastructure developed as a part of the NorduGrid ARC (Advanced Resource Connector) Grid middleware. It adopts the identity federation standard (SAML), as well as other Web Service standards. It focuses on a single sign-on solution at the middleware level for users to access Grids by only using their frequently used accounts, without being bothered to maintain X.509 credentials. Users can use their username/password only to access Grids developed in ARC middleware, as well as access Grids developed in other middlewares that requires users to provide X.509 certificates. Moreover, the single sign-on for workflow-like Grid applications (in which intermediate entities act on behalf of users) is also supported. As an important aspect of single sign-on, authorization is also considered by implementing an attribute-based authorization using SAML standard. In addition, the performance of single sign-on solution is measured. We identify performance limitations of security-related services inside this solution, and analyse the ways to avoid the limitations. To our knowledge, the work presented in this paper is the first evaluated implementation that utilizes identity federation for Grid usage on the middleware level.     

EMPEROR: An OGSA Grid Meta-Scheduler Based on Dynamic Resource Predictions

Scheduling constitutes an integral feature of Grid computing infrastructures, being also a key to realizing several of the Grid promises. In particular, scheduling can maximize the resources available to end users, accelerate the execution of jobs, while also supporting scalable and autonomic management of the resources comprising a Grid. Grid scheduling functionality hinges on middleware components called meta-schedulers, which undertake to automatically distribute jobs across the dispersed heterogeneous resources of a Grid. In this paper we present the design and implementation of a Grid meta-scheduler, which we call EMPEROR. EMPEROR provides a framework for implementing scheduling algorithms based on performance criteria. In implementing a particular instantiation of this framework, we have devised models for predicting host load and memory resources, and accordingly for estimating the running time of a task. These models hinge on time series analysis techniques and take into account results of the cluster computing literature. Apart from incorporating these models, EMPEROR provides fully fledged Grid scheduling functionality, which complies with OGSA standards as the later are reflected in the Globus toolkit. Specifically, EMPEROR interfaces to Globus middleware services (i.e., GSI, MDS, GRAM) towards discovering resources, implementing the scheduling algorithm and ultimately submitting jobs to local scheduling systems. By and large, EMPEROR is one of the few standards based meta-schedulers making use of dynamic scheduling information.     

网格体系结构详解

1.引言在“网格:面向虚拟组织的资源共享技术”一文中,我们对网格的研究领域及基本理念作了介绍,并介绍了网格体系结构的设计思想和体系结构框架。本文我们具体描述由Globus项目提出的网格体系结构——一种类似于Internet协议体系结构的层次式协议体系结构,并映射到Internet协议体系结构上。我们主要介绍构成网格体系结构的基本系统组件,详细规定/说明这些组件的目的与功能,并指出这些组件如何与别的组件交互。     

一种基于虚拟化技术的异构网格互操作机制研究与实现

针对目前网格系统在实现机制、技术支持上异构性大,缺乏互操作,提出一种基于虚拟化技术的异构网格互操作机制(Virtualization-based Grid Interoperation,VIGI).VIGI通过构建虚拟管理层和插件模块,实现异构网格互操作,同时保证各网格平台的独立性.GOS、CGSP是中国国家网格(CNGrid)和中国教育科研网格(ChinaGrid)两大异构网格支撑中间件.基于VIGI,实现了CNGrid和ChinaGrid互操作.实验验证虚拟化技术在异构网格互操作中的可行性和有效性.     

Grid benchmarking: vision,challenges, and current status

Grid benchmarking is an important and challenging topic of Grid computing research. In this paper, we present an overview of the key challenges that need to be addressed for the integration of benchmarking practices, techniques, and tools in emerging Grid computing infrastructures. We discuss the problems of performance representation, measurement, and interpretation in the context of Grid benchmarking, and propose the use of ontologies for organizing and describing benchmarking metrics. Finally, we present a survey of ongoing research efforts that develop benchmarks and benchmarking tools for the Grid. Copyright © 2006 John Wiley & Sons, Ltd.     

Cosmological Simulations and Data Exploration: a Testcase on the Usage of Grid Infrastructure

While new infrastructures for large computational challenges begin to be widely accessible to researchers, computational codes need to be re-designed to exploit new facilities. The Grid and the cloud computing concepts are changing the computational resource distribution and availability, and much effort start to be made to develop new codes for a better exploitation of new resources. This paper presents an example of the use of Grid resources, based on gLite middleware, to run cosmological simulations, that, up to now, are normally executed on Supercomputers. We have also used the Grid to explore and visualize the dataset. We discuss non particular the performance of FLY a parallel code implementing the octal-tree algorithm introduced by J. Barnes and P. Hut to compute the gravitational field efficiently. It simulates the evolution of the collisionless component of the material content of our Universe. FLY was originally developed to run on mainframe systems using the one-side communication paradigm, but we are now presenting a modified version of the computational algorithm to exploit the Grid environment. We also integrated the data exploration and visualization process on the Grid, to obtain preliminary results using the distributed facilities.     

广域网虚拟平台：网格计算和对等计算的融合

近几年，分布式计算领域出现了两个研究热点：网格计算(Grid Computing)和对等计算(PeertoPeer Computing)。网格计算和对等计算以不同的方法组织大规模分布式的资源，包括计算能力、存储资源和带宽等。两者的研究领域有一定的重叠，又有很好的互补性，融合将是必然趋势。我们构想了一个广域网虚拟平台，全世界的计算机连接成一个整体，任何人随时随地都可以得到所需的资源和服务。正是这一理想，驱动着网格计算和对等计算技术的快速进步。     

An overview of S-OGSA: A Reference Semantic Grid Architecture

The Grid's vision, of sharing diverse resources in a flexible, coordinated and secure manner through dynamic formation and disbanding of virtual communities, strongly depends on metadata. Currently, Grid metadata is generated and used in an ad hoc fashion, much of it buried in the Grid middleware's code libraries and database schemas. This ad hoc expression and use of metadata causes chronic dependency on human intervention during the operation of Grid machinery, leading to systems which are brittle when faced with frequent syntactic changes in resource coordination and sharing protocols.The Semantic Grid is an extension of the Grid in which rich resource metadata is exposed and handled explicitly, and shared and managed via Grid protocols. The layering of an explicit semantic infrastructure over the Grid Infrastructure potentially leads to increased interoperability and greater flexibility.In recent years, several projects have embraced the Semantic Grid vision. However, the Semantic Grid lacks a Reference Architecture or any kind of systematic framework for designing Semantic Grid components or applications. The Open Grid Service Architecture (OGSA) aims to define a core set of capabilities and behaviours for Grid systems. We propose a Reference Architecture that extends OGSA to support the explicit handling of semantics, and defines the associated knowledge services to support a spectrum of service capabilities. Guided by a set of design principles, Semantic-OGSA (S-OGSA) defines a model, the capabilities and the mechanisms for the Semantic Grid.We conclude by highlighting the commonalities and differences that the proposed architecture has with respect to other Grid frameworks.     

Grid today, clouds on the horizon

By the time of CCP 2008, the largest scientific machine in the world - the Large Hadron Collider - had been cooled down as scheduled to its operational temperature of below 2 degrees Kelvin and injection tests were starting. Collisions of proton beams at 5+5 TeV were expected within one to two months of the initial tests, with data taking at design energy (7+7 TeV) foreseen for 2009.In order to process the data from this world machine, we have put our “Higgs in one basket” - that of Grid computing [The Worldwide LHC Computing Grid (WLCG), in: Proceedings of the Conference on Computational Physics 2006 (CCP 2006), vol. 177, 2007, pp. 219-223]. After many years of preparation, 2008 saw a final “Common Computing Readiness Challenge” (CCRC'08) - aimed at demonstrating full readiness for 2008 data taking, processing and analysis. By definition, this relied on a world-wide production Grid infrastructure.But change - as always - is on the horizon. The current funding model for Grids - which in Europe has been through 3 generations of EGEE projects, together with related projects in other parts of the world, including South America - is evolving towards a long-term, sustainable e-infrastructure, like the European Grid Initiative (EGI) [The European Grid Initiative Design Study, website at http://web.eu-egi.eu/]. At the same time, potentially new paradigms, such as that of “Cloud Computing” are emerging.This paper summarizes the results of CCRC'08 and discusses the potential impact of future Grid funding on both regional and international application communities. It contrasts Grid and Cloud computing models from both technical and sociological points of view. Finally, it discusses the requirements from production application communities, in terms of stability and continuity in the medium to long term.