Workflow-Based Data Parallel Applications on the EGEE Production Grid Infrastructure

Setting up and deploying complex applications on a Grid infrastructure is still challenging and the programming models are rapidly evolving. Efficiently exploiting Grid parallelism is often not straight forward. In this paper, we report on the techniques used for deploying applications on the EGEE production Grid through four experiments coming from completely different scientific areas: nuclear fusion, astrophysics and medical imaging. These applications have in common the need for manipulating huge amounts of data and all are computationally intensive. All the cases studied show that the deployment of data intensive applications require the development of more or less elaborated application-level workload management systems on top of the gLite middleware to efficiently exploit the EGEE Grid resources. In particular, the adoption of high level workflow management systems eases the integration of large scale applications while exploiting Grid parallelism transparently. Different approaches for scientific workflow management are discussed. The MOTEUR workflow manager strategy to efficiently deal with complex data flows is more particularly detailed. Without requiring specific application development, it leads to very significant speed-ups.     

Workflow Concepts of the Java CoG Kit

Many scientific simulations and experiments require the coordination of numerous tasks posed by interdisciplinary research teams. Grids can provide access to the necessary high-end resources to conduct such tasks. The complex tasks and their interactions must be supported through convenient tools. To address this issue, we introduce a number of Grid abstractions that make the development of Grid middleware-independent tools possible and allow for the integration of a number of commodity tools. Our vision is implemented through an integrated approach based on a layered architecture that bridges the gap between Grid middleware and scientific applications. Our abstractions include specialized services, a Grid workflow engine and language, and Gridfaces – graphical abstractions that can be employed in science portals and standalone applications.     

Campus Grids Meet Applications: Modeling, Metascheduling and Integration

Air Quality Forecasting (AQF) is a new discipline that attempts to reliably predict atmospheric pollution. An AQF application has complex workflows and in order to produce timely and reliable forecast results, each execution requires access to diverse and distributed computational and storage resources. Deploying AQF on Grids is one option to satisfy such needs, but requires the related Grid middleware to support automated workflow scheduling and execution on Grid resources. In this paper, we analyze the challenges in deploying an AQF application in a campus Grid environment and present our current efforts to develop a general solution for Grid-enabling scientific workflow applications in the GRACCE project. In GRACCE, an application’s workflow is described using GAMDL, a powerful dataflow language for describing application logic. The GRACCE metascheduling architecture provides the functionalities required for co-allocating Grid resources for workflow tasks, scheduling the workflows and monitoring their execution. By providing an integrated framework for modeling and metascheduling scientific workflow applications on Grid resources, we make it easy to build a customized environment with end-to-end support for application Grid deployment, from the management of an application and its dataset, to the automatic execution and analysis of its results.The work has been performed as part of the University of Houston’s Sun Microsystems Center of Excellence in Geosciences [38].     

Practical Experience from Porting and Executing the Wien2k Application on the EGEE Production Grid Infrastructure

While the Grid promises to deliver a large number of computation nodes to a user, this computation power is not usable without the proper adaption of the application for the Grid. In this paper, we describe the methods used to port and execute a particular application, Wien2k, on the EGEE production Grid. First, the process of porting the application is described. Then, we investigate the measures necessary to execute the application in this production Grid environment efficiently. Although the focus is on this special application, we describe generic methods which can be applied to all applications. We specifically address: Creating a workflow from an application and mapping this workflow to a Grid workflow using the activity attraction pattern. We discuss workflow engines which support cycles in their application workflow. We investigate naïve and worker scheduling techniques. A short introduction into licensing on the Grid is given. Optimisation techniques such as deployment re-use are discussed. Different data transfer mechanisms, centralised data transfer, data re-use, storage element data transfer, and peer-to-peer data transfer are compared. The paper is concluded with suggestions for further workflow porting.     

网格数据访问中间件的设计与实现

网格服务的一个重要目标是实现数据资源的共享,然而网格环境的复杂性阻碍了对数据的有效访问。利用中间件屏蔽底层的复杂性,实现对网格环境下跨平台,异构数据资源的统一访问是当前研究的重点。该文结合中科院科学数据网格项目的特点,阐述了在开放网格服务架构下用中间件来实现网格中数据的统一访问的原理,并给出了实现的方法。     

Virtual workflow system for distributed collaborative scientific applications on Grids

Grid computing has become an effective computing technique in recent years. This paper develops a virtual workflow system to construct distributed collaborative applications for Grid users. The virtual workflow system consists three levels: abstract workflow system, translator and concrete workflow system. The research highlight of the implementation is that this workflow system is developed based on CORBA and Unicore Grid middleware. Furthermore, this implementation can support legacy application developed with Parco and C++ codes. This virtual workflow system can provide efficient GUI for users to organize distributed scientific collaborative applications and execute them on Grid resources. We present the design, implementation, and evaluation of this virtual workflow system in the paper.     

Grid Deployment Experiences: Grid Interoperation

Over recent years a number of Grid projects have emerged which have built Grid infrastructures that are now the computing backbones for various user communities. A significant number of these communities are limited to one Grid Infrastructure due to the different middleware and operations procedures used. Grid Interoperation is trying to bridge these differences and enable Virtual Organizations to access resources independent of the Grid project affiliation. Building upon the experiences the authors have gained while working on interoperation between EGEE and various other Grid infrastructures as well as through co-chairing the Grid Interoperation Now (GIN) efforts of the Open Grid Forum (OGF), this paper gives an overview of Grid Interoperation and describes various methods that can be used to connect Grid Infrastructures. The case is made for standardization in key areas and why the Grid community should move more aggressively towards standards.     

REST架构工作流中间件设计与实现

为解决异构应用间的流程化协作问题,比较了主流的面向服务架构实现方案,研究了一种REST架构工作流中间件.该中间件以工作流支撑服务的形式发布,为工作流应用系统提供统一的流程管理环境和任务流转能力.结合需求细化了工作流中间件外部访问接口,详细介绍了REST架构工作流中间件的资源发布方式,以及资源使用过程中工作流控制数据和工作流相关数据的序列化方式,并以实例验证了中间件的可用性.     

Introduce: An Open Source Toolkit for Rapid Development of Strongly Typed Grid Services

Service-oriented architectures and applications have gained wide acceptance in the Grid computing community. A number of tools and middleware systems have been developed to support application development using Grid Services architectures. Most of these efforts, however, have focused on low-level support for management and execution of Grid services, management of Grid-enabled resources, and deployment and execution of applications that make use of Grid services. Simple-to-use service development tools, which would allow a Grid service developer to leverage Grid technologies without needing to know low-level details, are becoming increasingly important for wider application of the Grid. In this paper, we describe an open-source, extensible toolkit, called Introduce, that supports easy development and deployment of Web Services Resource Framework (WSRF) compliant services. Introduce is designed to reduce the service development and deployment effort by hiding low level details of the Globus Toolkit and to enable the implementation of strongly typed services. In strongly typed services, a service produces and consumes data types that are well-defined and published in the Grid. This enables data-level syntactic interoperability so that clients and services can access and consume data elements programmatically and correctly. We expect that enabling strongly typed Grid services while lowering the difficulty of entry to the Grid via toolkits like Introduce will have a major impact to the success of the Grid and its wider adoption as a viable technology of choice in the commercial sector as well as in academic, medical, and government research.     

基于WebGIS信息整合与利用的林业电子政务

林业电子政务要向社会公众提供专业服务。面向服务的架构和万维网地理信息系统是处理林业电子政务信息资源整合和利用的重要技术工具。文章以林业电子政务中森林火点检测的应用流程为例展示上述技术的运用方法。工作流程可以利用分布式的遥感数据、矢量辅助数据资源、森林火点检测算法和万维网地图发布服务,通过一个网格的基础架构把这些资源集成起来形成林业专业应用。文章展示的试验床用Web服务技术封装了专业算法。网格中间件采用的是自主研制的SIRGE,调度资源的工作流中间件采用的是自主研制的SIGApp,它们的专长是可以处理空间数据。用于通过这个试验床可以访问基于不同来源数据和服务得到的森林火点信息,为林业电子政务服务。     

Digital Preservation in Grids and Clouds: A Middleware Approach

Digital preservation is the persistent archiving of digital assets for future access and reuse, irrespective of the underlying platform and software solutions. Existing preservation systems have a strong focus on Grids, but the advent of cloud technologies offers an attractive option. We describe a middleware system that enables a flexible choice between a Grid and a cloud for ad-hoc computations that arise during the execution of a preservation workflow and also for archiving digital objects. The choice between different infrastructures remains open during the lifecycle of the archive, ensuring a smooth switch between different solutions to accommodate the changing requirements of the organization that needs its digital assets preserved. We also offer insights on the costs, running times, and organizational issues of cloud computing, proving that the cloud alternative is particularly attractive for smaller organizations without access to a Grid or with limited IT infrastructure.     

ROAM: An Authorization Manager for Grids

The Resource Oriented Authorization Manager (ROAM) was created to provide a simple but flexible authorization system for the FusionGrid computational Grid. ROAM builds on and extends previous community efforts by both responding to access authorization requests and by providing a Web interface for resource management. ROAM works with the Globus Resource Allocation Manager (GRAM), and is general enough to be used by other virtual organizations that use Globus middleware or X.509/TLS authentication schemes to secure a Grid of distributed resources. In addition to describing ROAM, this paper discusses the basic design parameters of a Grid authorization system and the reasons for the choices made in the ROAM design.     

The AMGA Metadata Service

We present the AMGA metadata catalogue, which was developed as part of the EGEE (enabling Grids for EsciencE) project’s gLite Grid middleware. AMGA provides access to meta data for files stored on the Grid, as well as a simplified general access to relational data stored in database systems. Design and implementation of AMGA was done in close collaboration with the very diverse EGEE user community to make sure all functionality, performance and security requirements were met. In particular, AMGA targets the needs of the high energy physics community to rapidly access very large amounts of metadata, as well as the needs for security of the biomedical community. AMGA therefore tightly integrates fine grained access control making use of a virtual organisation management system. In addition, it offers advanced federation and features to increase dependability, performance and data security.     

Performance metrics and ontologies for Grid workflows

Many Grid workflow middleware services require knowledge about the performance behavior of Grid applications/services in order to effectively select, compose, and execute workflows in dynamic and complex Grid systems. To provide performance information for building such knowledge, Grid workflow performance tools have to select, measure, and analyze various performance metrics of workflows. However, there is a lack of a comprehensive study of performance metrics which can be used to evaluate the performance of a workflow executed in the Grid. Moreover, given the complexity of both Grid systems and workflows, semantics of essential performance-related concepts and relationships, and associated performance data in Grid workflows should be well described. In this paper, we analyze performance metrics that performance monitoring and analysis tools should provide during the evaluation of the performance of Grid workflows. Performance metrics are associated with multiple levels of abstraction. We introduce an ontology for describing performance data of Grid workflows and illustrate how the ontology can be utilized for monitoring and analyzing the performance of Grid workflows.     

A grid workflow Quality-of-Service estimation based on resource availability prediction

Accurate estimation of workflow Quality of Service (QoS) enhances the efficiency of scheduling algorithms. The availability and performance variations of Grid computing resources have made this estimation a great challenge. Most workflow QoS estimation algorithms are based on static performance of resources. In this paper, based on resources availability prediction, we propose an algorithm called WQE for estimating the QoS of a Grid workflow. WQE consists of two phases: resource monitoring and analysis and workflow QoS computation. In the first phase, two prediction algorithms are proposed to stochastically predict the availability state of resources. In the second phase, the QoS of each activity is estimated based on the host availability prediction result. The QoS of basic structures is computed by aggregating the QoS of their operands. Using a tree structure corresponding to the workflow, the QoS of basic structures is used to compute the total QoS of the workflow. The simulation results on Notre Dame University trace showed that the proposed method has higher estimation accuracy in comparison with HEFT.     

A standards-based interoperable single sign-on framework in ARC Grid middleware

Security infrastructure is one of the most challenging tasks in the development, integration and deployment of Grid middlewares. Even though the Grid community addresses the security issue through public key infrastructures (PKI) to support mutual authentication using X.509 certificates, maintaining X.509 credentials is not that easy for non-IT-experts, and has proved to be an obstacle for a more wide deployment of Grid technologies. The identity federation is an increasingly popular technology that can facilitate cross-domain single sign-on without requiring the users to maintain any credentials additional to their own institutional accounts. We believe that utilizing identity federation for Grid middlewares is a promising path for the Grid technology to get more widely used. This paper describes a single sign-on infrastructure developed as a part of the NorduGrid ARC (Advanced Resource Connector) Grid middleware. It adopts the identity federation standard (SAML), as well as other Web Service standards. It focuses on a single sign-on solution at the middleware level for users to access Grids by only using their frequently used accounts, without being bothered to maintain X.509 credentials. Users can use their username/password only to access Grids developed in ARC middleware, as well as access Grids developed in other middlewares that requires users to provide X.509 certificates. Moreover, the single sign-on for workflow-like Grid applications (in which intermediate entities act on behalf of users) is also supported. As an important aspect of single sign-on, authorization is also considered by implementing an attribute-based authorization using SAML standard. In addition, the performance of single sign-on solution is measured. We identify performance limitations of security-related services inside this solution, and analyse the ways to avoid the limitations. To our knowledge, the work presented in this paper is the first evaluated implementation that utilizes identity federation for Grid usage on the middleware level.     

Scientific workflow orchestration interoperating HTC and HPC resources

In this work we describe our developments towards the provision of a unified access method to different types of computing infrastructures at the interoperation level. For that, we have developed a middleware suite which bridges not interoperable middleware stacks used for building distributed computing infrastructures, UNICORE and gLite. Our solution allows to transparently access and operate on HPC and HTC resources from a single interface. Using Kepler as workflow manager, we provide users with the needed integration of codes to create scientific workflows accessing both types of infrastructures.     

基于扩展计算网的分层网格工作流支撑环境

针对实现资源共享和协同工作的网格技术应用研究与发展现状，提出了分层网格工作流体系结构:在设计层利用网格工作流逻辑和网格工作流语义建模网格工作流，在运行层利用网格工作流调度引擎和网格工作流实施引擎完成任务的实施，在资源层通过网格中间件和运行层交互。为支持上述结构，在(扩展)工作流网和计算网基础上，定义了扩展计算网模型。最后以实例说明了分层网格工作流建模支撑环境的具体应用。     

基于.net的校园网格异构数据统一访问接口

针对开放网格服务架构-数据访问与接口中间件只面向JDBC的局限性,提出构建一个基于.net架构的异构数据统一访问接口中间件。采用XML Web Service技术跨平台性和ADO.net增强的数据库访问能力,既实现了接口实现的通用性与易扩展性,又实现了接口对数据访问、数据库连接、数据格式转换、数据传输、数据库集成等底层细节的屏蔽。测试结果表明,该接口中间件具有通用性、灵活性和可靠性。