Virtualizing access to scientific applications with the Application Hosting Environment

The growing power and number of high performance computing resources made available through computational grids present major opportunities as well as a number of challenges to the user. At issue is how these resources can be accessed and how their power can be effectively exploited. In this paper we first present our views on the usability of contemporary high-performance computational resources. We introduce the concept of grid application virtualization as a solution to some of the problems with grid-based HPC usability. We then describe a middleware tool that we have developed to realize the virtualization of grid applications, the Application Hosting Environment (AHE), and describe the features of the new release, AHE 2.0, which provides access to a common platform of federated computational grid resources in standard and non-standard ways. Finally, we describe a case study showing how AHE supports clinical use of whole brain blood flow modelling in a routine and automated fashion.

Program summary

Program title: Application Hosting Environment 2.0Catalogue identifier: AEEJ_v1_0Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEEJ_v1_0.htmlProgram obtainable from: CPC Program Library, Queen's University, Belfast, N. IrelandLicensing provisions: GNU Public Licence, Version 2No. of lines in distributed program, including test data, etc.: not applicableNo. of bytes in distributed program, including test data, etc.: 1 685 603 766Distribution format: tar.gzProgramming language: Perl (server), Java (Client)Computer: x86Operating system: Linux (Server), Linux/Windows/MacOS (Client)RAM: 134 217 728 (server), 67 108 864 (client) bytesClassification: 6.5External routines: VirtualBox (server), Java (client)Nature of problem: The middleware that makes grid computing possible has been found by many users to be too unwieldy, and presents an obstacle to use rather than providing assistance [1,2]. Such problems are compounded when one attempts to harness the power of a grid, or a federation of different grids, rather than just a single resource on the grid.Solution method: To address the above problem, we have developed AHE, a lightweight interface, designed to simplify the process of running scientific codes on a grid of HPC and local resources. AHE does this by introducing a layer of middleware between the user and the grid, which encapsulates much of the complexity associated with launching grid applications.Unusual features: The server is distributed as a VirtualBox virtual machine. VirtualBox (http://www.virtualbox.org) must be downloaded and installed in order to run the AHE server virtual machine. Details of how to do this are given in the AHE 2.0 Quick Start Guide.Running time: Not applicableReferences:

• [1] 

  J. Chin, P.V. Coveney, Towards tractable toolkits for the grid: A plea for lightweight, useable middleware, NeSC Technical Report, 2004, http://nesc.ac.uk/technical_papers/UKeS-2004-01.pdf.

• [2] 

  P.V. Coveney, R.S. Saksena, S.J. Zasada, M. McKeown, S. Pickles, The Application Hosting Environment: Lightweight middleware for grid-based computational science, Computer Physics Communications 176 (2007) 406–418.

     

Fault-tolerant grid architecture and practice

Grid computing emerges as effective technologies to couple geographically dis-tributed resources and solve large-scale computational problems in wide area networks. The fault tolerance is a significant and complex issue in grid computing systems. Various techniques have been investigated to detect and correct faults in distributed computing systems. Unreliable fault detection is one of the most effective techniques. Globus as a grid middleware manages resources in a wide area network. The Globns fault detection service uses the well-known techniques basedon unreliable fault detectors to detect and report component failures. However, more powerful techniques are required to detect and correct both system-level and application-level faults in agrid system, and a convenient toolkit is also needed to maintain the consistency in the grid. Afault-tolerant grid platform (FTGP) based on an unreliable fault detector and the Globus faultdetection service is presented in this paper. The platform offers effective strategies in such threeaspects as grid key components, user tasks, and high-level applications.     

Customizable execution environments for evolutionary computation using BOINC + virtualization

Evolutionary algorithms (EAs) consume large amounts of computational resources, particularly when they are used to solve real-world problems that require complex fitness evaluations. Beside the lack of resources, scientists face another problem: the absence of the required expertise to adapt applications for parallel and distributed computing models. Moreover, the computing power of PCs is frequently underused at institutions, as desktops are usually devoted to administrative tasks. Therefore, the proposal in this work consists of providing a framework that allows researchers to massively deploy EA experiments by exploiting the computing power of their instituions’ PCs by setting up a Desktop Grid System based on the BOINC middleware. This paper presents a new model for running unmodified applications within BOINC with a web-based centralized management system for available resources. Thanks to this proposal, researchers can run scientific applications without modifying the application’s source code, and at the same time manage thousands of computers from a single web page. Summarizing, this model allows the creation of on-demand customized execution environments within BOINC that can be used to harness unused computational resources for complex computational experiments, such as EAs. To show the performance of this model, a real-world application of Genetic Programming was used and tested through a centrally-managed desktop grid infrastructure. Results show the feasibility of the approach that has allowed researchers to generate new solutions by means of an easy to use and manage distributed system.     

数据网格信息服务体系的研究与设计

网格信息服务是网格中间件的核心部分之一,提供各种分布式网格资源的静态与动态信息。现有的网格中间件几乎都是针对计算网格设计的,它们所提供的信息服务不能胜任数据网格的要求。论文主要对数据网格中的信息服务体系进行研究和设计。     

Grid Computing in China

Grid computing presents a new trend to distributed computation and Internet applications, which can construct a virtual single image of heterogeneous resources, provide uniform application interface and integrate widespread computational resources into super, ubiquitous and transparent aggregation. In the adoption of Grid computing, China, who is facing more resource heterogeneity and other specific demands, has put much effort on both research and practical utilization. In this paper, we introduce the major China Grid research projects and their perspective applications. First we give the overview of the four government-sponsored programs in Grid, namely the China National Grid, ChinaGrid, NSFC Grid, and ShanghaiGrid. Then we present six representative ongoing Grid systems in details, which are categorized into Grid middleware and Grid application. This paper provides the general picture of Grid computing in China, and shows the great efforts, devotion and confidence in China to use Grid technology to boost the society, economics and scientific research.     

Towards supporting multiple virtual private computing environments on computational Grids

Grid computing now becomes a practical computing paradigm and solution for distributed systems and applications. Currently increasing resources are involved in Grid environments and a large number of applications are running on computational Grids. Unfortunately Grid computing technologies are still far away from reach of inexperienced application users, e.g., computational scientists and engineers. A software layer is required to provide an easy interface of Grids to end users.To meet this requirement HEAVEN (Hosting European Application Virtual ENvironment) upperware is proposed to build on top of Grid middleware. This paper presents HEAVEN philosophy of virtual computing for Grids – a combinational idea of simulation and emulation approaches. The concept of Virtual Private Computing Environment (VPCE) is thereafter proposed and defined. The design and current implementation of HEAVEN upperware are discussed in detail. Use case of Ag2D application justifies the philosophy of HEAVEN virtual computing methodology and the design/implementation of HEAVEN upperware.     

REMUS: A Rerouting and Multiplexing System for Grid Connectivity Across Firewalls

The Grid provides unique opportunities for high-performance computing through distributed applications that execute over multiple remote resources. Participating institutions can form a virtual organization to maximize the utilization of collective resources as well as to facilitate collaborative projects. However, there are two design aspects in distributed environments like the Grid that can easily clash: security and resource sharing. It may be that resources are secure but are not entirely conducive to resource sharing, or networks are wide open for resource sharing but sacrifice security as a result. We developed REMUS, a rerouting and multiplexing system that provides a compromise through connection rerouting and wrappers. REMUS reroutes connections using proxies, ports and protocols that are already authorized across firewalls, avoiding the need to make new openings through the firewalls. We also encapsulate applications within wrappers, transparently rerouting the connections among Grid applications without modifying their programs. In this paper, we describe REMUS and the tests we conducted across firewalls using two Grid middleware case studies: Globus Toolkit 2.4 and Nimrod/G 3.0.     

Zorilla: a peer‐to‐peer middleware for real‐world distributed systems

The inherent complex nature of current distributed computing architectures hinders the widespread adoption of these systems for mainstream use. In general, users have access to a highly heterogeneous set of compute resources, which may include clusters, grids, desktop grids, clouds, and other compute platforms. This heterogeneity is especially problematic when running parallel and distributed applications. Software is needed which easily combines as many resources as possible into one coherent computing platform. In this paper, we introduce Zorilla: peer‐to‐peer (P2P) middleware that creates a single distributed environment from any available set of compute resources. Zorilla imposes minimal requirements on the resource used, is platform independent, and does not rely on central components. In addition to providing functionality on bare resources, Zorilla can exploit locally available middleware. Zorilla explicitly supports distributed and parallel applications, and allows resources from multiple sites to cooperate in a single computation. Zorilla makes extensive use of both virtualization and P2P techniques. We will demonstrate how virtualization and P2P combine into a simple design, while enhancing functionality and ease of use. Together, these techniques bring our goal a step closer: transparent, easy use of resources, even on very heterogeneous distributed systems. Copyright © 2011 John Wiley & Sons, Ltd.     

Service-oriented middleware for distributed data mining on the grid

Distribution of data and computation allows for solving larger problems and executing applications that are distributed in nature. The grid is a distributed computing infrastructure that enables coordinated resource sharing within dynamic organizations consisting of individuals, institutions, and resources. The grid extends the distributed and parallel computing paradigms allowing for resource negotiation and dynamical allocation, heterogeneity, open protocols, and services. Grid environments can be used both for compute-intensive tasks and data intensive applications by exploiting their resources, services, and data access mechanisms. Data mining algorithms and knowledge discovery processes are both compute and data intensive, therefore the grid can offer a computing and data management infrastructure for supporting decentralized and parallel data analysis. This paper discusses how grid computing can be used to support distributed data mining. Research activities in grid-based data mining and some challenges in this area are presented along with some promising future directions for developing grid-based distributed data mining.     

QoS in grid computing

Grid computing is already a mainstream paradigm for resource-intensive scientific applications, but it also promises to become the future model for enterprise applications. The grid enables resource sharing and dynamic allocation of computational resources, thus increasing access to distributed data, promoting operational flexibility and collaboration, and allowing service providers to scale efficiently to meet variable demands. Large-scale grids are complex systems composed of thousands of components from disjoined domains. Planning the capacity to guarantee quality of service (QoS) in such environments is a challenge because global service-level agreements (SLAs) depend on local SLAs. We provide a motivating example for grid computing in an enterprise environment and then discuss the how resource allocation affects SLAs.     

Distributed data mining on grids: services, tools, and applications

Data mining algorithms are widely used today for the analysis of large corporate and scientific datasets stored in databases and data archives. Industry, science, and commerce fields often need to analyze very large datasets maintained over geographically distributed sites by using the computational power of distributed and parallel systems. The grid can play a significant role in providing an effective computational support for distributed knowledge discovery applications. For the development of data mining applications on grids we designed a system called Knowledge Grid. This paper describes the Knowledge Grid framework and presents the toolset provided by the Knowledge Grid for implementing distributed knowledge discovery. The paper discusses how to design and implement data mining applications by using the Knowledge Grid tools starting from searching grid resources, composing software and data components, and executing the resulting data mining process on a grid. Some performance results are also discussed.     

一种新的网格环境模型——TGrid Model

在分析了现有网格环境不足的基础上，提出一种新的网格环境模型——基于树形结构的网格体系与环境TGrid，支持高性能计算，面向主题的资源共享和新一代的需求建模。它以树结构来组织网格节点和集成各种资源，实现了自底向上、多级、面向需求的资源抽象和多种资源融合。而且树型结构符合自然层次组织关系，容易实现网格系统的层次化管理，有利于减轻中心节点的负载和实现大规模应用的负载平衡，提高资源查找效率。同时，TGrid以虚拟资源的形式实现网格资源的共享，利用分布式JVM（TJVM）虚拟网格节点上CPU和主存资源，利用多数据库中间件（TDOD）实现数据库级资源集成和共享，利用Globus网格服务（GService）实现其他软件和数据资源共享。该树型网格为日益增长的网格应用的需求提供了新的解决方案。     

Grid Portal的原理和实现

网格门户(Grid Portal)是为科学领域的用户提供Web方式访问网格计算资源的WWW应用。Grid Portal底层采用Globus Toolkit中间件, 为授权用户访问网格资源提供便利,并提供增强的网格服务功能。采用Grid Portal能够大大拓展网格的应用范围。该文分析了Grid Portal的结 构和运作原理,并介绍了采用Gridport技术开发网格门户的经验。     

A Single Sign-On Infrastructure for Science Gateways on a Use Case for Structural Bioinformatics

Structural bioinformatics applies computational methods to analyze and model three-dimensional molecular structures. There is a huge number of applications available to work with structural data on large scale. Using these tools on distributed computing infrastructures (DCIs), however, is often complicated due to a lack of suitable interfaces. The MoSGrid (Molecular Simulation Grid) science gateway provides an intuitive user interface to several widely-used applications for structural bioinformatics, molecular modeling, and quantum chemistry. It ensures the confidentiality, integrity, and availability of data via a granular security concept, which covers all layers of the infrastructure. The security concept applies SAML (Security Assertion Markup Language) and allows trust delegation from the user interface layer across the high-level middleware layer and the Grid middleware layer down to the HPC facilities. SAML assertions had to be integrated into the MoSGrid infrastructure in several places: the workflow-enabled Grid portal WS-PGRADE (Web Services Parallel Grid Runtime and Developer Environment), the gUSE (Grid User Support Environment) DCI services, and the cloud file system XtreemFS. The presented security infrastructure allows a single sign-on process to all involved DCI components and, therefore, lowers the hurdle for users to utilize large HPC infrastructures for structural bioinformatics.     

Grid programming: some indications where we are headed

Grid computing enables the development of large scientific applications on an unprecedented scale. Grid-aware applications, also called meta-applications or multi-disciplinary applications, make use of coupled computational resources that are not available at a single site. In this light, the Grids let scientists solve larger or new problems by pooling together resources that could not be coupled easily before. It is well known that the programmer’s productivity in designing and implementing efficient distributed/parallel applications on high-performance computers is still usually a very time-consuming task. Grid computing makes the situation worse. Consequently, the development of Grid programming environments that would enable programmers to efficiently exploit this technology is an important and hot research issue.

After an introduction on the main Grid programming issues, this paper will review the most important approaches/projects conducted in this field worldwide.     

ICENI: Optimisation of component applications within a Grid environment

Effective exploitation of Computational Grids can only be achieved when applications are fully integrated with the Grid middleware and the underlying computational resources. Fundamental to this exploitation is information. Information about the structure and behaviour of the application, the capability of the computational and networking resources, and the availability and access to these resources by an individual, a group or an organisation.

In this paper we describe Imperial College e-Science Networked Infrastructure (ICENI), a Grid middleware framework developed within the London e-Science Centre. ICENI is a platform-independent framework that uses open and extensible XML derived protocols, within a framework built using Java and Jini, to explore effective application execution upon distributed federated resources. We match a high-level application specification, defined as a network of components, to an optimal combination of the currently available component implementations within our Grid environment, by using composite performance models. We demonstrate the effectiveness of this architecture through the high-level specification and solution of a set of linear equations by automatic and selection of optimal resources and implementations.     

Virtual workflow system for distributed collaborative scientific applications on Grids

Grid computing has become an effective computing technique in recent years. This paper develops a virtual workflow system to construct distributed collaborative applications for Grid users. The virtual workflow system consists three levels: abstract workflow system, translator and concrete workflow system. The research highlight of the implementation is that this workflow system is developed based on CORBA and Unicore Grid middleware. Furthermore, this implementation can support legacy application developed with Parco and C++ codes. This virtual workflow system can provide efficient GUI for users to organize distributed scientific collaborative applications and execute them on Grid resources. We present the design, implementation, and evaluation of this virtual workflow system in the paper.     

Studying protein folding on the Grid: experiences using CHARMM on NPACI resources under Legion

One benefit of a computational Grid is the ability to run high‐performance applications over distributed resources simply and securely. We demonstrated this benefit with an experiment in which we studied the protein‐folding process with the CHARMM molecular simulation package over a Grid managed by Legion, a Grid operating system. High‐performance applications can take advantage of Grid resources if the Grid operating system provides both low‐level functionality as well as high‐level services. We describe the nature of services provided by Legion for high‐performance applications. Our experiences indicate that human factors continue to play a crucial role in the configuration of Grid resources, underlying resources can be problematic, Grid services must tolerate underlying problems or inform the user, and high‐level services must continue to evolve to meet user requirements. Our experiment not only helped a scientist perform an important study, but also showed the viability of an integrated approach such as Legion's for managing a Grid. Copyright © 2004 John Wiley & Sons, Ltd.     

基于扩展计算网的分层网格工作流支撑环境

针对实现资源共享和协同工作的网格技术应用研究与发展现状，提出了分层网格工作流体系结构:在设计层利用网格工作流逻辑和网格工作流语义建模网格工作流，在运行层利用网格工作流调度引擎和网格工作流实施引擎完成任务的实施，在资源层通过网格中间件和运行层交互。为支持上述结构，在(扩展)工作流网和计算网基础上，定义了扩展计算网模型。最后以实例说明了分层网格工作流建模支撑环境的具体应用。     

Speeding up systems biology simulations of biochemical pathways using condor

Systems biology is a scientific field that uses computational modelling to study biological and biochemical systems. The simulation and analysis of models of these systems typically explore behaviour over a wide range of parameter values; as such, they are usually characterised by the need for nontrivial amounts of computing power. Grid computing provides access to such computational resources. In previous research, we created the grid‐enabled biochemical networks simulation environment to attempt to speed up system biology simulations over a grid (the UK National Grid Service and ScotGrid). Following on from this work, we have created the simulation modelling of the epidermal growth factor receptor microtubule‐associated protein kinase pathway utility, a standalone simulation tool dedicated to the modelling and analysis of the epidermal growth factor receptor microtubule‐associated protein kinase pathway. This builds on experiences from biochemical networks simulation environment by decoupling the simulation modelling elements from the Grid middleware. This new utility enables us to interface with different grid technologies. This paper therefore describes the new SIMAP utility and an empirical investigation of its performance when deployed over a desktop grid based on the high throughput computing middleware Condor. We present our results based on a case study with a model of the mammalian ErbB signalling pathway, a pathway strongly linked to cancer. Copyright © 2013 John Wiley & Sons, Ltd.