High-level abstractions for message-passing parallel programming

Large-scale scientific and engineering computation problems are usually complex and consequently the development of parallel programs for solving these problems is a difficult task. In this paper, we describe the graph-oriented programming (GOP) model and environment for building and evaluating parallel applications. The GOP model provides higher level abstractions for message-passing parallel programming and the software environment offers tools which can ease programmers for parallelizing, writing, and deploying scientific and engineering computing applications. We discuss the motivations and various issues in developing the model and the software environment, present the design of the system architecture and the components, and describe the evaluation of the environment implemented on top of MPI with a sample parallel scientific application program. With the support of the high-level abstractions provided by the proposed GOP environment, programming of parallel applications on various parallel architectures can be greatly simplified.     

A Decentralized Computational Infrastructure for Grid-Based Parallel Asynchronous Iterative Applications

Parallel asynchronous iterative algorithms relax synchronization and communication requirements, and can potentially extend Desktop Grids beyond embarrassingly parallel applications to support a broader class of parallel iterative applications. This paper presents the design and implementation of CometG, a decentralized (peer-to-peer) computational infrastructure that extends Desktop Grid environments to support these applications. CometG provides a decentralized and scalable tuple space, efficient communication and coordination support, and application-level abstractions that can be used to implement Desktop Grid applications based on parallel asynchronous iterative algorithms using the master-worker/BOT paradigm. The deployment and evaluations of CometG and a CometG-based application in a wide-area environment using the PlanetLab [7] test bed, as well as a campus network are presented.     

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.     

Dynamic Instrumentation, Performance Monitoring and Analysis of Grid Scientific Workflows

While existing work concentrates on developing QoS models of business workflows and Web services, few tools have been developed to support the monitoring and performance analysis of scientific workflows in Grids. This paper describes novel Grid services for dynamic instrumentation of Grid-based applications, performance monitoring and analysis of Grid scientific workflows. We describe a Grid dynamic instrumentation service that provides a widely accessible interface for other services and users to conduct the dynamic instrumentation of Grid applications during the runtime. We introduce a Grid performance analysis service for Grid scientific workflows. The analysis service utilizes various types of data including workflow graphs, monitoring data of resources, execution status of activities, and performance measurements obtained from the dynamic instrumentation of invoked applications, and provides a rich set of functionalities and features to support the online monitoring and performance analysis of scientific workflows. Workflows and their relevant information including performance metrics are stored and utilized for comparing the performance of constructs of different workflows and for supporting multi-workflow analysis. The work described in this paper is supported in part by the Austrian Science Fund as part of the Aurora Project under contract SFBF1104 and by the European Union through the IST-2002-511385 project K-WfGrid.     

Toward self-describing and workflow integrated Earth system models: A coupled atmosphere-ocean modeling system application

The complexity of Earth system models and their applications is increasing as a consequence of scientific advances, user demand, and the ongoing development of computing platforms, storage systems and distributed high-resolution observation networks. Multi-component Earth system models need to be redesigned to make interactions among model components and other applications external to the modeling system easier. To that end, the common component interfaces of Earth system models can be redesigned to increase interoperability between models and other applications such as various web services, data portals and science gateways. The models can be made self-describing so that the many configuration, build options and inputs of a simulation can be recorded. In this paper, we present a coupled modeling system that includes the proposed methodology to create self-describing models with common model component interfaces. The designed coupled atmosphere-ocean modeling system is also integrated into a scientific workflow system to simplify routine modeling tasks and relationships between these tasks and to demonstrate the enhanced interoperability between different technologies and components. Later on, the work environment is tested using a realistic Earth system modeling application. As can be seen through this example, a layered design for collecting provenance and metadata has the added benefit of documenting a run in far greater detail than before. In this way, it facilitates exploration and understanding of simulations and leads to possible reproducibility. In addition to designing self-describing Earth system models, the regular modeling tasks are also simplified and automated by using a scientific workflow which provides meaningful abstractions for the model, computing environment and provenance/metadata collection mechanisms. Our aim here is to solve a specific instance of a complex model integration problem by using a framework and scientific workflow approach together. The reader may also note that the methods presented in this paper might be also generalized to other types of Earth system models, leading to improved ease of use and flexibility. The initial results also show that the coupled atmosphere-ocean model, which is controlled by the designed workflow environment, is able to reproduce the Mediterranean Sea surface temperature when it is compared with the used CCSM3 initial and boundary conditions.     

Design and development of multidevice user interfaces through multiple logical descriptions

The increasing availability of new types of interaction platforms raises a number of issues for designers and developers. There is a need for new methods and tools to support development of nomadic applications, which can be accessed through a variety of devices. We present a solution, based on the use of three levels of abstractions, that allows designers to focus on the relevant logical aspects and avoid dealing with a plethora of low-level details. We have defined a number of transformations able to obtain user interfaces from such abstractions, taking into account the available platforms and their interaction modalities while preserving usability. The transformations are supported by an authoring tool, TERESA, which provides designers and developers with various levels of automatic support and several possibilities for tailoring such transformations to their needs.     

Region-based parallelization of irregular reductions on explicitly managed memory hierarchies

Multicore architectures are evolving with the promise of extreme performance for the classes of applications that require high performance and large bandwidth of memory. Irregular reduction is one of important computation patterns for many complex scientific applications, and it typically requires high performance and large bandwidth of memory. In this article, we propose region-based parallelization techniques for irregular reductions on multicore architectures with explicitly managed memory hierarchies. Managing memory hierarchy in software requires a lot of programming efforts and tends to be error-prone. The difficulties are even worse for applications with irregular data access patterns. To relieve the burden of memory management from programmers, we develop abstractions, particularly targeted to irregular reduction, for structuring parallel tasks, mapping the parallel tasks to processing units and scheduling data transfers between the memory hierarchies. Our framework employs iteration reordering based on regions of data along with dynamic scheduling of parallel tasks. We experimentally evaluate the effectiveness of our techniques for irregular reduction kernels on the Cell processor embedded in a Sony PlayStation3. Experimental results show the speedups of 8 to 14 on the six available SPEs.     

网格环境下费用约束的科学工作流可靠调度算法

网格基础设施是目前科学工作流应用规划、部署和执行的主要支撑环境.然而由于网格资源的自治、动态及异构性,如何在保障用户QoS约束下有效调度科学工作流是一个研究热点.针对费用约束下的科学工作流调度问题,为了提高其执行的可靠性,本文使用随机服务模型描述资源节点的动态服务能力并考虑本地任务负载对资源执行性能的影响,给出一种资源可靠性的评估方法,在此基础上提出一种费用约束下的科学工作流可靠调度算法RSASW.仿真实验结果表明RSASW算法相对于GAIN3,GreedyTime-CD及PFAS算法,对工作流的执行具有很好的可靠性保障.     

Scheduling DAG-based applications in multicluster environments with background workload using task duplication

Grids or multicluster computing environments are becoming increasingly popular to both scientific and commercial applications. Process scheduling remains a central issue to be effectively resolved in order to exploit the full potential that the grid or multicluster environment can offer. We use a directed acyclic graph (DAG) to model a process or an application where the nodes of the DAG represent the tasks of the process. Prior to the execution of a process in a multicluster environment, the tasks are required to be mapped onto the clusters. In this article, it is shown that the algorithm developed by He et al. [L. He, S.A. Jarvis, D.P. Spooner, D. Bacigalupo, G. Tan, and G.R. Nudd, Mapping DAG-based applications to multiclusters with background workload, Proceedings of the 2005 IEEE International Symposium on Cluster Computing and the Grid, Cardiff, 2005, pp 855–862.] for the multicluster DAG mapping problem can be significantly improved by incorporating the task duplication strategy. The proposed process scheduling algorithm has a time complexity O(| V|²(r+d+1)), where |V| represents the number of tasks; r, the number of clusters; and d, the maximum in-degree of tasks.     

A PTS-PGATS based approach for data-intensive scheduling in data grids

Grid computing is the combination of computer resources in a loosely coupled, heterogeneous, and geographically dispersed environment. Grid data are the data used in grid computing, which consists of large-scale data-intensive applications, producing and consuming huge amounts of data, distributed across a large number of machines. Data grid computing composes sets of independent tasks each of which require massive distributed data sets that may each be replicated on different resources. To reduce the completion time of the application and improve the performance of the grid, appropriate computing resources should be selected to execute the tasks and appropriate storage resources selected to serve the files required by the tasks. So the problem can be broken into two sub-problems: selection of storage resources and assignment of tasks to computing resources. This paper proposes a scheduler, which is broken into three parts that can run in parallel and uses both parallel tabu search and a parallel genetic algorithm. Finally, the proposed algorithm is evaluated by comparing it with other related algorithms, which target minimizing makespan. Simulation results show that the proposed approach can be a good choice for scheduling large data grid applications.     

Programming Grid Applications with GRID Superscalar

The aim of GRID superscalar is to reduce the development complexity of Grid applications to the minimum, in such a way that writing an application for a computational Grid may be as easy as writing a sequential application. Our assumption is that Grid applications would be in a lot of cases composed of tasks, most of them repetitive. The granularity of these tasks will be of the level of simulations or programs, and the data objects will be files. GRID superscalar allows application developers to write their application in a sequential fashion. The requirements to run that sequential application in a computational Grid are the specification of the interface of the tasks that should be run in the Grid, and, at some points, calls to the GRID superscalar interface functions and link with the run-time library.GRID superscalar provides an underlying run-time that is able to detect the inherent parallelism of the sequential application and performs concurrent task submission. In addition to a data-dependence analysis based on those input/output task parameters which are files, techniques such as file renaming and file locality are applied to increase the application performance. This paper presents the current GRID superscalar prototype based on Globus Toolkit 2.x, together with examples and performance evaluation of some benchmarks.     

A survey on approaches to gridification

The Grid shows itself as a globally distributed computing environment, in which hardware and software resources are virtualized to transparently provide applications with vast capabilities. Just like the electrical power grid, the Grid aims at offering a powerful yet easy‐to‐use computing infrastructure to which applications can be easily ‘plugged’ and efficiently executed. Unfortunately, it is still very difficult to Grid‐enable applications, since current tools force users to take into account many details when adapting applications to run on the Grid. In this paper, we survey some of the recent efforts in providing tools for easy gridification of applications and propose several taxonomies to identify approaches followed in the materialization of such tools. We conclude this paper by describing common features among the proposed approaches, and by pointing out open issues and future directions in the research and development of gridification methods. Copyright © 2007 John Wiley & Sons, Ltd.     

Visual Grid Workflow in Triana

In this paper, we describe the graphical abstractions for Grids and services that have been implemented within the Triana problem solving environment. We provide an overview of the ways in which Triana interacts with services (e.g., Web and P2P services) and then how we interact with core Grid components, such as resource managers and data management systems through the extensive use of the GridLab GAT interface. We describe in detail the GAT philosophy and implementation and then show how the various GAT primitives can be represented in an intuitive fashion within a Triana workflow. This approach, which we refer to as the Visual GAT, differs substantially from other approaches because we do not tie our implementation to any specific underlying Grid middleware technologies; rather, we base our implementation on application level requirements and model such primitives from a user’s perspective by hiding as much complexity as possible without undermining the core capabilities required. We provide a use case to demonstrate the Visual GAT implementation and show how legacy applications can seamlessly be distributed and integrated in a dynamic fashion within complex data-driven workflow scenarios.     

Grid Service Orchestration Using the Business Process Execution Language (BPEL)

Modern scientific applications often need to be distributed across Grids. Increasingly applications rely on services, such as job submission, data transfer or data portal services. We refer to such services as Grid services. While the invocation of Grid services could be hard coded in theory, scientific users want to orchestrate service invocations more flexibly. In enterprise applications, the orchestration of web services is achieved using emerging orchestration standards, most notably the Business Process Execution Language (BPEL). We describe our experience in orchestrating scientific workflows using BPEL. We have gained this experience during an extensive case study that orchestrates Grid services for the automation of a polymorph prediction application. Using this example, we explain the extent with which the BPEL language supports the definition of scientific workflows. We then describe the reliability, performance and scalability that can be achieved by executing a complex scientific workflow with ActiveBPEL, an industrial strength but freely available BPEL engine. *The work has been funded by the UK EPSRC through grants GR/R97207/01 (e-Materials) and GR/S90843/01 (OMII Managed Programme).     

Distributed data mining on grids: services, tools, and applications

Data mining algorithms are widely used today for the analysis of large corporate and scientific datasets stored in databases and data archives. Industry, science, and commerce fields often need to analyze very large datasets maintained over geographically distributed sites by using the computational power of distributed and parallel systems. The grid can play a significant role in providing an effective computational support for distributed knowledge discovery applications. For the development of data mining applications on grids we designed a system called Knowledge Grid. This paper describes the Knowledge Grid framework and presents the toolset provided by the Knowledge Grid for implementing distributed knowledge discovery. The paper discusses how to design and implement data mining applications by using the Knowledge Grid tools starting from searching grid resources, composing software and data components, and executing the resulting data mining process on a grid. Some performance results are also discussed.     

Simultaneous Scheduling of Replication and Computation for Data-Intensive Applications on the Grid

Managing large datasets has become one major application of Grids. Life science applications usually manage large databases that should be replicated to scale applications. The growing number of users and the simple access to Internet-based application has stressed Grid middleware. Such environment are thus asked to manage data and schedule computation tasks at the same time. These two important operations have to be tightly coupled. This paper presents an algorithm (Scheduling and Replication Algorithm, SRA) that combines data management and scheduling using a steady-state approach. Using a model of the platform, the number of requests as well as their distribution, the number and size of databases, we define a linear program to satisfy all the constraints at every level of the platform in steady-state. The solution of this linear program will give us a placement for the databases on the servers as well as providing, for each kind of job, the server on which they should be executed. Our theoretical results are validated using simulation and logs from a large life science application. This work was supported in part by the ACI GRID and Grid5000 projects of the French Department of Research.     

Trustworthy remote compiling services for grid-based scientific applications

Grid computing, which is characterized by large-scale sharing and collaboration of dynamic resources, is becoming an emerging computing platform on a global scale for data-intensive and computation-intensive scientific application. However, the complications of large-scale scientific computations and simulations harnessing massive computing resources are compounded by extensive heterogeneity in environments arising from “the Grid.” Scientists and engineers lack an intuitive grid-based compilation tool, which has contributed to the difficulty of exploiting these diverse resources and developing their applications on the grid. While manual configuration of various toolkits simplifying the end-to-end completion of a job is adequate for a computational grid with a limited number of nodes, the compilation procedure becomes inefficient for a computational grid with an increasing number of heterogeneous computational service providers. On the other hand, a global-scale computational grid is a potentially untrustworthy computing environment. How to take advantage of the potentially untrustworthy grid resources to provide trustworthy computational services for large-scale scientific applications is another critical issue. In this article, a remote compiling service for a heterogeneous computational grid is developed. In addition to running compilation tasks, the remote compiling service provides security enforcement and validation facilities, including intermediate value checking, secure source program submission, restricted compilation, and binary inspection, to support trustworthy compilation and execution of grid-based scientific applications. Overall, it is expected that our remote compiling services on the grid can tackle the heterogeneity problem of the grid and provide a secure, trustworthy, reliable, and state-of-the-art mechanism to develop grid-aware scientific applications.

Campus Grids Meet Applications: Modeling, Metascheduling and Integration

Air Quality Forecasting (AQF) is a new discipline that attempts to reliably predict atmospheric pollution. An AQF application has complex workflows and in order to produce timely and reliable forecast results, each execution requires access to diverse and distributed computational and storage resources. Deploying AQF on Grids is one option to satisfy such needs, but requires the related Grid middleware to support automated workflow scheduling and execution on Grid resources. In this paper, we analyze the challenges in deploying an AQF application in a campus Grid environment and present our current efforts to develop a general solution for Grid-enabling scientific workflow applications in the GRACCE project. In GRACCE, an application’s workflow is described using GAMDL, a powerful dataflow language for describing application logic. The GRACCE metascheduling architecture provides the functionalities required for co-allocating Grid resources for workflow tasks, scheduling the workflows and monitoring their execution. By providing an integrated framework for modeling and metascheduling scientific workflow applications on Grid resources, we make it easy to build a customized environment with end-to-end support for application Grid deployment, from the management of an application and its dataset, to the automatic execution and analysis of its results.The work has been performed as part of the University of Houston’s Sun Microsystems Center of Excellence in Geosciences [38].     

Studying protein folding on the Grid: experiences using CHARMM on NPACI resources under Legion

One benefit of a computational Grid is the ability to run high‐performance applications over distributed resources simply and securely. We demonstrated this benefit with an experiment in which we studied the protein‐folding process with the CHARMM molecular simulation package over a Grid managed by Legion, a Grid operating system. High‐performance applications can take advantage of Grid resources if the Grid operating system provides both low‐level functionality as well as high‐level services. We describe the nature of services provided by Legion for high‐performance applications. Our experiences indicate that human factors continue to play a crucial role in the configuration of Grid resources, underlying resources can be problematic, Grid services must tolerate underlying problems or inform the user, and high‐level services must continue to evolve to meet user requirements. Our experiment not only helped a scientist perform an important study, but also showed the viability of an integrated approach such as Legion's for managing a Grid. Copyright © 2004 John Wiley & Sons, Ltd.     

Analysis of Scheduling and Replica Optimisation Strategies for Data Grids Using OptorSim

Many current international scientific projects are based on large scale applications that are both computationally complex and require the management of large amounts of distributed data. Grid computing is fast emerging as the solution to the problems posed by these applications. To evaluate the impact of resource optimisation algorithms, simulation of the Grid environment can be used to achieve important performance results before any algorithms are deployed on the Grid. In this paper, we study the effects of various job scheduling and data replication strategies and compare them in a variety of Grid scenarios using several performance metrics. We use the Grid simulator , and base our simulations on a world-wide Grid testbed for data intensive high energy physics experiments. Our results show that scheduling algorithms which take into account both the file access cost of jobs and the workload of computing resources are the most effective at optimising computing and storage resources as well as improving the job throughput. The results also show that, in most cases, the economy-based replication strategies which we have developed improve the Grid performance under changing network loads.