Large scale distributed visualization on computational Grids: A review

Advances in science and engineering have put high demands on tools for high performance large-scale data exploration and analysis. Visualization is a powerful technology for analyzing data and presenting results. Todays science and engineering have benefited from state-of-the-art of Grid technologies and modern visualization systems. To visualize the large amount of data, rendering technologies are widely used to parallelize visualization tasks over distributed resources on computational Grids. It raises the necessity to balance the computational load and to minimize the network bandwidth requirements. This article explains in Grid environments how new approaches of visualization architecture and load-balancing algorithms address these challenges in a principled fashion. The Grid infrastructure that supports large scale distributed visualization is also introduced. Some typical visualization systems on Grids are referenced for discussions.     

Management of a parameter sweep for scientific applications on cluster environments

The GEOsciences Network (GEON, www.geongrid.org ) is a large‐scale collaborative cyberinfrastructure project involving information technology and geoscience researchers from multiple institutions. The GEON infrastructure provides portal, middleware, and data resources to facilitate scientific discovery for domain scientists using applications, tools, and services. It consists of both a service‐oriented Web/Grid framework and application toolkits, using the Web service and portlet programming model to represent applications. Based on those grid environments, we have developed the SYNSEIS (SYNthetic SEISmogram) tool within the GEON infrastructure to support personalized experiments in seismology. In this paper, we present an overview of SYNSEIS from a user point of view, and demonstrate how one can use a simple management scheme to perform a parameter sweep and distribute the work in computational resources, using a scientific application that was not specifically designed to perform parameter sweeps. The performance advantages to be gained by using this scheme with scientific codes for dealing with a large number of jobs on computational grids are very substantial. In particular, we identify the earthquake simulations in the SYNSEIS tool as an example application that can benefit from running jobs on computational resources and subsequently promote the sharing of computational resources among partner sites involved in the GEON project. Finally, we also discuss the parallel scaling behavior of our primary earthquake simulation application. Copyright © 2010 John Wiley & Sons, Ltd.     

Distributed data mining on grids: services, tools, and applications

Data mining algorithms are widely used today for the analysis of large corporate and scientific datasets stored in databases and data archives. Industry, science, and commerce fields often need to analyze very large datasets maintained over geographically distributed sites by using the computational power of distributed and parallel systems. The grid can play a significant role in providing an effective computational support for distributed knowledge discovery applications. For the development of data mining applications on grids we designed a system called Knowledge Grid. This paper describes the Knowledge Grid framework and presents the toolset provided by the Knowledge Grid for implementing distributed knowledge discovery. The paper discusses how to design and implement data mining applications by using the Knowledge Grid tools starting from searching grid resources, composing software and data components, and executing the resulting data mining process on a grid. Some performance results are also discussed.     

The Virtual Laboratory: a toolset to enable distributed molecular modelling for drug design on the World‐Wide Grid

Computational Grids are emerging as a new paradigm for sharing and aggregation of geographically distributed resources for solving large‐scale compute and data intensive problems in science, engineering and commerce. However, application development, resource management and scheduling in these environments is a complex undertaking. In this paper, we illustrate the development of a Virtual Laboratory environment by leveraging existing Grid technologies to enable molecular modelling for drug design on geographically distributed resources. It involves screening millions of compounds in the chemical database (CDB) against a protein target to identify those with potential use for drug design. We have used the Nimrod‐G parameter specification language to transform the existing molecular docking application into a parameter sweep application for executing on distributed systems. We have developed new tools for enabling access to ligand records/molecules in the CDB from remote resources. The Nimrod‐G resource broker along with molecule CDB data broker is used for scheduling and on‐demand processing of docking jobs on the World‐Wide Grid (WWG) resources. The results demonstrate the ease of use and power of the Nimrod‐G and virtual laboratory tools for grid computing. Copyright © 2003 John Wiley & Sons, Ltd.     

ISENGARD: an infrastructure for supporting e‐science and grid application development

Grid computing facilitates the aggregation and coordination of resources that are distributed across multiple administrative domains for large‐scale and complex e‐Science experiments. Writing, deploying, and testing grid applications over highly heterogeneous and distributed resources are complex and challenging. The process requires grid‐enabled programming tools that can handle the complexity and scale of the infrastructure. However, while a large amount of research has been undertaken into grid middleware, little work has been directed specifically at the area of grid application development tools. This paper presents the design and implementation of ISENGARD, an infrastructure for supporting e‐Science and grid application development. ISENGARD provides services, tools, and APIs that simplify grid software development. Copyright © 2010 John Wiley & Sons, Ltd.     

A Grid‐enabled problem‐solving environment for advanced reservoir uncertainty analysis

Uncertainty analysis is critical for conducting reservoir performance prediction. However, it is challenging because it relies on (1) massive modeling‐related, geographically distributed, terabyte, or even petabyte scale data sets (geoscience and engineering data), (2) needs to rapidly perform hundreds or thousands of flow simulations, being identical runs with different models calculating the impacts of various uncertainty factors, (3) an integrated, secure, and easy‐to‐use problem‐solving toolkit to assist uncertainty analysis. We leverage Grid computing technologies to address these challenges. We design and implement an integrated problem‐solving environment ResGrid to effectively improve reservoir uncertainty analysis. The ResGrid consists of data management, execution management, and a Grid portal. Data Grid tools, such as metadata, replica, and transfer services, are used to meet massive size and geographically distributed characteristics of data sets. Workflow, task farming, and resource allocation are used to support large‐scale computation. A Grid portal integrates the data management and the computation solution into a unified easy‐to‐use interface, enabling reservoir engineers to specify uncertainty factors of interest and perform large‐scale reservoir studies through a web browser. The ResGrid has been used in petroleum engineering. Copyright © 2008 John Wiley & Sons, Ltd.     

Distributed data mining on Agent Grid: Issues,platform and development toolkit

Centralized data mining techniques are widely used today for the analysis of large corporate and scientific data stored in databases. However, industry, science, and commerce fields often need to analyze very large datasets maintained over geographically distributed sites by using the computational power of distributed systems. The Grid can play a significant role in providing an effective computational infrastructure support for this kind of data mining. Similarly, the advent of multi-agent systems has brought us a new paradigm for the development of complex distributed applications. During the past decades, there have been several models and systems proposed to apply agent technology building distributed data mining (DDM). Through a combination of these two techniques, we investigated the critical issues to build DDM on Grid infrastructure and design an Agent Grid Intelligent Platform as a testbed. We also implement an integrated toolkit VAStudio for quickly developing agent-based DDM applications and compare its function with other systems.     

Resource brokering using a multi‐site resource allocation strategy for computational grids

Grid computing employs heterogeneous resources which may be installed on different platforms, hardware/software, computer architectures, and perhaps using different computer languages to solve large‐scale computational problems. As many more Grids are being developed worldwide, the number of multi‐institutional collaborations is growing rapidly. However, to realize Grid computing's full potential, it is expected that Grid participants must be able to share one another's resources. This paper presents a resource broker that employs the multi‐site resource allocation (MSRA) strategy and the dynamic domain‐based network information model that we propose to allocate Grid resources to submitted jobs, where the Grid resources may be dispersed at different sites, and owned and governed by different organizations or institutes. The jobs and resources may also belong to different clusters/sites. Resource statuses collected by the Ganglia, and network bandwidths gathered by the Network Weather Service, are both considered in the proposed scheduling approach. A dynamic domain‐based model for network information measurement is also proposed to choose the most appropriate resources that meet the jobs' execution requirements. Experimental results show that MSRA outperformed the other tested strategies. Copyright © 2010 John Wiley & Sons, Ltd.     

Speeding up systems biology simulations of biochemical pathways using condor

Systems biology is a scientific field that uses computational modelling to study biological and biochemical systems. The simulation and analysis of models of these systems typically explore behaviour over a wide range of parameter values; as such, they are usually characterised by the need for nontrivial amounts of computing power. Grid computing provides access to such computational resources. In previous research, we created the grid‐enabled biochemical networks simulation environment to attempt to speed up system biology simulations over a grid (the UK National Grid Service and ScotGrid). Following on from this work, we have created the simulation modelling of the epidermal growth factor receptor microtubule‐associated protein kinase pathway utility, a standalone simulation tool dedicated to the modelling and analysis of the epidermal growth factor receptor microtubule‐associated protein kinase pathway. This builds on experiences from biochemical networks simulation environment by decoupling the simulation modelling elements from the Grid middleware. This new utility enables us to interface with different grid technologies. This paper therefore describes the new SIMAP utility and an empirical investigation of its performance when deployed over a desktop grid based on the high throughput computing middleware Condor. We present our results based on a case study with a model of the mammalian ErbB signalling pathway, a pathway strongly linked to cancer. Copyright © 2013 John Wiley & Sons, Ltd.     

GridBLAST: a Globus‐based high‐throughput implementation of BLAST in a Grid computing framework

Improvements in the performance of processors and networks have made it feasible to treat collections of workstations, servers, clusters and supercomputers as integrated computing resources or Grids. However, the very heterogeneity that is the strength of computational and data Grids can also make application development for such an environment extremely difficult. Application development in a Grid computing environment faces significant challenges in the form of problem granularity, latency and bandwidth issues as well as job scheduling. Currently existing Grid technologies limit the development of Grid applications to certain classes, namely, embarrassingly parallel, hierarchical parallelism, work flow and database applications. Of all these classes, embarrassingly parallel applications are the easiest to develop in a Grid computing framework. The work presented here deals with creating a Grid‐enabled, high‐throughput, standalone version of a bioinformatics application, BLAST, using Globus as the Grid middleware. BLAST is a sequence alignment and search technique that is embarrassingly parallel in nature and thus amenable to adaptation to a Grid environment. A detailed methodology for creating the Grid‐enabled application is presented, which can be used as a template for the development of similar applications. The application has been tested on a ‘mini‐Grid’ testbed and the results presented here show that for large problem sizes, a distributed, Grid‐enabled version can help in significantly reducing execution times. Copyright © 2005 John Wiley & Sons, Ltd.     

Campus Grids Meet Applications: Modeling, Metascheduling and Integration

Air Quality Forecasting (AQF) is a new discipline that attempts to reliably predict atmospheric pollution. An AQF application has complex workflows and in order to produce timely and reliable forecast results, each execution requires access to diverse and distributed computational and storage resources. Deploying AQF on Grids is one option to satisfy such needs, but requires the related Grid middleware to support automated workflow scheduling and execution on Grid resources. In this paper, we analyze the challenges in deploying an AQF application in a campus Grid environment and present our current efforts to develop a general solution for Grid-enabling scientific workflow applications in the GRACCE project. In GRACCE, an application’s workflow is described using GAMDL, a powerful dataflow language for describing application logic. The GRACCE metascheduling architecture provides the functionalities required for co-allocating Grid resources for workflow tasks, scheduling the workflows and monitoring their execution. By providing an integrated framework for modeling and metascheduling scientific workflow applications on Grid resources, we make it easy to build a customized environment with end-to-end support for application Grid deployment, from the management of an application and its dataset, to the automatic execution and analysis of its results.The work has been performed as part of the University of Houston’s Sun Microsystems Center of Excellence in Geosciences [38].     

A Science Driven Production Cyberinfrastructure��the Open Science Grid

This article describes the Open Science Grid, a large distributed computational infrastructure in the United States which supports many different high-throughput scientific applications, and partners (federates) with other infrastructures nationally and internationally to form multi-domain integrated distributed systems for science. The Open Science Grid consortium not only provides services and software to an increasingly diverse set of scientific communities, but also fosters a collaborative team of practitioners and researchers who use, support and advance the state of the art in large-scale distributed computing. The scale of the infrastructure can be expressed by the daily throughput of around seven hundred thousand jobs, just under a million hours of computing, a million file transfers, and half a petabyte of data movement. In this paper we introduce and reflect on some of the OSG capabilities, usage and activities.     

网格环境下分层并行多群体协作PSO框架设计与实现

针对利用广域范围内的计算资源参与PSO执行,从而提高工程最优化问题计算效率并降低计算成本,提出一个网格环境下分层并行多群体协作PSO（G-LPMCPSO）框架。首先给出一个适应负载不均衡和计算资源异构网格环境下的并行多群体协作PSO（PMCPSO）算法;然后着重阐述了如何利用标准的网格技术和PMCPSO算法设计并实现G-LPMCPSO框架,该框架具有一个扩展的GridRPC API用于隐藏网格环境的复杂性和一个元任务调度器用于无缝的资源发现和选取;最后,根据理论分析及实验结果,证明利用网格技术及PMCPSO可以提供一个可靠的框架用于加速解决科学工程最优化问题。     

Workflow-Based Data Parallel Applications on the EGEE Production Grid Infrastructure

Setting up and deploying complex applications on a Grid infrastructure is still challenging and the programming models are rapidly evolving. Efficiently exploiting Grid parallelism is often not straight forward. In this paper, we report on the techniques used for deploying applications on the EGEE production Grid through four experiments coming from completely different scientific areas: nuclear fusion, astrophysics and medical imaging. These applications have in common the need for manipulating huge amounts of data and all are computationally intensive. All the cases studied show that the deployment of data intensive applications require the development of more or less elaborated application-level workload management systems on top of the gLite middleware to efficiently exploit the EGEE Grid resources. In particular, the adoption of high level workflow management systems eases the integration of large scale applications while exploiting Grid parallelism transparently. Different approaches for scientific workflow management are discussed. The MOTEUR workflow manager strategy to efficiently deal with complex data flows is more particularly detailed. Without requiring specific application development, it leads to very significant speed-ups.     

Grid programming: some indications where we are headed

Grid computing enables the development of large scientific applications on an unprecedented scale. Grid-aware applications, also called meta-applications or multi-disciplinary applications, make use of coupled computational resources that are not available at a single site. In this light, the Grids let scientists solve larger or new problems by pooling together resources that could not be coupled easily before. It is well known that the programmer’s productivity in designing and implementing efficient distributed/parallel applications on high-performance computers is still usually a very time-consuming task. Grid computing makes the situation worse. Consequently, the development of Grid programming environments that would enable programmers to efficiently exploit this technology is an important and hot research issue.

After an introduction on the main Grid programming issues, this paper will review the most important approaches/projects conducted in this field worldwide.     

网格环境下费用约束的科学工作流可靠调度算法

网格基础设施是目前科学工作流应用规划、部署和执行的主要支撑环境.然而由于网格资源的自治、动态及异构性,如何在保障用户QoS约束下有效调度科学工作流是一个研究热点.针对费用约束下的科学工作流调度问题,为了提高其执行的可靠性,本文使用随机服务模型描述资源节点的动态服务能力并考虑本地任务负载对资源执行性能的影响,给出一种资源可靠性的评估方法,在此基础上提出一种费用约束下的科学工作流可靠调度算法RSASW.仿真实验结果表明RSASW算法相对于GAIN3,GreedyTime-CD及PFAS算法,对工作流的执行具有很好的可靠性保障.     

Semantic services for grid-based, large-scale science

The process of large-scale science must evolve to facilitate the next steps of scientific discovery. Grid technology and semantic tools will be valuable in dealing with the complex multidisciplinary simulation and data environments that next-generation science will require. We envision semantic Web-like tools that automatically check the validity of sequences of composed operations and data, and automatically construct intermediate steps in a loosely specified sequence. These tools should also automatically construct sequences of operations that are consistent with a discipline model representing permitted relationships among simulation and analysis operations and data for particular disciplines, such as climatology or high-energy physics. These tools are called semantic services.     

The application hosting environment: Lightweight middleware for grid-based computational science

Grid computing is distributed computing performed transparently across multiple administrative domains. Grid middleware, which is meant to enable access to grid resources, is currently widely seen as being too heavyweight and, in consequence, unwieldy for general scientific use. Its heavyweight nature, especially on the client-side, has severely restricted the uptake of grid technology by computational scientists. In this paper, we describe the Application Hosting Environment (AHE) which we have developed to address some of these problems. The AHE is a lightweight, easily deployable environment designed to allow the scientist to quickly and easily run legacy applications on distributed grid resources. It provides a higher level abstraction of a grid than is offered by existing grid middleware schemes such as the Globus Toolkit. As a result, the computational scientist does not need to know the details of any particular underlying grid middleware and is isolated from any changes to it on the distributed resources. The functionality provided by the AHE is ‘application-centric’: applications are exposed as web services with a well-defined standards-compliant interface. This allows the computational scientist to start and manage application instances on a grid in a transparent manner, thus greatly simplifying the user experience. We describe how a range of computational science codes have been hosted within the AHE and how the design of the AHE allows us to implement complex workflows for deployment on grid infrastructure.