The Virtual Laboratory: a toolset to enable distributed molecular modelling for drug design on the World‐Wide Grid

Computational Grids are emerging as a new paradigm for sharing and aggregation of geographically distributed resources for solving large‐scale compute and data intensive problems in science, engineering and commerce. However, application development, resource management and scheduling in these environments is a complex undertaking. In this paper, we illustrate the development of a Virtual Laboratory environment by leveraging existing Grid technologies to enable molecular modelling for drug design on geographically distributed resources. It involves screening millions of compounds in the chemical database (CDB) against a protein target to identify those with potential use for drug design. We have used the Nimrod‐G parameter specification language to transform the existing molecular docking application into a parameter sweep application for executing on distributed systems. We have developed new tools for enabling access to ligand records/molecules in the CDB from remote resources. The Nimrod‐G resource broker along with molecule CDB data broker is used for scheduling and on‐demand processing of docking jobs on the World‐Wide Grid (WWG) resources. The results demonstrate the ease of use and power of the Nimrod‐G and virtual laboratory tools for grid computing. Copyright © 2003 John Wiley & Sons, Ltd.     

The Legion support for advanced parameter-space studies on a grid

Parameter-space (p-space) studies involve running a single application several times with different parameter sets. Since the jobs are mutually independent, many computing resources can be recruited to conduct an entire study in a distributed manner. The p-space studies are attractive applications for grids, which are networked collections of computing and other resources. Legion is a grid infrastructure that facilitates the secure and easy use of heterogeneous, geographically distributed resources by providing the illusion of a single virtual machine from those resources. Legion provides tools and services that support advanced p-space studies, i.e., studies that make complex demands such as transparent access to distributed files, fault-tolerance and security. We demonstrate these benefits with a protein-folding experiment in which a molecular simulation package was run over a grid managed by Legion.     

网格局部性及其优化研究

数据网格已逐步在科学研究领域得到应用.提高数据网格的性能以适应分布式数据管理已经成为研究数据网格的一个热点.提出了网格局部性的概念,分析了网格局部性对数据网格性能的影响,并从增强网格局部性的角度对数据网格的性能进行优化,提出了综合跳一扩散副本替换策略(jump-DRP)和参考生物外激素的任务调度策略(JARIP).实验结果表明,考虑了网格局部性因素的jump-DRP与JARIP的策略组合提高了网格平台的任务处理性能,并对各类应用背景及任务的复杂程度具有鲁棒性.     

Management of a parameter sweep for scientific applications on cluster environments

The GEOsciences Network (GEON, www.geongrid.org ) is a large‐scale collaborative cyberinfrastructure project involving information technology and geoscience researchers from multiple institutions. The GEON infrastructure provides portal, middleware, and data resources to facilitate scientific discovery for domain scientists using applications, tools, and services. It consists of both a service‐oriented Web/Grid framework and application toolkits, using the Web service and portlet programming model to represent applications. Based on those grid environments, we have developed the SYNSEIS (SYNthetic SEISmogram) tool within the GEON infrastructure to support personalized experiments in seismology. In this paper, we present an overview of SYNSEIS from a user point of view, and demonstrate how one can use a simple management scheme to perform a parameter sweep and distribute the work in computational resources, using a scientific application that was not specifically designed to perform parameter sweeps. The performance advantages to be gained by using this scheme with scientific codes for dealing with a large number of jobs on computational grids are very substantial. In particular, we identify the earthquake simulations in the SYNSEIS tool as an example application that can benefit from running jobs on computational resources and subsequently promote the sharing of computational resources among partner sites involved in the GEON project. Finally, we also discuss the parallel scaling behavior of our primary earthquake simulation application. Copyright © 2010 John Wiley & Sons, Ltd.     

Life sciences grid in EUROGRID and GRIP projects

In this paper we describe deployment of most important life sciences applications on the grid. The build grid is heterogenous and consist of systems of different architecture as well as operating systems and various middleware. We have used UNICORE infrastructure as framework for development dedicated user interface to the number of existing computational chemistry codes and molecular biology databases. Developed solution allows for access to the resources provided with UNICORE as well as Globus with exactly the same interface which gives access to the general grid functionality such as single login, job submission and control mechanism. Jarosław Wypychowski: He is a student at the Faculty of Mathematics and Computer Science, Warsaw University, Poland. He is involved in the development of grid tools. He has been working as programmer in the private company. Jarosław Pytliński, M.Sc.: He received his M.Sc. in 2002 from Department of Mathematic and Computer Science of Nicolaus Copernicus University in Torun. His thesis on “Quantum Chemistry Computations in Grid Environment” was distincted in XIX Polish Contest for the best M.Sc. Thesis of Computer Science. He also worked in Laboratory of High Performance Systems at UCI, Torun. His interests are Artificial Intelligence and GRID technology. Łukasz Skorwider, M.Sc.: He is programmer in the private pharmaceutical company. He obtained M.Sc. degree from the Faculty of Mathematics and Computer Science N. Copernicus University. As graduate student he was involved in the development of grid tools for drug design. His private and professional interest is Internet technology. Mirosław Nazaruk, M.Sc.: He is a senior computer and network administrator at ICM Warsaw University. He provides professional support for the users of the high performance facilities located at the ICM. He obtained M.Sc. in Computer Science from Warsaw University in 1991. Before joining ICM, he was a member of technical staff at Institute of Applied Mathematics, Warsaw University. Krzysztof Benedyczak: He is a student at the Faculty of Mathematics and Computer Science, N. Copernicus University, Torun, Poland. He is involved in the development of grid tools. Michał Wroński: He is a student at the Faculty of Mathematics and Computer Science, N. Copernicus University, Torun, Poland. He is involved in the development of grid tools. Piotr Bała, Ph.D.: He is an adiunkt at Faculty of Mathematics and Computer Science N. Copernicus University, Torun, Poland, and tightly cooperates with ICM, Warsaw University. He obtained Ph.D. in Physics in 1993 in Institute of Physics, N. Copernicus University and in 2000 habilitation in physics. From 2001 he was appointed director of Laboratory of Parallel and Distributed Processing at Faculty of Mathematics, N. Copernicus University. His main research interest is development and application of Quantum-Classical Molecular Dynamics and Approximated Valence Bond method to study of enzymatic reactions in biological systems. In the last few years, he has been involved in development of parallel and grid tools for large scale scientific applications.     

Design of a grid service-based platform for in silico protein-ligand screenings

Grid computing offers the powerful alternative of sharing resources on a worldwide scale, across different institutions to run computationally intensive, scientific applications without the need for a centralized supercomputer. Much effort has been put into development of software that deploys legacy applications on a grid-based infrastructure and efficiently uses available resources. One field that can benefit greatly from the use of grid resources is that of drug discovery since molecular docking simulations are an integral part of the discovery process. In this paper, we present a scalable, reusable platform to choreograph large virtual screening experiments over a computational grid using the molecular docking simulation software DOCK. Software components are applied on multiple levels to create automated workflows consisting of input data delivery, job scheduling, status query, and collection of output to be displayed in a manageable fashion for further analysis. This was achieved using Opal OP to wrap the DOCK application as a grid service and PERL for data manipulation purposes, alleviating the requirement for extensive knowledge of grid infrastructure. With the platform in place, a screening of the ZINC 2,066,906 compound "drug-like" subset database against an enzyme's catalytic site was successfully performed using the MPI version of DOCK 5.4 on the PRAGMA grid testbed. The screening required 11.56 days laboratory time and utilized 200 processors over 7 clusters.     

Intelligent self‐adaptive resources selection for grid applications

Grid computing is considered a promising trend, which enables the sharing of a wide variety of computational and storage resources geographically distributed. Despite the advantages of such paradigm, several problems have emerged during the last decade; most of them caused by an inefficient utilization of grid resources. The present contribution proposes an approach to improve the grid resources selection process. An optimization model for choosing grid resources in an intelligent way has been designed. A mathematical formulation to monitor the resources efficiency has also been established. Furthermore, the model provides a self‐adaptive capability to grid applications, enhancing them for dealing with the changing environmental conditions. The model applies an artificial intelligence algorithm for ensuring an efficient selection. In particular, three different versions have been implemented. Each of them uses a different algorithm. Finally, during the evaluation phase of the model, the experimental tests were performed in a real grid infrastructure. The results show that the model improves the infrastructure throughput, by increasing the finished tasks rate and by reducing the applications execution time. Copyright © 2014 John Wiley & Sons, Ltd.     

JEL: unified resource tracking for parallel and distributed applications

When parallel applications are run in large‐scale distributed environments, such as grids, peer‐to‐peer (P2P) systems, and clouds, the set of resources used can change dynamically as machines crash, reservations end, and new resources become available. It is vital for applications to respond to these changes. Therefore, it is necessary to keep track of the available resources—a problem which is known to be notoriously difficult. In this article we argue that resource tracking must be provided as the standard functionality in the lower parts of the software stack. We propose a general solution to resource tracking: the Join–Elect–Leave (JEL) model. JEL provides unified resource tracking for parallel and distributed applications across environments. JEL is a simple yet powerful model based on notifying when resources have Joined or Left the computation. We demonstrate that JEL is suitable for resource tracking in a wide variety of programming models, ranging from the fixed resource sets traditionally used in MPI‐1 to flexible grid‐oriented programming models. We compare several JEL implementations, and show these to perform and scale well in several real‐world scenarios involving grids, clouds and P2P systems applied concurrently, and wide‐area systems with failing resources. Using JEL, we have won the first prize in a number of international distributed computing competitions. Copyright © 2010 John Wiley & Sons, Ltd.     

A survey on approaches to gridification

The Grid shows itself as a globally distributed computing environment, in which hardware and software resources are virtualized to transparently provide applications with vast capabilities. Just like the electrical power grid, the Grid aims at offering a powerful yet easy‐to‐use computing infrastructure to which applications can be easily ‘plugged’ and efficiently executed. Unfortunately, it is still very difficult to Grid‐enable applications, since current tools force users to take into account many details when adapting applications to run on the Grid. In this paper, we survey some of the recent efforts in providing tools for easy gridification of applications and propose several taxonomies to identify approaches followed in the materialization of such tools. We conclude this paper by describing common features among the proposed approaches, and by pointing out open issues and future directions in the research and development of gridification methods. Copyright © 2007 John Wiley & Sons, Ltd.     

Neuroscience instrumentation and distributed analysis of brain activity data: a case for eScience on global Grids

The distribution of knowledge (by scientists) and data sources (advanced scientific instruments), and the need for large‐scale computational resources for analyzing massive scientific data are two major problems commonly observed in scientific disciplines. Two popular scientific disciplines of this nature are brain science and high‐energy physics. The analysis of brain‐activity data gathered from the MEG (magnetoencephalography) instrument is an important research topic in medical science since it helps doctors in identifying symptoms of diseases. The data needs to be analyzed exhaustively to efficiently diagnose and analyze brain functions and requires access to large‐scale computational resources. The potential platform for solving such resource intensive applications is the Grid. This paper presents the design and development of MEG data analysis system by leveraging Grid technologies, primarily Nimrod‐G, Gridbus, and Globus. It describes the composition of the neuroscience (brain‐activity analysis) application as parameter‐sweep application and its on‐demand deployment on global Grids for distributed execution. The results of economic‐based scheduling of analysis jobs for three different optimizations scenarios on the world‐wide Grid testbed resources are presented along with their graphical visualization. Copyright © 2005 John Wiley & Sons, Ltd.     

Development of Science Gateways Using QCG — Lessons Learned from the Deployment on Large Scale Distributed and HPC Infrastructures

Today, various Science Gateways created in close collaboration with scientific communities provide access to remote and distributed HPC, Grid and Cloud computing resources and large-scale storage facilities. However, as we have observed there are still many entry barriers for new users and various limitations for active scientists. In this paper we present our latest achievements and software solutions that significantly simplify the use of large scale and distributed computing. We describe several Science Gateways that have been successfully created with the help of our application tools and the QCG (Quality in Cloud and Grid) middleware, in particular Vine Toolkit, QCG-Portal and QCG-Now, and make the use of HPC, Grid and Cloud more straightforward and transparent. Additionally, we share the best practices and lessons learned after creating jointly with user communities many domain-specific Science Gateways, e.g. dedicated for physicists, medical scientists, chemists, engineers and external communities performing multi-scale simulations. As our deployed software solutions have reached recently a critical mass of active users in the PLGrid e-infrastructure in Poland, we also discuss in this paper how changing technologies, visual design and user experience could impact the way we should re-design Science Getaways or even develop new attractive tools, e.g. desktop or mobile-based applications in the future. Finally, we present information and statistics regarding the behaviour of users to help readers understand how new capabilities and functionalities may influence the growth of user interest in Science Gateways and HPC technologies.     

The application hosting environment: Lightweight middleware for grid-based computational science

Grid computing is distributed computing performed transparently across multiple administrative domains. Grid middleware, which is meant to enable access to grid resources, is currently widely seen as being too heavyweight and, in consequence, unwieldy for general scientific use. Its heavyweight nature, especially on the client-side, has severely restricted the uptake of grid technology by computational scientists. In this paper, we describe the Application Hosting Environment (AHE) which we have developed to address some of these problems. The AHE is a lightweight, easily deployable environment designed to allow the scientist to quickly and easily run legacy applications on distributed grid resources. It provides a higher level abstraction of a grid than is offered by existing grid middleware schemes such as the Globus Toolkit. As a result, the computational scientist does not need to know the details of any particular underlying grid middleware and is isolated from any changes to it on the distributed resources. The functionality provided by the AHE is ‘application-centric’: applications are exposed as web services with a well-defined standards-compliant interface. This allows the computational scientist to start and manage application instances on a grid in a transparent manner, thus greatly simplifying the user experience. We describe how a range of computational science codes have been hosted within the AHE and how the design of the AHE allows us to implement complex workflows for deployment on grid infrastructure.     

Data resource discovery model based on hybrid architecture in data grid environment

Today, the management of massive data collections draws much attention as data grids have been developed to deal with large computational problems and provide the opportunity for sharing geographically distributed resources for large‒scale data‒intensive applications. Therefore, finding an effective approach to discover data resources in order to promote better interactions between application communities or virtual organizations becomes a critical challenge. Traditional grid resource discovery models are mostly based on central and hierarchical architecture that can lead to bottlenecking with the expansion of the grid scale. Although the Peer‒to‒Peer (P2P) technique is integrated into the grid in order to improve the performance in recent years, each P2P structure still has drawbacks that require several compensatory strategies. In this paper, based on the unstructured super‒node‒based architecture from the P2P system, we design a structured logic resource tree in each domain in order to effectively alleviate the load on the super‒node, and we propose a query recording learning algorithm based on this hybrid architecture to reduce traffic in the network and greatly shorten the response time. The model and algorithm are validated by simulations and compared with the traditional super‒peer model and the flooding‒based approach. Copyright © 2014 John Wiley & Sons, Ltd.     

GAMESH: A grid architecture for scalable monitoring and enhanced dependable job scheduling

Grid computing is a largely adopted paradigm to federate geographically distributed data centers. Due to their size and complexity, grid systems are often affected by failures that may hinder the correct and timely execution of jobs, thus causing a non-negligible waste of computing resources. Despite the relevance of the problem, state-of-the-art management solutions for grid systems usually neglect the identification and handling of failures at runtime. Among the primary goals to be considered, we claim the need for novel approaches capable to achieve the objectives of scalable integration with efficient monitoring solutions and of fitting large and geographically distributed systems, where dynamic and configurable tradeoffs between overhead and targeted granularity are necessary. This paper proposes GAMESH, a Grid Architecture for scalable Monitoring and Enhanced dependable job ScHeduling. GAMESH is conceived as a completely distributed and highly efficient management infrastructure, concentrating on two crucial aspects for large-scale and multi-domain grid environments: (i) the scalable dissemination of monitoring data and (ii) the troubleshooting of job execution failures. GAMESH has been implemented and tested in a real deployment encompassing geographically distributed data centers across Europe. Experimental results show that GAMESH (i) enables the collection of measurements of both computing resources and conditions of task scheduling at geographically sparse sites, while imposing a limited overhead on the entire infrastructure, and (ii) provides a failure-aware scheduler able to improve the overall system performance, even in the presence of failures, by coordinating local job schedulers at multiple domains.     

Large scale distributed visualization on computational Grids: A review

Advances in science and engineering have put high demands on tools for high performance large-scale data exploration and analysis. Visualization is a powerful technology for analyzing data and presenting results. Todays science and engineering have benefited from state-of-the-art of Grid technologies and modern visualization systems. To visualize the large amount of data, rendering technologies are widely used to parallelize visualization tasks over distributed resources on computational Grids. It raises the necessity to balance the computational load and to minimize the network bandwidth requirements. This article explains in Grid environments how new approaches of visualization architecture and load-balancing algorithms address these challenges in a principled fashion. The Grid infrastructure that supports large scale distributed visualization is also introduced. Some typical visualization systems on Grids are referenced for discussions.     

Distributed computing practice for large‐scale science and engineering applications

It is generally accepted that the ability to develop large‐scale distributed applications has lagged seriously behind other developments in cyberinfrastructure. In this paper, we provide insight into how such applications have been developed and an understanding of why developing applications for distributed infrastructure is hard. Our approach is unique in the sense that it is centered around half a dozen existing scientific applications; we posit that these scientific applications are representative of the characteristics, requirements, as well as the challenges of the bulk of current distributed applications on production cyberinfrastructure (such as the US TeraGrid). We provide a novel and comprehensive analysis of such distributed scientific applications. Specifically, we survey existing models and methods for large‐scale distributed applications and identify commonalities, recurring structures, patterns and abstractions. We find that there are many ad hoc solutions employed to develop and execute distributed applications, which result in a lack of generality and the inability of distributed applications to be extensible and independent of infrastructure details. In our analysis, we introduce the notion of application vectors: a novel way of understanding the structure of distributed applications. Important contributions of this paper include identifying patterns that are derived from a wide range of real distributed applications, as well as an integrated approach to analyzing applications, programming systems and patterns, resulting in the ability to provide a critical assessment of the current practice of developing, deploying and executing distributed applications. Gaps and omissions in the state of the art are identified, and directions for future research are outlined. Copyright © 2012 John Wiley & Sons, Ltd.     

Scheduling parameter sweep applications on global Grids: a deadline and budget constrained cost–time optimization algorithm

Computational Grids and peer‐to‐peer (P2P) networks enable the sharing, selection, and aggregation of geographically distributed resources for solving large‐scale problems in science, engineering, and commerce. The management and composition of resources and services for scheduling applications, however, becomes a complex undertaking. We have proposed a computational economy framework for regulating the supply of and demand for resources and allocating them for applications based on the users' quality‐of‐service requirements. The framework requires economy‐driven deadline‐ and budget‐constrained (DBC) scheduling algorithms for allocating resources to application jobs in such a way that the users' requirements are met. In this paper, we propose a new scheduling algorithm, called the DBC cost–time optimization scheduling algorithm, that aims not only to optimize cost, but also time when possible. The performance of the cost–time optimization scheduling algorithm has been evaluated through extensive simulation and empirical studies for deploying parameter sweep applications on global Grids. Copyright © 2005 John Wiley & Sons, Ltd.     

云计算:系统实例与研究现状

针对云计算这样一个范畴综述了当前云计算所采用的技术,剖析其背后的技术含义以及当前云计算参与企业所采用的云计算实现方案.云计算包含两个方面的含义:一方面是底层构建的云计算平台基础设施,是用来构造上层应用程序的基础;另外一方面是构建在这个基础平台之上的云计算应用程序.主要是针对云计算的基础架构的研究与实现状况给出综述,对于云计算的应用也有所涉及.云计算有3个最基本的特征:第1个是基础设施架构在大规模的廉价服务器集群之上;第二是应用程序与底层服务协作开发,最大限度地利用资源;第3个是通过多个廉价服务器之间的冗余,通过软件获得高可用性.云计算达到了两个分布式计算的重要目标:可扩展性和高可用性.可扩展性表达了云计算能够无缝地扩展到大规模的集群之上,甚至包含数千个节点同时处理.高可用性代表了云计算能够容忍节点的错误,甚至有很大一部分节点发生失效也不会影响程序的正确运行.通过此文可以了解云计算的当前发展状况以及未来的研究趋势.     

A taxonomy of computer‐based simulations and its mapping to parallel and distributed systems simulation tools

In recent years, extensive research has been conducted in the area of simulation to model large complex systems and understand their behavior, especially in parallel and distributed systems. At the same time, a variety of design principles and approaches for computer‐based simulation have evolved. As a result, an increasing number of simulation tools have been designed and developed. Therefore, the aim of this paper is to develop a comprehensive taxonomy for design of computer‐based simulations, and apply this taxonomy to categorize and analyze various simulation tools for parallel and distributed systems. Copyright © 2004 John Wiley & Sons, Ltd.     

Configuring large‐scale storage using a middleware with machine learning

The proliferation of cloud services and other forms of service‐oriented computing continues to accelerate. Alongside this development is an ever‐increasing need for storage within the data centres that host these services. Management applications used by cloud providers to configure their infrastructure should ideally operate in terms of high‐level policy goals, and not burden administrators with the details presented by particular instances of storage systems. One common technology used by cloud providers is the Storage Area Network (SAN). Support for seamless scalability is engineered into SAN devices. However, SAN infrastructure has a very large parameter space: their optimal deployment is a difficult challenge, and subsequent management in cloud storage continues to be difficult. parindent = 10pt In this article, we discuss our work in SAN configuration middleware, which aims to provide users of large‐scale storage infrastructure such as cloud providers with tools to assist them in their management and evolution of heterogeneous SAN environments. We propose a middleware rather than a stand‐alone tool so that the middleware can be a proxy for interacting with, and informing, a central repository of SAN configurations. Storage system users can have their SAN configurations validated against a knowledge base of best practices that are contained within the central repository. Desensitized information is exported from local management applications to the repository, and the local middleware can subscribe to updates that proactively notify storage users should particular configurations be updated to be considered as sub‐optimal, or unsafe. Copyright © 2011 John Wiley & Sons, Ltd.