A Taxonomy of Workflow Management Systems for Grid Computing

With the advent of Grid and application technologies, scientists and engineers are building more and more complex applications to manage and process large data sets, and execute scientific experiments on distributed resources. Such application scenarios require means for composing and executing complex workflows. Therefore, many efforts have been made towards the development of workflow management systems for Grid computing. In this paper, we propose a taxonomy that characterizes and classifies various approaches for building and executing workflows on Grids. We also survey several representative Grid workflow systems developed by various projects world-wide to demonstrate the comprehensiveness of the taxonomy. The taxonomy not only highlights the design and engineering similarities and differences of state-of-the-art in Grid workflow systems, but also identifies the areas that need further research.     

A semantic framework for automatic generation of computational workflows using distributed data and component catalogues

Computational workflows are a powerful paradigm to represent and manage complex applications, particularly in large-scale distributed scientific data analysis. Workflows represent application components that result in individual computations as well as their interdependences in terms of dataflow. Workflow systems use these representations to manage various aspects of workflow creation and execution for users, such as the automatic assignment of execution resources. This article describes an approach to automating a new aspect of the process: the selection of application components and data sources. We present a novel approach that enables users to specify varying degrees of detail and amount of constraints in a workflow request, including the specification of constraints on input, intermediate or output data in the workflow, abstract workflow component classes rather than specific component implementations, and generic reusable workflow templates that express a pre-defined combination of components. The algorithm elaborates the user request into a set of fully ground workflows with specific choices of data sources and codes to be used so that they can be submitted for mapping and execution. The algorithm searches through the space of possible candidate workflows by creating increasingly more specialized versions of the original template and eliminating candidates that violate constraints cumulated in the candidate workflow as components and data sources are selected. A novel feature of our approach is that it assumes a distributed architecture where data and component catalogues are separate from the workflow system. The algorithm explicitly poses queries to external catalogues, and therefore any reasoning regarding data or component properties is not assumed to occur within the workflow system. We describe our implementation of this approach in the Wings workflow system. This implementation uses the W3C Web Ontology Language and associated reasoners to implement the workflow system as well as the data and component catalogues. This research demonstrates the use of artificial intelligence techniques to support the kinds of automation envisioned by the scientific community for large-scale distributed scientific data analysis.     

The Charité Grid Portal: User-friendly and Secure Access to Grid-based Resources and Services

The Charité Grid Portal combines portal components from different groups and projects to provide domain researchers a gateway to Grid-based biomedical applications. Trusted users can securely access and employ Grid resources and services. In this paper, five portal components are presented: (1) The credential management administrates the user-credentials and authenticates them to the Grid. (2) The brain imaging data analysis (FSL) submits workflows to the Grid as part of Medical Image processing. (3) The integrated web services of Generic Workflow Execution Service (GWES) manage workflows executed by users in the Grid. (4) The data management component provides secure and efficient data management in a Grid environment, and enables high-speed data transports between user and Grid. (5) The lung sound analysis application provides twofold-pseudonymization before data-transferred to the Grid. The implementation as standardized portlets allows easy integration of specific components into different Grid portals such as the VO specific MediGRID and PneumoGrid portals.     

面向云计算的工作流技术

云计算的发展,对提高服务质量与压缩运行成本提出了新的要求.在此背景下,工作流技术被认为是一种较为优越的解决方案:从云计算用户的角度看,工作流提供了对复杂应用的抽象定义、灵活配置和自动化运行;从云计算服务提供者的角度看,工作流实现了任务的自动调度、资源的优化和管理.文章介绍工作流技术与云计算;阐述了云工作流产生的背景;深入剖析了云工作流的技术特征以及与其他工作流(业务工作流、网格工作流)的异同;列举了云工作流的4个实现案例,并对其进行比较.最后,在总结全文的同时,展望了云工作流技术的发展前景.     

Workflow View Driven Cross-Organizational Interoperability in a Web Service Environment

Workflow technology has recently been employed not only within businesses but also as a framework for implementing e-services over the Internet. Such e-services typically require collaborative enactment of workflows across multiple organizations. In this paper, we propose the use of workflow views as a fundamental support mechanism for the interoperability of multiple workflows across business organizations. We present a meta-model of workflow views and their semantics using a cross-organization workflow example based on a supply-chain e-service. We also formulate an interoperation model of workflow views and its consistency criteria. Finally, this paper presents an implementation of the model based on XML and contemporary Web services technologies, with adaptation to our E-ADOME workflow engine.     

基于构件的本体扩展工作流的构架研究与应用

几乎所有的行业都涉及工作流,用以协调任务之间的执行。由构件组成的工作流,可以提高系统的复用性,加快系统的开发和配置速度。但是构件之间句法语义的不同,决定了它们是不能直接组装的。论文提出基于本体扩展的工作流构件,使用本体来解决不同构件之间的元数据映射,并实现了基于构件的本体扩展工作流的框架,做出了具体实施。     

Optimizing Grid-Based Workflow Execution

Large-scale applications can be expressed as a set of tasks with data dependencies between them, also known as application workflows. Due to the scale and data processing requirements of these applications, they require Grid computing and storage resources. So far, the focus has been on developing easy to use interfaces for composing these workflows and finding an optimal mapping of tasks in the workflow to the Grid resources in order to minimize the completion time of the application. After this mapping is done, a workflow execution engine is required to run the workflow over the mapped resources. In this paper, we show that the performance of the workflow execution engine in executing the workflow can also be a critical factor in determining the workflow completion time. Using Condor as the workflow execution engine, we examine the various factors that affect the completion time of a fine granularity astronomy workflow. We show that changing the system parameters that influence these factors and restructuring the workflow can drastically reduce the completion time of this class of workflows. We also examine the effect on the optimizations developed for the astronomy application on a coarser granularity biology application. We were able to reduce the completion time of the Montage and the Tomography application workflows by 90% and 50%, respectively.     

基于主动工作流视图的跨组织工作流管理和监控

跨组织工作流可以有效的利用不同企业的资源,提高企业的生产率,因而得到了越来越广泛的应用。如何使企业在增强对外交互能力的同时,保护企业内部的敏感信息,是跨组织工作流应用中的重要问题。工作流视图能有选择的隐藏企业私有工作流的细节,同时对外提供面向过程的接口,是一种有效的实现跨组织工作流的方法。在引入主动工作流视图概念的基础上,提出基于主动工作流视图的跨组织工作流管理和监控的体系结构,探讨了跨组织工作流的主动监控及其实现。     

Fine-Grain Interoperability of Scientific Workflows in Distributed Computing Infrastructures

Today there exist a wide variety of scientific workflow management systems, each designed to fulfill the needs of a certain scientific community. Unfortunately, once a workflow application has been designed in one particular system it becomes very hard to share it with users working with different systems. Portability of workflows and interoperability between current systems barely exists. In this work, we present the fine-grained interoperability solution proposed in the SHIWA European project that brings together four representative European workflow systems: ASKALON, MOTEUR, WS-PGRADE, and Triana. The proposed interoperability is realised at two levels of abstraction: abstract and concrete. At the abstract level, we propose a generic Interoperable Workflow Intermediate Representation (IWIR) that can be used as a common bridge for translating workflows between different languages independent of the underlying distributed computing infrastructure. At the concrete level, we propose a bundling technique that aggregates the abstract IWIR representation and concrete task representations to enable workflow instantiation, execution and scheduling. We illustrate case studies using two real-workflow applications designed in a native environment and then translated and executed by a foreign workflow system in a foreign distributed computing infrastructure.     

企业应用集成中流程集成模型的研究

提出企业应用集成的三个层次,分析流程集成层次的重要性和难点;通过对流程集成的业务分析,参考交通模型,将独立应用内部流程视同市内交通,流程集成视同城际、洲际交通,给出流程集成的业务模型。在此基础上,提出流程集成平台的设计模型,该设计模型由集成流程管理平台和集成流程接口管理平台两部分组成。最后,给出流程平台的体系结构设计示例,并针对该模型中几个较复杂、有特点的模块给出设计示例。     

基于语义的网格工作流复合技术及实现

作为网格环境的基本服务,网格工作流的大规模共享和重用是使网格成为问题求解的智能化集成环境的一项核心技术。工作流复合技术能够通过已存在的解决方法的重用简化复杂的科学应用问题,而基于语义的工作流复合降低了需要了解工作流语法细节才能重用网格工作流带来的复杂性。首先运用知识表达技术提出一个基于目标概念的网格工作流语义模板,其次介绍了一个实现基于语义的工作流复合的工作流管理原型系统,其中详细阐述了工作流建模过程中基于语义的工作流复合的实现。     

Virtual workflow system for distributed collaborative scientific applications on Grids

Grid computing has become an effective computing technique in recent years. This paper develops a virtual workflow system to construct distributed collaborative applications for Grid users. The virtual workflow system consists three levels: abstract workflow system, translator and concrete workflow system. The research highlight of the implementation is that this workflow system is developed based on CORBA and Unicore Grid middleware. Furthermore, this implementation can support legacy application developed with Parco and C++ codes. This virtual workflow system can provide efficient GUI for users to organize distributed scientific collaborative applications and execute them on Grid resources. We present the design, implementation, and evaluation of this virtual workflow system in the paper.     

工作流模型挖掘算法及其应用研究

工作流管理系统由工作流模型所驱动,但产业界的实践表明定义工作流模型的工作不仅费时而且易错。工作流挖掘技术能够帮助解决这一问题,并能为现有工作流的分析与优化提供参考。简要介绍三种典型且具有应用价值的工作流模型挖掘算法,并应用其中一种挖掘算法,详细讨论了一个实际的工作流模型挖掘过程。挖掘过程以某Staffware系统的工作流日志文件为起点,包括数据预处理、初始工作流模型挖掘、初始工作流模型化简三个主要步骤,具体实现可通过一个工作流模型挖掘子系统参与完成。     

Campus Grids Meet Applications: Modeling, Metascheduling and Integration

Air Quality Forecasting (AQF) is a new discipline that attempts to reliably predict atmospheric pollution. An AQF application has complex workflows and in order to produce timely and reliable forecast results, each execution requires access to diverse and distributed computational and storage resources. Deploying AQF on Grids is one option to satisfy such needs, but requires the related Grid middleware to support automated workflow scheduling and execution on Grid resources. In this paper, we analyze the challenges in deploying an AQF application in a campus Grid environment and present our current efforts to develop a general solution for Grid-enabling scientific workflow applications in the GRACCE project. In GRACCE, an application’s workflow is described using GAMDL, a powerful dataflow language for describing application logic. The GRACCE metascheduling architecture provides the functionalities required for co-allocating Grid resources for workflow tasks, scheduling the workflows and monitoring their execution. By providing an integrated framework for modeling and metascheduling scientific workflow applications on Grid resources, we make it easy to build a customized environment with end-to-end support for application Grid deployment, from the management of an application and its dataset, to the automatic execution and analysis of its results.The work has been performed as part of the University of Houston’s Sun Microsystems Center of Excellence in Geosciences [38].     

基于动态有色Petri网的网格服务工作流模型的研究

在深入了解网格技术、网格服务和网格工作流的概念、特点及其应用的基础上,提出了一种可行的网格服务工作流系统模型,重点介绍了动态优化建模技术、动态调度算法的实现思想.定义了一种动态有色Petri网作为服务工作流的建模工具,支持服务工作流的动态优化建模和动态调度,并为服务工作流模型提供性能评价依据.验证表明采用该模型能够很好地满足用户的QoS要求,并且有助于提高资源利用率.     

Bundle and Pool Architecture for Multi-Language, Robust, Scalable Workflow Executions

In this paper, we leverage the previous work on the SHIWA bundling format and expand on this specification in order to facilitate workflow execution within a multi-workflow environment. We introduce a scalable and robust execution pool environment that supports workflows consisting of sub-workflows built upon a multitude of different workflow engines and environments, and also provide a common workflow representation for seamless connectivity through serialization to workflow bundles. We also present a meta-workflow scenario based upon this system. Workflow bundles employ the lightweight Open Archives Initiative Object Reuse and Exchange (ORE) Web-based standard, to provide a common format for representing and sharing workflows and the associated metadata required for their execution. This generalized bundling approach is already available within five workflow engines and has proven a useful environment for inter-workflow experimentation. The execution pool facilitates federated access to multiple distributed computing infrastructures supported by the underlying workflow engines subscribed to the pool. Workflow bundles are exposed using the eXtensible Messaging and Presence Protocol (XMPP), which provides the necessary communication backbone to enable multiple workflow engine agents to asynchronously publish and subscribe to bundles in meta-workflow pipelines. We present experiments showing the scalability and robustness of the pool execution approach with results showing that overheads remain controlled for up to 150 workflow agents, and that agent failures have very limited impact. We then demonstrate the applicability of our architecture by describing how a Java-based music analysis workflow can be distributed within such a multi-workflow environment consisting of the Triana and MOTEUR workflow engines.     

A Distributed Workflow Management System with Case Study of Real-life Scientific Applications on Grids

Next-generation scientific applications feature complex workflows comprised of many computing modules with intricate inter-module dependencies. Supporting such scientific workflows in wide-area networks especially Grids and optimizing their performance are crucial to the success of collaborative scientific discovery. We develop a Scientific Workflow Automation and Management Platform (SWAMP), which enables scientists to conveniently assemble, execute, monitor, control, and steer computing workflows in distributed environments via a unified web-based user interface. The SWAMP architecture is built entirely on a seamless composition of web services: the functionalities of its own are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. SWAMP also incorporates a class of efficient workflow mapping schemes to achieve optimal end-to-end performance based on rigorous performance modeling and algorithm design. The performance superiority of SWAMP over existing workflow mapping schemes is justified by extensive simulations, and the system efficacy is illustrated by large-scale experiments on real-life scientific workflows for climate modeling through effective system implementation, deployment, and testing on the Open Science Grid.