Abstract,link, publish,exploit: An end to end framework for workflow sharing

Scientific workflows are increasingly used to manage and share scientific computations and methods to analyze data. A variety of systems have been developed that store the workflows executed and make them part of public repositories However, workflows are published in the idiosyncratic format of the workflow system used for the creation and execution of the workflows. Browsing, linking and using the stored workflows and their results often becomes a challenge for scientists who may only be familiar with one system. In this paper we present an approach for addressing this issue by publishing and exploiting workflows as data on the Web with a representation that is independent from the workflow system used to create them. In order to achieve our goal, we follow the Linked Data Principles to publish workflow inputs, intermediate results, outputs and codes; and we reuse and extend well established standards like W3C PROV. We illustrate our approach by publishing workflows and consuming them with different tools designed to address common scenarios for workflow exploitation.     

Modeling and Managing Interactions among Business Processes

Most workflow management systems (WfMSs) only support the separate andindependent execution of business processes. However, processes often needto interact with each other, in order to synchronize the execution of theiractivities, to exchange process data, to request execution of services, orto notify progresses in process execution. Recent market trends also raisethe need for cooperation and interaction between processes executed in differentorganizations, posing additional challenges. In fact, in order to reduce costsand provide better services, companies are pushed to increase cooperation and toform virtual enterprises, where business processes span across organizationalboundaries and are composed of cooperating workflows executed in differentorganizations. Workflow interaction in a cross-organizational environment iscomplicated by the heterogeneity of workflow management platforms on top ofwhich workflows are defined and executed and by the different and possiblycompeting business policies and business goals that drive process executionin each organization.In this paper we propose a model and system that enable interactionbetween workflows executed in the same or in different organizations. Weextend traditional workflow models by allowing workflows to publish andsubscribe to events, and by enabling the definition of points in the processexecution where events should be sent or received. Event notifications aremanaged by a suitable event service that is capable of filtering andcorrelating events, and of dispatching them to the appropriate targetworkflow instances. The extended model can be easily mapped onto anyworkflow model, since event specific constructs can be specified by means ofordinary workflow activities, for which we provide the implementation. Inaddition, the event service is easily portable to different platforms, anddoes not require integration with the WfMS that supports the cooperatingworkflows. Therefore, the proposed approach is applicable in virtually anyenvironment and is independent on the specific platform adopted     

Deriving scientific workflows from algebraic experiment lines: A practical approach

The exploratory nature of a scientific computational experiment involves executing variations of the same workflow with different approaches, programs, and parameters. However, current approaches do not systematize the derivation process from the experiment definition to the concrete workflows and do not track the experiment provenance down to the workflow executions. Therefore, the composition, execution, and analysis for the entire experiment become a complex task. To address this issue, we propose the Algebraic Experiment Line (AEL). AEL uses a data-centric workflow algebra, which enriches the experiment representation by introducing a uniform data model and its corresponding operators. This representation and the AEL provenance model map concepts from the workflow execution data to the AEL derived workflows with their corresponding experiment abstract definitions. We show how AEL has improved the understanding of a real experiment in the bioinformatics area. By combining provenance data from the experiment and its corresponding executions, AEL provenance queries navigate from experiment concepts defined at high abstraction level to derived workflows and their execution data. It also shows a direct way of querying results from different trials involving activity variations and optionalities, only present at the experiment level of abstraction.     

Using patterns to design rules in workflows

In order to design workflows in changing and dynamic environments, a flexible, correct, and rapid realization of models of the activity flow is required. In particular, techniques are needed to design workflows capable of adapting themselves effectively when exceptional situations occur during process execution. The authors present an approach to flexible workflow design based on rules and patterns developed in the framework of the WIDE project. Rules allow a high degree of flexibility during workflow design by modeling exceptional aspects of the workflow separately from the main activity flow. Patterns model frequently occurring exceptional situations in a generalized way by providing the designer with skeletons of rules and suggestions about their instantiation, together with indications on relationships with other rules, with the activity flow, and with related information. Pattern based design relies on a pattern catalog containing patterns to be reused and on a formal basis for specializing and instantiating available patterns     

用SQL实现工作流的并发控制

工作流技术在信息系统的应用中，并发控制机制的设计是经常要面临的问题。给出一种基于将工作流中数据和任务分离的工作流并发控制机制，在保证工作流正确性的前提下，引入“数据约束”和“任务约束”的概念来提高工作流的工作性能和降低工作流设计的复杂性，并用数据库中SQL语言强有力的约束控制加以实现。     

作业流访问控制策略

作业流是一系列任务，它由一个任务集和一个任务关系集组成，其中，任务关系集说明各任务的执行顺序。该文在给出作业流定义的基础上，提出了一种基于作业流Petri网描述的访问控制机制，并介绍了该机制的实现。     

A delay-based dynamic scheduling algorithm for bag-of-task workflows with stochastic task execution times in clouds

Bag-of-Tasks (BoT) workflows are widespread in many big data analysis fields. However, there are very few cloud resource provisioning and scheduling algorithms tailored for BoT workflows. Furthermore, existing algorithms fail to consider the stochastic task execution times of BoT workflows which leads to deadline violations and increased resource renting costs. In this paper, we propose a dynamic cloud resource provisioning and scheduling algorithm which aims to fulfill the workflow deadline by using the sum of task execution time expectation and standard deviation to estimate real task execution times. A bag-based delay scheduling strategy and a single-type based virtual machine interval renting method are presented to decrease the resource renting cost. The proposed algorithm is evaluated using a cloud simulator ElasticSim which is extended from CloudSim. The results show that the dynamic algorithm decreases the resource renting cost while guaranteeing the workflow deadline compared to the existing algorithms.     

允许违反局部时间约束的科学工作流调度策略

提高科学工作流在云环境中的执行效率、降低执行费用受到广泛关注。用户期望的局部QoS约束与工作流的总体执行效率之间往往存在矛盾。针对该现象,在前期的研究基础上提出一种允许违反局部时间约束的科学工作流调度策略。通过对已聚簇的工作流任务集使用任务后向优先合并的方法,可实现任务间空闲时间片的合理利用,进而优化科学工作流的执行时间;另外,为充分利用任务的松弛时间,提高工作流的整体执行效率,允许部分任务的调度违反局部最晚完成时间的约束。实验结果表明,该策略能提前科学工作流的最早完成时间,提高处理机的利用率,并最终降低工作流的执行费用。     

Optimizing Grid-Based Workflow Execution

Large-scale applications can be expressed as a set of tasks with data dependencies between them, also known as application workflows. Due to the scale and data processing requirements of these applications, they require Grid computing and storage resources. So far, the focus has been on developing easy to use interfaces for composing these workflows and finding an optimal mapping of tasks in the workflow to the Grid resources in order to minimize the completion time of the application. After this mapping is done, a workflow execution engine is required to run the workflow over the mapped resources. In this paper, we show that the performance of the workflow execution engine in executing the workflow can also be a critical factor in determining the workflow completion time. Using Condor as the workflow execution engine, we examine the various factors that affect the completion time of a fine granularity astronomy workflow. We show that changing the system parameters that influence these factors and restructuring the workflow can drastically reduce the completion time of this class of workflows. We also examine the effect on the optimizations developed for the astronomy application on a coarser granularity biology application. We were able to reduce the completion time of the Montage and the Tomography application workflows by 90% and 50%, respectively.     

A Provenance-based Adaptive Scheduling Heuristic for Parallel Scientific Workflows in Clouds

In the last years, scientific workflows have emerged as a fundamental abstraction for structuring and executing scientific experiments in computational environments. Scientific workflows are becoming increasingly complex and more demanding in terms of computational resources, thus requiring the usage of parallel techniques and high performance computing (HPC) environments. Meanwhile, clouds have emerged as a new paradigm where resources are virtualized and provided on demand. By using clouds, scientists have expanded beyond single parallel computers to hundreds or even thousands of virtual machines. Although the initial focus of clouds was to provide high throughput computing, clouds are already being used to provide an HPC environment where elastic resources can be instantiated on demand during the course of a scientific workflow. However, this model also raises many open, yet important, challenges such as scheduling workflow activities. Scheduling parallel scientific workflows in the cloud is a very complex task since we have to take into account many different criteria and to explore the elasticity characteristic for optimizing workflow execution. In this paper, we introduce an adaptive scheduling heuristic for parallel execution of scientific workflows in the cloud that is based on three criteria: total execution time (makespan), reliability and financial cost. Besides scheduling workflow activities based on a 3-objective cost model, this approach also scales resources up and down according to the restrictions imposed by scientists before workflow execution. This tuning is based on provenance data captured and queried at runtime. We conducted a thorough validation of our approach using a real bioinformatics workflow. The experiments were performed in SciCumulus, a cloud workflow engine for managing scientific workflow execution.     

An algorithm of automatic workflow optimization

This paper considers an algorithm of automatic workflow optimization that, unlike well-known redesign algorithms for workflows [1, 2], can analyze arbitrary structures containing conditional branches and cycles. This algorithm operates with workflows without structural conflicts and, in the course of operation, uses execution conditions obtained as a result of application of the Boolean verification algorithm (BVA) proposed earlier in [3]. A modified BVA is proposed and its computational complexity is estimated.     

满足工作流执行时限的可抢占虚拟机实例配置和调度方法研究

随着云计算的迅速发展,将工作流部署到云计算平台已经成为了常见的选择。相比于传统的本地工作流,云工作流不仅要考虑计算时长等要求,还要考虑其产生的经济开销。而云计算服务商为了提高资源利用率,提供了可抢占虚拟机实例这种非常廉价但是不稳定的资源。针对工作流在云计算中的调度和执行问题,提出一种满足工作流执行时限的可抢占虚拟机实例配置和调度方法。该方法使用马尔科夫模型和动态规划方法,对可抢占虚拟机实例的价格进行预测,并得到成本最低的出价策略。同时,结合工作流的执行时限要求,在估计的出价策略下对工作流中使用的实例进行配置。实验结果显示,相比于全部使用按需付费虚拟机实例,该方法在满足工作流执行时限的前提下最高可以节省89.9%的计算成本。     

Organizational Management in Workflow Applications – Issues and Perspectives

Business processes automation requires the specification of process structures as well as the definition of resources involved in the execution of these processes. While the modeling of business processes and workflows is well researched, the link between the organizational elements and process activities is less well understood, and current developments in the web services choreography area completely neglect the organizational aspect of workflow applications. The purpose of this paper is to give an overview of the organizational aspects of workflow technology in the context of the workflow life cycle, to provide a review of existing work, and to develop guidelines for the design of a workflow-enabled organization, which can be used by both workflow vendors and users.     

VINCASim:一种网格工作流可靠性仿真工具

网格工作流作为综合利用网格资源求解问题的"编程"技术已得到广泛应用,其可靠性保障研究得到越来越多的关注.然而,面对网格环境固有的复杂性和不确定性,如何时可靠性保障方法有效、方便地进行评测,是一个有待深入探索的挑战性问题.以VINCA网格工作流为参考原型,对影响其可靠性的因素进行了系统的分析,抽象出了网格工作流系统组件模型和可靠性属性模型,并基于GridSim实现了一个可配置、易扩展的网格工作流可靠性仿真工具包--VINCASim.此工具可以通过配置方便地模拟节点失效、工作流引擎失效、网络连接失效、流程执行异常等出错情形,提供扩展接口支持以编程的方式引入各种可靠性保障方法,为评测不同方法的有效性提供可控的、可重复的实验平台.通过场景示例说明了该工具的扩展性和易用性.     

A new optimization phase for scientific workflow management systems

Scientific workflows have emerged as an important tool for combining the computational power with data analysis for all scientific domains in e-science, especially in the life sciences. They help scientists to design and execute complex in silico experiments. However, with rising complexity it becomes increasingly impractical to optimize scientific workflows by trial and error. To address this issue, we propose to insert a new optimization phase into the common scientific workflow life cycle. This paper describes the design and implementation of an automated optimization framework for scientific workflows to implement this phase. Our framework was integrated into Taverna, a life-science oriented workflow management system and offers a versatile programming interface (API), which enables easy integration of arbitrary optimization methods. We have used this API to develop an example plugin for parameter optimization that is based on a Genetic Algorithm. Two use cases taken from the areas of structural bioinformatics and proteomics demonstrate how our framework facilitates setup, execution, and monitoring of workflow parameter optimization in high performance computing e-science environments.     

基于构件的本体扩展工作流的构架研究与应用

几乎所有的行业都涉及工作流,用以协调任务之间的执行。由构件组成的工作流,可以提高系统的复用性,加快系统的开发和配置速度。但是构件之间句法语义的不同,决定了它们是不能直接组装的。论文提出基于本体扩展的工作流构件,使用本体来解决不同构件之间的元数据映射,并实现了基于构件的本体扩展工作流的框架,做出了具体实施。