作业流访问控制策略

作业流是一系列任务，它由一个任务集和一个任务关系集组成，其中，任务关系集说明各任务的执行顺序。该文在给出作业流定义的基础上，提出了一种基于作业流Petri网描述的访问控制机制，并介绍了该机制的实现。     

分布式系统中计算作业流的均衡调度算法

分布式系统中计算作业流被映射到节点后无法进行动态调整,使关键作业无法及时执行而造成作业间等待。针对该问题,提出一种计算作业流均衡调度算法。算法对映射到分布式节点的作业根据其依赖关系得出阶位值,依据该值在分布式节点上进行动态优先值调整,使关键作业尽早完成,减少作业之间的等待,缩短计算作业流执行时间。实际系统应用表明,该算法对作业管理系统中投入的计算作业流的快速执行有较强优越性。     

基于Web机制的作业管理系统的设计与实现

传统的作业管理系统一般局限在局域网范围内，用户量有限，作业的执行状态和结果不能通过Internet网远程监控和查询。文章在考察Web技术的基础上设计并实现了一种基于Web机制的作业管理系统，文中阐述了系统的应用模式和体系结构，以及系统中各个子系统的功能及相互关系，引入了作业网络的概念，描述了作业间复杂的逻辑依赖关系，也可表达作业网络间的层次和嵌套关系，这进一步扩大了系统的应用领域。     

一种基于流作业的网格作业调度研究

在基于网格环境的一些网格应用中,用户需要提交一种作业类型,该作业可以被分解为逻辑上独立的元作业,这些元作业不存在依赖和通讯关系,并且它们的执行需要大量的数据移动。针对这种作业类型,本文提出了一种基于流作业的网格调度模型。在该模型中,这些独立的元作业像"流"一样自主地流向各个计算节点去执行,各计算节点接收的流量取决于其计算能力,并避免"断流"问题。同时,该模型还分离了作业流和数据流,实现了作业逻辑控制和数据控制的分离,提高了调度的灵活性。本文将该调度模型应用于药物虚拟筛选应用中,该模型能够充分利用计算节点的计算能力。     

一种基于Workflow的作业流管理系统研究

将工作流技术应用于作业管理系统,提出了基于Workflow的作业流管理系统,首先给出了系统的体系结构,然后从作业流的定义与描述,作业流的调度与执行等方面具体分析了作业流管理模型,该系统是对现有作业管理系统和工作流管理系统的动态扩充,有着非常广阔的应用前景。     

基于累计工作量的在线大数据分析作业调度算法

针对Hadoop和Spark等大数据分析系统中无先验知识任务的高效执行问题，设计了基于累计工作量（CRW）的任务调度器CRWScheduler。该调度器根据CRW将任务在低权重队列与高权重队列间切换；在为作业分配资源时，同时考虑到作业所在的队列和其瞬时占用资源量，无需作业先验知识即显著提升系统性能。基于Apache Hadoop YARN实现了CRWScheduler原型，在28个节点的基准测试集群上的实验表明，与YARN的公平调度机制相比，作业流时间（JFT）平均降低21%，其中95百分位的作业流时间（JFT）最多降低了35%，并且在与任务级调度程序协作时可获得进一步的性能提升。     

一种并发程序依赖性分析方法

并发程序的依赖性分析是并发程序分析，理解，调试，测试和维护的重要手段，由于并发程序执行的不确定性，目前，尚有很多难点有待解决，针对Ada任务机制，首先提出了一种简洁，有效的并发程序表示方法－并发程序流图，然后讨论了由任务间同步引起的同步依赖和由访问共享变量引起的任务数据依赖，建立了并发程序依赖图，并在此基础上给出了一种有效的并发程序依赖性分析算法，得到一个比较精确的依赖性，较好地解决了并发程序依赖关系不可传递性问题。     

基于LT-backfilling算法的集群作业调度系统

在胖节点集群系统中,目前的reservations, backfilling等主流调度算法未能充分考虑单节点过载情况。该文在传统backfilling算法基础上,充分考虑节点当前负载,将预约、回填策略相结合,提出了一种新的LT-backfilling调度算法。设计了基于XML业务流程模板的二层作业调度系统,用户不仅可以提交单个作业,而且可以提交一组具有数据相关性的作业流,使得作业调度系统更加实用化。 LT-backfilling;负载均衡;作业调度;XML;作业流     

面向Dataflow的异构集群混合式资源调度框架研究

Dataflow模型的使用,使得大数据计算的批处理和流处理融合为一体.但是,现有的针对大数据计算的集群资源调度框架,要么面向流处理,要么面向批处理,不适合批处理与流处理作业共享集群资源的需求.另外,GPU用于大数据分析计算时,由于缺乏有效的CPU-GPU资源解耦方式,降低了资源使用效率.在分析现有的集群资源调度框架的基础上,设计并实现了一种可以感知批处理/流处理应用的混合式资源调度框架HRM.它以共享状态架构为基础,采用乐观封锁协议和悲观封锁协议相结合的方式,确保流处理作业和批处理作业的不同资源要求.在计算节点上,提供CPU-GPU资源的灵活绑定,采用队列堆叠技术,不但满足流处理作业的实时性需求,也减少了反馈延迟并实现了GPU资源的共享.通过模拟大规模作业的调度,结果显示, HRM的调度延迟只有集中式调度框架的75%左右;使用实际负载测试,批处理与流处理共享集群时,使用HRM调度框架, CPU资源利用率提高25%以上;而使用细粒度作业调度方法,不但GPU利用率提高2倍以上,作业的完成时间也能够减少50%左右.     

C程序单元级依赖性分析

程序依赖性分析是软件分析的一个基本内容，目前的相关工作大多集中在语句级的分析方面。人们同样需要单元级的依赖信息来考察单元间的信息流向及整个程序的构架。本文针对C程序中函数间的调用依赖、参数传递依赖、全局数据依赖以及文件间的包含依赖和外部变量定义依赖进行了分析，并提出单元依赖图表达这些关系。基于此图，本文采用基于信息论的方法度量了单元间的耦合性。单元依赖图中保留的函数调用间的互斥关系提高了度量的准确性。相关的分析思想和技术适用于分析使用其它高级程序设计语言编写的软件。     

On-line scheduling mesh jobs with dependencies

We study an on-line problem of scheduling parallel jobs on two-dimensional meshes. Parallel jobs arrive dynamically according to the dependencies between them, which are unknown before the jobs appear. Each job may need more than one processor simultaneously and is required to be scheduled on a submesh of the processors which are located on a two-dimensional mesh, i.e., a job must be scheduled on a rectangle of given dimensions. The objective is to minimize the maximum completion time (makespan). We deal with a UET job system, in which all job processing times are equal. We show a lower bound of 3.859 and present a 5.25-competitive algorithm. It significantly improves a previous lower bound of 3.25 and a previous upper bound of 46/7. We consider also the rotated two-dimensional mesh, in which the parallel jobs can be rotated and the rotation of all the jobs is feasible. A lower bound of 3.535 is proven and an on-line algorithm with competitive ratio of at most 4.25 is derived.     

基于语义约束的数据转换模式研究

XML是因特网中不同企业之间进行信息交流的一种标准的数据转换模式,为了加速数据之间的转换,企业组织通过定义公共数据文档接口来实现基于XML的应用。本文提出了一种新的将关系数据模式转换为XML模式的方法。在转换的过程中,不仅考虑关系模式的结构,而且考虑语义约束,比如内部函数依赖。该方法的输入模式是具有多值函数依赖的关系模式,输出模式是X-Schema。最后,通过实验对该转换方法进行了验证。     

Modular analysis and modelling of risk scenarios with dependencies

The risk analysis of critical infrastructures such as the electric power supply or telecommunications is complicated by the fact that such infrastructures are mutually dependent. We propose a modular approach to the modelling and analysis of risk scenarios with dependencies. Our approach may be used to deduce the risk level of an overall system from previous risk analyses of its constituent systems. A custom made assumption-guarantee style is put forward as a means to describe risk scenarios with external dependencies. We also define a set of deduction rules facilitating various kinds of reasoning, including the analysis of mutual dependencies between risk scenarios expressed in the assumption-guarantee style.     

Discovering branching and fractional dependencies in databases

The discovery of dependencies between attributes in databases is an important problem in data mining, and can be applied to facilitate future decision-making. In the present paper some properties of the branching dependencies are examined. We define a minimal branching dependency and we propose an algorithm for finding all minimal branching dependencies between a given set of attributes and a given attribute in a relation of a database. Our examination of the branching dependencies is motivated by their application in a database storing realized sales of products. For example, finding out that arbitrary p products have totally attracted at most q new users can prove to be crucial in supporting the decision making.In addition, we also consider the fractional and the fractional branching dependencies. Some properties of these dependencies are examined. An algorithm for finding all fractional dependencies between a given set of attributes and a given attribute in a database relation is proposed. We examine the general case of an arbitrary relation, as well as a particular case where the problem of discovering the fractional dependencies is considerably simplified.     

弹性数据相关与软件流水

最差路径是有分支循环软件流水的一大障碍.对于有分支循环,某些数据相关(称为弹性相关)在循环的动态执行中可能产生、也可能不产生实例.据此,可将严重限制并行性的弹性相关用限制较松的虚构相关代替,再进行软件流水.若调度没有遵守原来的弹性相关,则使用下推变换修正.从而缓解或者完全解除了最差路径的限制.该方法与经典的控制猜测互补,特点是允许调度含错,然后纠错.     

Grid Resource Availability Prediction-Based Scheduling and Task Replication

The frequent and volatile unavailability of volunteer-based Grid computing resources challenges Grid schedulers to make effective job placements. The manner in which host resources become unavailable will have different effects on different jobs, depending on their runtime and their ability to be checkpointed or replicated. A multi-state availability model can help improve scheduling performance by capturing the various ways a resource may be available or unavailable to the Grid. This paper uses a multi-state model and analyzes a machine availability trace in terms of that model. Several prediction techniques then forecast resource transitions into the model’s states. We analyze the accuracy of our predictors, which outperform existing approaches. We also propose and study several classes of schedulers that utilize the predictions, and a method for combining scheduling factors. We characterize the inherent tradeoff between job makespan and the number of evictions due to failure, and demonstrate how our schedulers can navigate this tradeoff under various scenarios. Lastly, we propose job replication techniques, which our schedulers utilize to replicate those jobs that are most likely to fail. Our replication strategies outperform others, as measured by improved makespan and fewer redundant operations. In particular, we define a new metric for replication efficiency, and demonstrate that our multi-state availability predictor can provide information that allows our schedulers to be more efficient than others that blindly replicate all jobs or some static percentage of jobs.     

The outcome advisor

A new classification method called The Outcome Advisor^® (OA) is presented which is an outgrowth of statistical pattern recognition and the Patrick-Fischer Generalized K-nearest Neighbor Decision Rule. Involved are new definitions of relative frequency and correlation. Training examples are store and processing begins once findings (a focus) are presented. An almost unlimited number of inferences can be made as an inference system and any feature can be used to define categories as a classification system. Implementable as a new neural net structure which is distribution free, multi-dimensional dependencies in the feature space for each category are learned utilizing a new definition of relative frequency. The new method may help explain how certain neural net structures may be estimating multidimensional dependencies. The OA has been trained and tested on established data bases and has improved performance as measured by experimental probability of error.     

混部数据中心负载特征及其任务调度优化分析

随着现代互联网数据中心的规模越来越大,数据中心面临着能耗、可靠性、可管理性与可扩展性等方面的挑战。同时,数据中心承载的服务多样,既有在线Web服务,也有离线批处理任务。在线任务要求较低的延迟,而离线任务要求较高的吞吐量。为了提高服务器利用率,降低数据中心能耗,当前数据中心往往将在线任务和离线任务混合部署到同一个计算集群中。在混部场景下,如何同时满足在线和离线任务的不同要求,是目前面临的关键挑战。分析了阿里巴巴于2018年发布的含有4034台服务器的混部计算集群在8天内的日志数据(cluster-trace-v2018),从静态配置信息、动态混部运行状态、离线批处理作业DAG依赖结构等出发,揭示其负载特征,包括任务倾斜与容器部署的相关关系等,根据任务依赖关系与关键路径,提出了相应的任务调度优化策略。     

Analysis of dependencies in advanced transaction models

Transactional dependencies play an important role in coordinating and executing the subtransactions in advanced transaction processing models, such as, nested transactions and workflow transactions. Researchers have formalized the notion of transactional dependencies and have shown how various advanced transaction models can be expressed using different kinds of dependencies. Incorrect specification of dependencies can result in unpredictable behavior of the advanced transaction, which, in turn, can lead to unavailability of resources and information integrity problems. In this work, we focus on how to correctly specify dependencies in an advanced transaction. We enumerate the different kinds of dependencies that may be present in an advanced transaction and classify them into two broad categories: event ordering and event enforcement dependencies. Different event ordering and event enforcement dependencies in an advanced transaction often interact in subtle ways resulting in conflicts and redundancies. We describe the different types of conflicts that can arise due to the presence of multiple dependencies and describe how one can detect such conflicts. An advanced transaction may also contain redundant dependencies—these are dependencies that can be logically derived from other dependencies. We show how such extraneous dependencies can be eliminated to get an equivalent set of dependencies that has the same effect as the original set. Our dependency analysis is done in the context of a generalized advanced transaction model that is capable of expressing different kinds of advanced transactions. Recommended by: Amit Sheth     

A simultaneous and iterative approach for parallel machine scheduling with sequence-dependent family setups

In this paper, we address a parallel machine scheduling problem to minimize the total weighted completion time, where product families are involved. Major setups occur when processing jobs of different families, and sequence dependencies are also taken into account. Considering its high practical relevance, we focus on the special case where all jobs of the same family have identical processing times. In order to avoid redundant setups, batching jobs of the same family can be performed. We first develop a variable neighborhood search algorithm (VNS) to solve the interrelated subproblems in a simultaneous manner. To further reduce computing time, we also propose an iterative scheme which alternates between a specific heuristic to form batches and a VNS scheme to schedule entire batches. Computational experiments are conducted which confirm the benefits of batching. Test results also show that both simultaneous and iterative approach outperform heuristics based on a fixed batch size and list scheduling. Furthermore, the iterative procedure succeeds in balancing solution quality and computing time.