用于高性能计算的作业调度能效性研究综述

由于科学研究与商业应用等对高性能计算的需求与日俱增,高性能计算的性能和系统规模得到迅速发展。但是,急剧增长的功耗严重限制了高性能计算系统的设计和使用,使得低功耗技术成为高性能计算领域的关键技术。作为整个系统的核心组件,作业调度系统立足有限的系统资源,对用户提交的应用进行作业-资源分配,其能效性对于整个高性能计算系统的能耗控制与调节起到至关重要的作用。首先介绍主要的能量效率技术和常用的作业调度策略,然后对当前高性能计算作业调度能效性进行分析,并讨论了其面临的挑战及未来发展方向。     

Toward Cloud Computing QoS Architecture: Analysis of Cloud Systems and Cloud Services

Cloud can be defined as a new computing paradigm that provides scalable, on-demand, and virtualized resources for users. In this style of computing, users can access a shared pool of computing resources which are provisioned with minimal management efforts of users. Yet there are some obstacles and concerns about the use of clouds. Guaranteeing quality of service (QoS) by service providers can be regarded as one of the main concerns for companies tending to use it. Service provisioning in clouds is based on service level agreements representing a contract negotiated between users and providers. According to this contract, if a provider cannot satisfy its agreed application requirements, it should pay penalties as compensation. In this paper, we intend to carry out a comprehensive survey on the models proposed in literature with respect to the implementation principles to address the QoS guarantee issue.      

SLA negotiation and enforcement policies for revenue maximization and client classification in cloud providers

In Cloud Computing markets, owners of computing resources negotiate with their potential clients to sell computing power. The terms of the Quality of Service (QoS) to be provided as well as the economic conditions are established in a Service-Level Agreement (SLA). The common objective of a Cloud provider is to maximize its economic profit. However, there are situations in which providers must differentiate the SLAs with respect to the type of client that is willing to access the resources or the agreed QoS, e.g. when the hardware resources are shared between users of the company that own the resources and external users.This article proposes two sets of policies to manage SLAs with respect to the business objectives of a Cloud provider: Revenue Maximization or classification of clients. The criterion to classify clients is established according to the relationship between client and provider (external user, internal or another privileged relationship) and the QoS that the client purchases (cheap contracts or extra QoS by paying an extra fee). The validity of the policies is demonstrated through exhaustive experiments.     

Enhancing performance of failure-prone clusters by adaptive provisioning of cloud resources

In this paper, we investigate Cloud computing resource provisioning to extend the computing capacity of local clusters in the presence of failures. We consider three steps in the resource provisioning including resource brokering, dispatch sequences, and scheduling. The proposed brokering strategy is based on the stochastic analysis of routing in distributed parallel queues and takes into account the response time of the Cloud provider and the local cluster while considering computing cost of both sides. Moreover, we propose dispatching with probabilistic and deterministic sequences to redirect requests to the resource providers. We also incorporate checkpointing in some well-known scheduling algorithms to provide a fault-tolerant environment. We propose two cost-aware and failure-aware provisioning policies that can be utilized by an organization that operates a cluster managed by virtual machine technology, and seeks to use resources from a public Cloud provider. Simulation results demonstrate that the proposed policies improve the response time of users’ requests by a factor of 4.10 under a moderate load with a limited cost on a public Cloud.     

云存储系统节能研究综述

云存储系统作为云计算的重要组成部分,是各种云计算服务的基础。但随云存储系统规模的不断扩大和在设计时对能耗因素的忽略,使其日益暴露出高能耗、低效率的问题。因为云存储系统占整个云计算中心能耗的27%～40%,所以无论从降低服务提供商的运营成本,还是从降低能耗以保护环境的角度出发,研究云存储系统中的节能技术都具有很大的现实意义与应用前景。将存储系统中的能耗优化问题分为基于硬件的节能方法与基于调度的节能方法两大类进行讨论;并将基于调度的节能方法分为基于节点调度、基于数据调度和基于缓存预取技术3类进行综合比较;最后,对适应节能的云存储体系结构、节能模式下的QoS保证、节能模式与计算模式的匹配以及纠删码容错技术下的节能研究4个方向进行了展望。     

Monitoring Power Data: A first step towards a unified energy efficiency evaluation toolset for HPC data centers

The energy consumption of High Performance Computing (HPC) systems, which are the key technology for many modern computation-intensive applications, is rapidly increasing in parallel with their performance improvements. This increase leads HPC data centers to focus on three major challenges: the reduction of overall environmental impacts, which is driven by policy makers; the reduction of operating costs, which are increasing due to rising system density and electrical energy costs; and the 20 MW power consumption boundary for Exascale computing systems, which represent the next thousandfold increase in computing capability beyond the currently existing petascale systems. Energy efficiency improvements will play a major part in addressing these challenges.This paper presents a toolset, called Power Data Aggregation Monitor (PowerDAM), which collects and evaluates data from all aspects of the HPC data center (e.g. environmental information, site infrastructure, information technology systems, resource management systems, and applications). The aim of PowerDAM is not to improve the HPC data center's energy efficiency, but is to collect energy relevant data for analysis without which energy efficiency improvements would be non-trivial and incomplete. Thus, PowerDAM represents a first step towards a truly unified energy efficiency evaluation toolset needed for improving the overall energy efficiency of HPC data centers.     

Energy-Aware Scheduling for Tasks with Target-Time in Blockchain based Data Centres

Cloud computing infrastructures have intended to provide computing services to end-users through the internet in a pay-per-use model. The extensive deployment of the Cloud and continuous increment in the capacity and utilization of data centers (DC) leads to massive power consumption. This intensifying scale of DCs has made energy consumption a critical concern. This paper emphasizes the task scheduling algorithm by formulating the system model to minimize the makespan and energy consumption incurred in a data center. Also, an energy-aware task scheduling in the Blockchain-based data center was proposed to offer an optimal solution that minimizes makespan and energy consumption. The established model was analyzed with a target-time responsive precedence scheduling algorithm. The observations were analyzed and compared with the traditional scheduling algorithms. The outcomes exhibited that the developed solution incurs better performance with a response to resource utilization and decreasing energy consumption. The investigation revealed that the applied strategy considerably enhanced the effectiveness of the designed schedule.

     

Ontology-based access control model for security policy reasoning in cloud computing

There are many security issues in cloud computing service environments, including virtualization, distributed big-data processing, serviceability, traffic management, application security, access control, authentication, and cryptography, among others. In particular, data access using various resources requires an authentication and access control model for integrated management and control in cloud computing environments. Cloud computing services are differentiated according to security policies because of differences in the permitted access right between service providers and users. RBAC (Role-based access control) and C-RBAC (Context-aware RBAC) models do not suggest effective and practical solutions for managers and users based on dynamic access control methods, suggesting a need for a new model of dynamic access control that can address the limitations of cloud computing characteristics. This paper proposes Onto-ACM (ontology-based access control model), a semantic analysis model that can address the difference in the permitted access control between service providers and users. The proposed model is a model of intelligent context-aware access for proactively applying the access level of resource access based on ontology reasoning and semantic analysis method.     

CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms

Cloud computing is a recent advancement wherein IT infrastructure and applications are provided as ‘services’ to end‐users under a usage‐based payment model. It can leverage virtualized services even on the fly based on requirements (workload patterns and QoS) varying with time. The application services hosted under Cloud computing model have complex provisioning, composition, configuration, and deployment requirements. Evaluating the performance of Cloud provisioning policies, application workload models, and resources performance models in a repeatable manner under varying system and user configurations and requirements is difficult to achieve. To overcome this challenge, we propose CloudSim: an extensible simulation toolkit that enables modeling and simulation of Cloud computing systems and application provisioning environments. The CloudSim toolkit supports both system and behavior modeling of Cloud system components such as data centers, virtual machines (VMs) and resource provisioning policies. It implements generic application provisioning techniques that can be extended with ease and limited effort. Currently, it supports modeling and simulation of Cloud computing environments consisting of both single and inter‐networked clouds (federation of clouds). Moreover, it exposes custom interfaces for implementing policies and provisioning techniques for allocation of VMs under inter‐networked Cloud computing scenarios. Several researchers from organizations, such as HP Labs in U.S.A., are using CloudSim in their investigation on Cloud resource provisioning and energy‐efficient management of data center resources. The usefulness of CloudSim is demonstrated by a case study involving dynamic provisioning of application services in the hybrid federated clouds environment. The result of this case study proves that the federated Cloud computing model significantly improves the application QoS requirements under fluctuating resource and service demand patterns. Copyright © 2010 John Wiley & Sons, Ltd.     

APU: 一种精确评估超线程处理器算力消耗程度的方法

伴随着云计算的发展,以及软件即服务(SaaS)、方法即服务(FaaS)等服务框架的提出,数据中心作为服务的提供商,面临着持续性的资源管理挑战:一方面需要保证服务质量(quality of service, QoS),另一方面又需要控制资源成本.为了在提升资源使用率的同时确保负载压力在可承受范围内波动,一种精确衡量当前算力消耗程度的方法成为关键性的研究问题.传统的评估指标CPU利用率,由于虚拟化技术的成熟以及并行技术的发展,无法应对资源竞争所产生的干扰,失去了评估精度.而当前数据中心的主流处理器基本都开启了超线程技术,这导致评估超线程处理器算力消耗程度的需求亟待解决.为了应对这一评估挑战,基于超线程机制的理解以及线程行为的建模,提出一种评估超线程处理器算力消耗的方法 APU.同时考虑到不同权限的用户能访问的系统层级不同,还提出了两种实现方案:一种基于硬件层支持的实现,以及一种基于操作系统层支持的实现. APU方法利用传统CPU利用率指标作为输入,没有其他维度的需求,免去了新监测工具的开发部署代价,也无需特殊硬件体系结构的支持,确保该方法的通用性和易用性.最后通过SPEC基准测试程序进一步...     

云工作流中基于分时虚拟机的任务层调度算法

云计算是新的一种面向市场的商业计算模式,向用户按需提供服务,云计算的商业特性使其关注向用户提供服务的服务质量。任务调度和资源分配是云计算中两个关键的技术,所使用的虚拟化技术使得其资源分配和任务调度有别于以往的并行分布式计算。目前主要的调度算法是借鉴网格环境下的调度策略,研究基于QoS的调度算法,存在执行效率较低的问题。我们对云工作流任务层调度进行深入研究,分析由底层资源虚拟化形成的虚拟机的特性,结合工作流任务的各类QoS约束,提出了基于虚拟机分时特性的任务层ACS调度算法。经过试验,我们提出的算法相比于文献[1]中的算法在对于较多并行任务的执行上存在较大的优势,能够很好的利用虚拟的分时特性,优化任务到虚拟机的调度。     

基于组合双向拍卖的云资源调度方法

针对跨数据中心的资源调度问题，提出了一种基于组合双向拍卖（PCDA）的资源调度方案。首先，将云资源拍卖分为三个部分：云用户代理报价、云资源提供商要价、拍卖代理组织拍卖；其次，在定义用户的优先级及任务紧迫度的基础上，在拍卖过程中估算每一个工作发生的服务等级协议（SLA）违规并以此计算云提供商的收益，同时每轮竞拍允许成交多项交易；最终达到根据用户等级合理分配云资源调度的效果。仿真实验结果表明该算法保证了竞拍成功率，与传统一次拍卖成交一项的组合双向拍卖方案相比，PCDA在竞拍时间段产生的能耗降低了35.00%，拍卖云提供商的利润提高了约38.84%。     

A security and cost aware scheduling algorithm for heterogeneous tasks of scientific workflow in clouds

Security is increasingly critical for various scientific workflows that are big data applications and typically take quite amount of time being executed on large-scale distributed infrastructures. Cloud computing platform is such an infrastructure that can enable dynamic resource scaling on demand. Nevertheless, based on pay-per-use and hourly-based pricing model, users should pay attention to the cost incurred by renting virtual machines (VMs) from cloud data centers. Meanwhile, workflow tasks are generally heterogeneous and require different instance series (i.e., computing optimized, memory optimized, storage optimized, etc.). In this paper, we propose a security and cost aware scheduling (SCAS) algorithm for heterogeneous tasks of scientific workflow in clouds. Our proposed algorithm is based on the meta-heuristic optimization technique, particle swarm optimization (PSO), the coding strategy of which is devised to minimize the total workflow execution cost while meeting the deadline and risk rate constraints. Extensive experiments using three real-world scientific workflow applications, as well as CloudSim simulation framework, demonstrate the effectiveness and practicality of our algorithm.     

虚拟化云计算数据中心资源节能调度算法研究

针对当前云计算数据中心资源调度过程耗时长、能耗高、数据传输准确性较低的问题,提出基于VR沉浸式的虚拟化云计算数据中心资源节能调度算法。构建云计算数据中心资源采样模型,结合虚拟现实(virtual reality,VR)互动装置输出、转换、调度中心资源,提取中心资源的关联规则特征量,采用嵌入式模糊聚类融合分析方法三维重构中心资源,建立虚拟化云计算数据中心资源的信息融合中心,采用决策相关性分析方法,结合差异化融合特征量实现对数据中心资源调度,实现虚拟化云计算数据中心资源实时节能调度。仿真结果表明,采用该方法进行虚拟化云计算数据中心资源节能调度的数据传输准确性较高,时间开销较短,能耗较低,在中心资源调度中具有很好的应用价值。     

Prediction of job characteristics for intelligent resource allocation in HPC systems: a survey and future directions

Nowadays, high-performance computing (HPC) clusters are increasingly popular. Large volumes of job logs recording many years of operation traces have been accumulated. In the same time, the HPC cloud makes it possible to access HPC services remotely. For executing applications, both HPC end-users and cloud users need to request specific resources for different workloads by themselves. As users are usually not familiar with the hardware details and software layers, as well as the performance behavior of the underlying HPC systems. It is hard for them to select optimal resource configurations in terms of performance, cost, and energy efficiency. Hence, how to provide on-demand services with intelligent resource allocation is a critical issue in the HPC community. Prediction of job characteristics plays a key role for intelligent resource allocation. This paper presents a survey of the existing work and future directions for prediction of job characteristics for intelligent resource allocation in HPC systems. We first review the existing techniques in obtaining performance and energy consumption data of jobs. Then we survey the techniques for single-objective oriented predictions on runtime, queue time, power and energy consumption, cost and optimal resource configuration for input jobs, as well as multi-objective oriented predictions. We conclude after discussing future trends, research challenges and possible solutions towards intelligent resource allocation in HPC systems.     

Novel resource allocation algorithms to performance and energy efficiency in cloud computing

The rapid growth in demand for computational power has led to a shift to the cloud computing model established by large-scale virtualized data centers. Such data centers consume enormous amounts of electrical energy. Cloud providers must ensure that their service delivery is flexible to meet various consumer requirements. However, to support green computing, cloud providers also need to minimize the cloud infrastructure energy consumption while conducting the service delivery. In this paper, for cloud environments, a novel QoS-aware VMs consolidation approach is proposed that adopts a method based on resource utilization history of virtual machines. Proposed algorithms have been implemented and evaluated using CloudSim simulator. Simulation results show improvement in QoS metrics and energy consumption as well as demonstrate that there is a trade-off between energy consumption and quality of service in the cloud environment.     

SLA-based admission control for a Software-as-a-Service provider in Cloud computing environments

Software as a Service (SaaS) provides access to applications to end users over the Internet without upfront investment in infrastructure and software. To serve their customers, SaaS providers utilise resources of internal data centres or rent resources from a public Infrastructure as a Service (IaaS) provider. In-house hosting can increase administration and maintenance costs whereas renting from an IaaS provider can impact the service quality due to its variable performance. To overcome these limitations, we propose innovative admission control and scheduling algorithms for SaaS providers to effectively utilise public Cloud resources to maximize profit by minimizing cost and improving customer satisfaction level. Furthermore, we conduct an extensive evaluation study to analyse which solution suits best in which scenario to maximize SaaS provider?s profit. Simulation results show that our proposed algorithms provide substantial improvement (up to 40% cost saving) over reference ones across all ranges of variation in QoS parameters.     

Environmental-aware virtual data center network

Cloud computing services have recently become a ubiquitous service delivery model, covering a wide range of applications from personal file sharing to being an enterprise data warehouse. Building green data center networks providing cloud computing services is an emerging trend in the Information and Communication Technology (ICT) industry, because of Global Warming and the potential GHG emissions resulting from cloud services. As one of the first worldwide initiatives provisioning ICT services entirely based on renewable energy such as solar, wind and hydroelectricity across Canada and around the world, the GreenStar Network (GSN) was developed to dynamically transport user services to be processed in data centers built in proximity to green energy sources, reducing Greenhouse Gas (GHG) emissions of ICT equipments. Regarding the current approach, which focuses mainly in reducing energy consumption at the micro-level through energy efficiency improvements, the overall energy consumption will eventually increase due to the growing demand from new services and users, resulting in an increase in GHG emissions. Based on the cooperation between Mantychore FP7 and the GSN, our approach is, therefore, much broader and more appropriate because it focuses on GHG emission reductions at the macro-level. This article presents some outcomes of our implementation of such a network model, which spans multiple green nodes in Canada, Europe and the USA. The network provides cloud computing services based on dynamic provision of network slices through relocation of virtual data centers.     

云计算资源调度研究综述

资源调度是云计算的一个主要研究方向.首先对云计算资源调度的相关研究现状进行深入调查和分析;然后重点讨论以降低云计算数据中心能耗为目标的资源调度方法、以提高系统资源利用率为目标的资源管理方法、基于经济学的云资源管理模型,给出最小能耗的云计算资源调度模型和最小服务器数量的云计算资源调度模型,并深入分析和比较现有的云资源调度方法;最后指出云计算资源管理的未来重要研究方向:基于预测的资源调度、能耗与性能折衷的调度、面向不同应用负载的资源管理策略与机制、面向计算能力(CPU、内存)和网络带宽的综合资源分配、多目标优化的资源调度,以便为云计算研究提供有益的参考.     

Resource management for bursty streams on multi-tenancy cloud environments

The number of applications that need to process data continuously over long periods of time has increased significantly over recent years. The emerging Internet of Things and Smart Cities scenarios also confirm the requirement for real time, large scale data processing. When data from multiple sources are processed over a shared distributed computing infrastructure, it is necessary to provide some Quality of Service (QoS) guarantees for each data stream, specified in a Service Level Agreement (SLA). SLAs identify the price that a user must pay to achieve the required QoS, and the penalty that the provider will pay the user in case of QoS violation. Assuming maximization of revenue as a Cloud provider’s objective, then it must decide which streams to accept for storage and analysis; and how many resources to allocate for each stream. When the real-time requirements demand a rapid reaction, dynamic resource provisioning policies and mechanisms may not be useful, since the delays and overheads incurred might be too high. Alternatively, idle resources that were initially allocated for other streams could be re-allocated, avoiding subsequent penalties. In this paper, we propose a system architecture for supporting QoS for concurrent data streams to be composed of self-regulating nodes. Each node features an envelope process for regulating and controlling data access and a resource manager to enable resource allocation, and selective SLA violations, while maximizing revenue. Our resource manager, based on a shared token bucket, enables: (i) the re-distribution of unused resources amongst data streams; and (ii) a dynamic re-allocation of resources to streams likely to generate greater profit for the provider. We extend previous work by providing a Petri-net based model of system components, and we evaluate our approach on an OpenNebula-based Cloud infrastructure.