Multi-objective list scheduling of workflow applications in distributed computing infrastructures

Executing large-scale applications in distributed computing infrastructures (DCI), for example modern Cloud environments, involves optimization of several conflicting objectives such as makespan, reliability, energy, or economic cost. Despite this trend, scheduling in heterogeneous DCIs has been traditionally approached as a single or bi-criteria optimization problem. In this paper, we propose a generic multi-objective optimization framework supported by a list scheduling heuristic for scientific workflows in heterogeneous DCIs. The algorithm approximates the optimal solution by considering user-specified constraints on objectives in a dual strategy: maximizing the distance to the user’s constraints for dominant solutions and minimizing it otherwise. We instantiate the framework and algorithm for a four-objective case study comprising makespan, economic cost, energy consumption, and reliability as optimization goals. We implemented our method as part of the ASKALON environment (Fahringer et al., 2007) for Grid and Cloud computing and demonstrate through extensive real and synthetic simulation experiments that our algorithm outperforms related bi-criteria heuristics while meeting the user constraints most of the time.     

A green energy-efficient scheduling algorithm using the DVFS technique for cloud datacenters

Information and communication technology (ICT) has a profound impact on environment because of its large amount of CO₂ emissions. In the past years, the research field of “green” and low power consumption networking infrastructures is of great importance for both service/network providers and equipment manufacturers. An emerging technology called Cloud computing can increase the utilization and efficiency of hardware equipment. The job scheduler is needed by a cloud datacenter to arrange resources for executing jobs. In this paper, we propose a scheduling algorithm for the cloud datacenter with a dynamic voltage frequency scaling technique. Our scheduling algorithm can efficiently increase resource utilization; hence, it can decrease the energy consumption for executing jobs. Experimental results show that our scheme can reduce more energy consumption than other schemes do. The performance of executing jobs is not sacrificed in our scheme. We provide a green energy-efficient scheduling algorithm using the DVFS technique for Cloud computing datacenters.     

Environment-conscious scheduling of HPC applications on distributed Cloud-oriented data centers

The use of High Performance Computing (HPC) in commercial and consumer IT applications is becoming popular. HPC users need the ability to gain rapid and scalable access to high-end computing capabilities. Cloud computing promises to deliver such a computing infrastructure using data centers so that HPC users can access applications and data from a Cloud anywhere in the world on demand and pay based on what they use. However, the growing demand drastically increases the energy consumption of data centers, which has become a critical issue. High energy consumption not only translates to high energy cost which will reduce the profit margin of Cloud providers, but also high carbon emissions which are not environmentally sustainable. Hence, there is an urgent need for energy-efficient solutions that can address the high increase in the energy consumption from the perspective of not only the Cloud provider, but also from the environment. To address this issue, we propose near-optimal scheduling policies that exploit heterogeneity across multiple data centers for a Cloud provider. We consider a number of energy efficiency factors (such as energy cost, carbon emission rate, workload, and CPU power efficiency) which change across different data centers depending on their location, architectural design, and management system. Our carbon/energy based scheduling policies are able to achieve on average up to 25% of energy savings in comparison to profit based scheduling policies leading to higher profit and less carbon emissions.     

Energy-Aware Scheduling for Tasks with Target-Time in Blockchain based Data Centres

Cloud computing infrastructures have intended to provide computing services to end-users through the internet in a pay-per-use model. The extensive deployment of the Cloud and continuous increment in the capacity and utilization of data centers (DC) leads to massive power consumption. This intensifying scale of DCs has made energy consumption a critical concern. This paper emphasizes the task scheduling algorithm by formulating the system model to minimize the makespan and energy consumption incurred in a data center. Also, an energy-aware task scheduling in the Blockchain-based data center was proposed to offer an optimal solution that minimizes makespan and energy consumption. The established model was analyzed with a target-time responsive precedence scheduling algorithm. The observations were analyzed and compared with the traditional scheduling algorithms. The outcomes exhibited that the developed solution incurs better performance with a response to resource utilization and decreasing energy consumption. The investigation revealed that the applied strategy considerably enhanced the effectiveness of the designed schedule.

     

A multi-start local search heuristic for an energy efficient VMs assignment on top of the OpenNebula cloud manager

Reducing energy consumption is an increasingly important issue in cloud computing, more specifically when dealing with a large-scale cloud. Minimizing energy consumption can significantly reduce the amount of energy bills and the greenhouse gas emissions. Therefore, many researches are carried out to develop new methods in order to consume less energy. In this paper, we present an Energy-aware Multi-start Local Search algorithm (EMLS-ONC) that optimizes the energy consumption of an OpenNebula-based Cloud. Moreover, we propose a Pareto Multi-Objective version of the EMLS-ONC called EMLS-ONC-MO dealing with both the energy consumption and the Service Level Agreement (SLA). The objective is to find a Pareto tradeoff between reducing the energy consumption of the cloud while preserving the performance of Virtual Machines (VMs). The different schedulers have been experimented using different arrival scenarios of VMs and different hardware configurations (artificial and real). The results show that EMLS-ONC and EMLS-ONC-MO outperform the other energy- and performance-aware algorithms in addition to the one provided in OpenNebula by a significant margin on the considered criteria. Besides, EMLS-ONC and EMLS-ONC-MO are proved to be able to assign at least as many VMs as the other algorithms.     

Artificial bee colony based energy‐aware resource utilization technique for cloud computing

Cloud computing is a form of distributed computing, which promises to deliver reliable services through next‐generation data centers that are built on virtualized compute and storage technologies. It is becoming truly ubiquitous and with cloud infrastructures becoming essential components for providing Internet services, there is an increase in energy‐hungry data centers deployed by cloud providers. As cloud providers often rely on large data centers to offer the resources required by the users, the energy consumed by cloud infrastructures has become a key environmental and economical concern. Much energy is wasted in these data centers because of under‐utilized resources hence contributing to global warming. To conserve energy, these under‐utilized resources need to be efficiently utilized and to achieve this, jobs need to be allocated to the cloud resources in such a way so that the resources are used efficiently and there is a gain in performance and energy efficiency. In this paper, a model for energy‐aware resource utilization technique has been proposed to efficiently manage cloud resources and enhance their utilization. It further helps in reducing the energy consumption of clouds by using server consolidation through virtualization without degrading the performance of users’ applications. An artificial bee colony based energy‐aware resource utilization technique corresponding to the model has been designed to allocate jobs to the resources in a cloud environment. The performance of the proposed algorithm has been evaluated with the existing algorithms through the CloudSim toolkit. The experimental results demonstrate that the proposed technique outperforms the existing techniques by minimizing energy consumption and execution time of applications submitted to the cloud. Copyright © 2014 John Wiley & Sons, Ltd.     

Green Approach for Joint Management of Geo-Distributed Data Centers and Interconnection Networks

Every time an Internet user downloads a video, shares a picture, or sends an email, his/her device addresses a data center and often several of them. These complex systems feed the web and all Internet applications with their computing power and information storage, but they are very energy hungry. The energy consumed by Information and Communication Technology (ICT) infrastructures is currently more than 4% of the worldwide consumption and it is expected to double in the next few years. Data centers and communication networks are responsible for a large portion of the ICT energy consumption and this has stimulated in the last years a research effort to reduce or mitigate their environmental impact. Most of the approaches proposed tackle the problem by separately optimizing the power consumption of the servers in data centers and of the network. However, the Cloud computing infrastructure of most providers, which includes traditional telcos that are extending their offer, is rapidly evolving toward geographically distributed data centers strongly integrated with the network interconnecting them. Distributed data centers do not only bring services closer to users with better quality, but also provide opportunities to improve energy efficiency exploiting the variation of prices in different time zones, the locally generated green energy, and the storage systems that are becoming popular in energy networks. In this paper, we propose an energy aware joint management framework for geo-distributed data centers and their interconnection network. The model is based on virtual machine migration and formulated using mixed integer linear programming. It can be solved using state-of-the art solvers such as CPLEX in reasonable time. The proposed approach covers various aspects of Cloud computing systems. Alongside, it jointly manages the use of green and brown energies using energy storage technologies. The obtained results show that significant energy cost savings can be achieved compared to a baseline strategy, in which data centers do not collaborate to reduce energy and do not use the power coming from renewable resources.     

云应用部署配置模型

云计算已经成为广泛使用的计算范型,越来越多的大规模分布式系统已经或正在向云平台部署和迁移.用户在部署和管理维护应用系统时通常需要管理底层基础设施资源细节,或者使用平台提供方的应用部署和管理服务,前者使得应用部署和运行时管理易于出错且费时费力,而后者则降低了系统管理的灵活性,很难满足用户的个性化需求.针对这一问题,本文提出了一种高层抽象模型来描述云应用的部署配置和管理需求.需求模型采用声明式机制定义期望的系统状态,而无需描述实现目标状态所需的执行步骤和细节.本文基于开源云计算平台OpenStack和自动化配置管理工具Puppet进行了原型实现,通过一个应用案例验证模型的有效性.     

Rafhyc: an Architecture for Constructing Resilient Services on Federated Hybrid Clouds

Federated hybrid clouds is a model of service access and delivery to community cloud infrastructures. This model opens an opportunity window to allow the integration of the enhanced science (eScience) with the Cloud paradigm. The eScience is computationally intensive science that is carried out in highly distributed computing infrastructures. Nowadays, the eScience big issue on Cloud Computing is how to leverage on-demand computing in scientific research. This requires innovation at multiple levels, from architectural design to software platforms. This paper characterizes the requirements of a federated hybrid cloud model of Infrastructure as a Service (IaaS) to provide eScience. Additionally, an architecture is defined for constructing Platform as a Service (PaaS) and Software as a Service (SaaS) in a resilient manner over federated resources. This architecture is named Rafhyc (for Resilient Architecture of Federated HYbrid Clouds). This paper also describes a prototype implementation of the Rafhyc architecture, which integrates an interoperable community middleware, named DIRAC, with federated hybrid clouds. In this way DIRAC is providing SaaS for scientific computing purposes, demonstrating that Rafhyc architecture can bring together eScience and federated hybrid clouds.     

Cloud monitoring: A survey

Nowadays, Cloud Computing is widely used to deliver services over the Internet for both technical and economical reasons. The number of Cloud-based services has increased rapidly and strongly in the last years, and so is increased the complexity of the infrastructures behind these services. To properly operate and manage such complex infrastructures effective and efficient monitoring is constantly needed.Many works in literature have surveyed Cloud properties, features, underlying technologies (e.g. virtualization), security and privacy. However, to the best of our knowledge, these surveys lack a detailed analysis of monitoring for the Cloud. To fill this gap, in this paper we provide a survey on Cloud monitoring. We start analyzing motivations for Cloud monitoring, providing also definitions and background for the following contributions. Then, we carefully analyze and discuss the properties of a monitoring system for the Cloud, the issues arising from such properties and how such issues have been tackled in literature. We also describe current platforms, both commercial and open source, and services for Cloud monitoring, underlining how they relate with the properties and issues identified before. Finally, we identify open issues, main challenges and future directions in the field of Cloud monitoring.¹     

Performance evaluation of NoSQL big-data applications using multi-formalism models

Starting with the birth of Web 2.0, the quantity of data managed by large-scale web services has grown exponentially, posing new challenges and infrastructure requirements. This has led to new programming paradigms and architectural choices, such as map-reduce and NoSQL databases, which constitute two of the main peculiarities of the specialized massively distributed systems referred to as Big Data architectures. The underlying computer infrastructures usually face complexity requirements, resulting from the need for efficiency and speed in computing over huge evolving data sets. This is achieved by taking advantage from the features of new technologies, such as the automatic scaling and replica provisioning of Cloud environments. Although performances are a key issue for the considered applications, few performance evaluation results are currently available in this field. In this work we focus on investigating how a Big Data application designer can evaluate the performances of applications exploiting the Apache Hive query language for NoSQL databases, built over a Apache Hadoop map-reduce infrastructure.This paper presents a dedicated modeling language and an application, showing first how it is possible to ease the modeling process and second how the semantic gap between modeling logic and the domain can be reduced, by means of vertical multiformalism modeling.     

A resource elasticity framework for QoS-aware execution of cloud applications

Cloud infrastructures consisting of heterogeneous resources are increasingly being utilized for hosting large-scale distributed applications from diverse users with discrete needs. The multifarious cloud applications impose varied demands for computational resources along with multitude of performance implications. Successful hosting of cloud applications necessitates service providers to take into account the heterogeneity existing in the behavior of users, applications and system resources while respecting the user’s agreed Quality of Service (QoS) criteria. In this work, we propose a QoS-Aware Resource Elasticity (QRE) framework that allows service providers to make an assessment of the application behavior and develop mechanisms that enable dynamic scalability of cloud resources hosting the application components. Experimental results conducted on the Amazon EC2 cloud clearly demonstrate the effectiveness of our approach while complying with the agreed QoS attributes of users.     

Optimal online control for sleep mode in green base stations

In this paper, we investigate network sleep mode schemes for reducing energy consumption of radio access networks. We first propose, using Markov Decision Processes (MDPs), an optimal controller that associates to each traffic an activation/deactivation policy that maximizes a multiple objective function of the Quality of Service (QoS) and the energy consumption. We focus on a practical implementation issue, namely the ping pong effect resulting in unnecessary ON/OFF oscillations, that may affect the stability of the system. We illustrate our results numerically using theoretical models of the radio access network, and apply the developed mechanisms on a large-scale network simulator. Knowing that an offline optimization is not suitable for a large-scale network nor does it fit all traffic configurations, we propose, using an online controller that derives dynamically the optimal policy based on the dynamics of users in the cell. The design of our online controller is based on a simple ∊-greedy algorithm and learns the optimal threshold policy for activation/deactivation of network resources.     

Searching for Software on the EGEE Infrastructure

Several large-scale Grid infrastructures are currently in operation around the world, federating an impressive collection of computational resources, a wide variety of application software, and hundreds of user communities. To better serve the current and prospective users of Grid infrastructures, it is important to develop advanced software retrieval services that could help users locate software components suitable to their needs. In this paper, we present the design and implementation of Minersoft, a distributed, multi-threaded harvester for application software located in large-scale Grid infrastructures. Minersoft crawls the sites of a Grid infrastructure, discovers installed software resources, annotates them with keyword-rich metadata, and creates inverted indexes that can be used to support full-text software retrieval. We present insights derived from the implementation and deployment of Minersoft on EGEE, one of the largest Grid production services currently in operation. Experimental results show that Minersoft achieves a high performance in crawling EGEE sites and discovering software-related files, and a high efficiency in supporting software retrieval.     

Energy consumption analysis of data stream processing: a benchmarking approach

Energy efficiency of data analysis systems has become a very important issue in recent times because of the increasing costs of data center operations. Although distributed streaming workloads have increasingly been present in modern data centers, energy‐efficient scheduling of such applications remains as a significant challenge. In this paper, we conduct an energy consumption analysis of data stream processing systems in order to identify their energy consumption patterns. We follow stream system benchmarking approach to solve this issue. Specifically, we implement Linear Road benchmark on six stream processing environments (S4, Storm, ActiveMQ, Esper, Kafka, and Spark Streaming) and characterize these systems' performance on a real‐world data center. We study the energy consumption characteristics of each system with varying number of roads as well as with different types of component layouts. We also use a microbenchmark to capture raw energy consumption characteristics. We observed that S4, Esper, and Spark Streaming environments had highest average energy consumption efficiencies compared with the other systems. Using a neural networkbased technique with the power/performance information gathered from our experiments, we developed a model for the power consumption behavior of a streaming environment. We observed that energy‐efficient execution of streaming application cannot be specifically attributed to the system CPU usage. We observed that communication between compute nodes with moderate tuple sizes and scheduling plans with balanced system overhead produces better power consumption behaviors in the context of data stream processing systems. Copyright © 2016 John Wiley & Sons, Ltd.     

Dynamic grid quorum: a reconfigurable grid quorum and its power optimization algorithm

Power-saving has become a central issue for well-configured SOC platforms. In particular, as a high percentage of the total energy is used by the storage systems, the cost effectiveness of data management is equally as important as reliability and availability. To address this issue, we propose the dynamic grid quorum as a method for reducing the power consumption of large-scale distributed storage systems. The basic principle of our approach is to skew the workload toward a small number of quorums. This can be realized using the following three techniques. First, our system allows reconfiguration by exchanging nodes without any data migration, so that high-capacity nodes can be reallocated to busier quorums. Second, for more effective skewing of the workload, we introduce the notion of dual allocation, which makes it possible to consider two distinct allocations in the same grid for write and read quorums. Finally, we present an optimization algorithm to find a pair of a strategy and an allocation of nodes, which minimizes power for a given system setting and its workload. We also demonstrate that the dynamic grid quorum saves, on average, 14–25% energy compared with static configurations, when the intensity of the total workload changes.     

Infrastructure Federation Through Virtualized Delegation of Resources and Services

Infrastructure federation is becoming an increasingly important issue for modern Distributed Computing Infrastructures (DCIs): Dynamic elasticity of quasi-static Grid environments, incorporation of special-purpose resources into commoditized Cloud infrastructures, cross-community collaboration for increasingly diverging areas of modern e-Science, and Cloud Bursting pose major challenges on the technical level for many resource and middleware providers. Especially with respect to increasing costs of operating data centers, the intelligent yet automated and secure sharing of resources is a key factor for success. With the D-Grid Scheduler Interoperability (DGSI) project within the German D-Grid Initiative, we provide a strategic technology for the automatically negotiated, SLA-secured, dynamically provisioned federation of resources and services for Grid-and Cloud-type infrastructures. This goal is achieved by complementing current DCI schedulers with the ability to federate infrastructure for the temporary leasing of resources and rechanneling of workloads. In this work, we describe the overall architecture and SLA-secured negotiation protocols within DGSI and depict an advanced mechanism for resource delegation through means of dynamically provisioned, virtualized middleware. Through this methodology, we provide the technological foundation for intelligent capacity planning and workload management in a cross-infrastructure fashion.     

面向云计算环境的能耗测量和管理方法

云计算引领了计算机科学的一场重大变革,但与此同时,也不可避免地带来了日益凸显的能源消耗问题,因此,云计算能耗管理成为近几年的研究热点.云计算系统的能耗测量和管理直接关系到云计算的可持续发展,能耗数据不仅关系到能耗模型的建立,而且也是检验云计算资源调度算法的基础.为此,在广泛研究现有能耗测量方法的基础上,归纳总结了当前云计算环境的4种能耗测量方法:基于软件或硬件的直接测量方法、基于能耗模型的估算方法、基于虚拟化技术的能耗测量方法、基于仿真的能耗评估方法,并分析和比较了它们的优势、缺陷和适用环境.在此基础上,指出了云计算能耗管理的未来重要研究趋势:智能主机电源模块、面向不同类型应用的能耗模型、混合任务负载的能耗模型、可动态管理的高效云仿真工具、动态异构分布式集群的能耗管理、面向大数据分析处理和任务调度的节能方法以及新能源供电环境下的节能规划,为云计算节能领域的研究指明了方向.     

A hybrid HPC/cloud distributed infrastructure: Coupling EC2 cloud resources with HPC clusters to run large tightly coupled multiscale applications

In this paper, we report on the experimental results of running a large, tightly coupled, distributed multiscale computation over a hybrid High Performance Computing (HPC) infrastructures. We connected EC2 based cloud clusters located in USA to university clusters located in Switzerland. We ran a concurrent multiscale MPI based application on this infrastructure and measured the overhead induced by extending our HPC clusters with EC2 resources. Our results indicate that accommodating some parts of the multiscale computation on cloud resources can lead to low performance without a proper adjustment of CPUs power and workload. However, by enforcing a load-balancing strategy one can benefit from the extra Cloud resources.