VMStore: Distributed storage system for multiple virtual machines

Desktop virtualization is a very hot concept in both industry and academic communities. Since virtualized desktop system is based on multiple virtual machines (VM), it is necessary to design a distributed storage system to manage the VM images. In this paper, we design a distributed storage system, VMStore, by taking into account three important characteristics: high performance VM snapshot, booting optimization from multiple images and redundancy removal of images data. We adopt a direct index structure of...     

基于任务分类的虚拟CPU调度模型

为了桥接语义鸿沟,提升I/O性能,需要对执行不同类型负载的虚拟CPU（vCPU）采取不同的调度策略,故而虚拟CPU调度算法亟需优化。基于KVM虚拟化平台提出一种基于任务分类的虚拟CPU调度模型STC（virtual CPU scheduler based on task classification）,它将虚拟CPU（vCPU）和物理CPU分别分为两个类型,分别为short vCPU和long vCPU,以及short CPU 和long CPU,不同类型的vCPU分配至对应类型的物理CPU上执行。同时,基于机器学习理论,STC构建分类器,通过提取任务行为特征将任务分为两类,I/O密集型的任务分配至short vCPU上,而计算密集型任务则分配至long vCPU上。STC在保证计算性能的基础上,提高了I/O的响应速度。实验结果表明,STC与系统默认的CFS相比,网络延时降低18%,网络吞吐率提高17%~25%,并且保证了整个系统的资源共享公平性。     

Large-scale virtual machines provisioning in clouds: challenges and approaches

The scale of global data center market has been explosive in recent years. As the market grows, the demand for fast provisioning of the virtual resources to support elastic, manageable, and economical computing over the cloud becomes high. Fast provisioning of large-scale virtual machines (VMs), in particular, is critical to guarantee quality of service (QoS). In this paper, we systematically review the existing VM provisioning schemes and classify them in three main categories. We discuss the features and research status of each category, and introduce two recent solutions, VMThunder and VMThunder+, both of which can provision hundreds of VMs in seconds.     

Platform-independent modeling and prediction of application resource usage characteristics

Application resource usage models can be used in the decision making process for ensuring quality-of-service as well as for capacity planning, apart from their general use in performance modeling, optimization, and systems management. Current solutions for modeling application resource usage tend to address parts of the problem by either focusing on a specific application, or a specific platform, or on a small subset of system resources. We propose a simple and flexible approach for modeling application resource usage in a platform-independent manner that enables the prediction of application resource usage on unseen platforms. The technique proposed is application agnostic, requiring no modification to the application (binary or source) and no knowledge of application-semantics. We implement a Linux-based prototype and evaluate it using four different workloads including real-world applications and benchmarks. Our experiments reveal prediction errors that are bound within 6-24% of the observed for these workloads when using the proposed approach.     

云数据中心虚拟资源管理研究综述

云计算数据中心是数据中心架构和发展的方向,云数据中心的虚拟资源管理问题是当前的研究热点。总结了数据中心的发展历程,综述了云数据中心的虚拟化技术、虚拟资源提供与部署、资源调度模型与算法以及基于能量优化和负载均衡的虚拟机迁移技术的研究现状。最后展望了云数据中心虚拟资源管理领域有待进一步研究的方向。     

Resource provisioning based on preempting virtual machines in distributed systems

Resource provisioning is one of the main challenges in large‐scale distributed systems such as federated Grids. Recently, many resource management systems in these environments have started to use the lease abstraction and virtual machines (VMs) for resource provisioning. In the large‐scale distributed systems, resource providers serve requests from external users along with their own local users. The problem arises when there is not sufficient resources for local users, who have higher priority than external ones, and need resources urgently. This problem could be solved by preempting VM‐based leases from external users and allocating them to the local ones. However, preempting VM‐based leases entails side effects in terms of overhead time as well as increasing makespan of external requests. In this paper, we model the overhead of preempting VMs. Then, to reduce the impact of these side effects, we propose and compare several policies that determine the proper set of lease(s) for preemption. We evaluate the proposed policies through simulation as well as real experimentation in the context of InterGrid under different working conditions. Evaluation results demonstrate that the proposed preemption policies serve up to 72% more local requests without increasing the rejection ratio of external requests. Copyright © 2013 John Wiley & Sons, Ltd.     

Failure-aware resource management for high-availability computing clusters with distributed virtual machines

In large-scale networked computing systems, component failures become norms instead of exceptions. Failure-aware resource management is crucial for enhancing system availability and achieving high performance. In this paper, we study how to efficiently utilize system resources for high-availability computing with the support of virtual machine (VM) technology. We design a reconfigurable distributed virtual machine (RDVM) infrastructure for networked computing systems. We propose failure-aware node selection strategies for the construction and reconfiguration of RDVMs. We leverage the proactive failure management techniques in calculating nodes’ reliability states. We consider both the performance and reliability status of compute nodes in making selection decisions. We define a capacity–reliability metric to combine the effects of both factors in node selection, and propose Best-fit algorithms with optimistic and pessimistic selection strategies to find the best qualified nodes on which to instantiate VMs to run user jobs. We have conducted experiments using failure traces from production systems and the NAS Parallel Benchmark programs on a real-world cluster system. The results show the enhancement of system productivity by using the proposed strategies with practically achievable accuracy of failure prediction. With the Best-fit strategies, the job completion rate is increased by 17.6% compared with that achieved in the current LANL HPC cluster. The task completion rate reaches 91.7% with 83.6% utilization of relatively unreliable nodes.     

DMM:A dynamic memory mapping model for virtual machines

Memory virtualization is an important part in the design of virtual machine monitors(VMM).In this paper,we proposed dynamic memory mapping(DMM) model,a mechanism that allows the VMM to change the mapping between a virtual machine's physical memory and the underlying hardware resource while the virtual machine is running.By utilizing DMM,the VMM can implement many novel memory management policies,such as Demand Paging,Swapping,Ballooning,Memory Sharing and Copy-On-Write,while preserving compatibility with va...     

Multi-Capacity Combinatorial Ordering GA in Application to Cloud resources allocation and efficient virtual machines consolidation

This paper describes a novel approach making use of genetic algorithms to find optimal solutions for multi-dimensional vector bin packing problems with the goal to improve cloud resource allocation and Virtual Machines (VMs) consolidation. Two algorithms, namely Combinatorial Ordering First-Fit Genetic Algorithm (COFFGA) and Combinatorial Ordering Next Fit Genetic Algorithm (CONFGA) have been developed for that and combined. The proposed hybrid algorithm targets to minimise the total number of running servers and resources wastage per server. The solutions obtained by the new algorithms are compared with latest solutions from literature. The results show that the proposed algorithm COFFGA outperforms other previous multi-dimension vector bin packing heuristics such as Permutation Pack (PP), First Fit (FF) and First Fit Decreasing (FFD) by 4%, 34%, and 39%, respectively. It also achieved better performance than the existing genetic algorithm for multi-capacity resources virtual machine consolidation (RGGA) in terms of performance and robustness. A thorough explanation for the improved performance of the newly proposed algorithm is given.     

Adaptive parallel application resource remapping through the live migration of virtual machines

In this paper we present ARRIVE-F, a novel open source framework which addresses the issue of heterogeneity in virtualized compute farms, such as those hosted by a cloud infrastructure provider. Unlike the previous attempts, our framework is not based on linear frequency models and does not require source code modifications or off-line profiling. The heterogeneous compute farm is first divided into a number of homogeneous sub-clusters. The framework then carries out a lightweight ‘online’ profiling of the CPU, communication and memory subsystems of all the active jobs in the compute farm. From this, it constructs a performance model to predict the execution times of each job on all the distinct sub-clusters in the compute farm. Based upon the predicted execution times, the framework is able to relocate the compute jobs to the currently best-suited hardware platforms such that the overall throughput of the compute farm is increased. We utilize the live migration feature of virtual machine monitors to migrate the job from one sub-cluster to another.The prediction accuracy of our performance estimation model is over 80%. The implementation of ARRIVE-F is lightweight, with an overhead of 3%. Experiments on a synthetic workload of scientific benchmarks show that we are able to improve the throughput of a moderately heterogeneous compute farm by up to 25%, with a time saving of up to 33%.     

Stochastic modeling and analysis of hybrid mobility in reconfigurable distributed virtual machines

Virtualization provides a vehicle to manage the available resources and enhance their utilization in network computing. System dynamics requires virtual machines be distributed and reconfigurable. To construct reconfigurable distributed virtual machines, service migration moves the runtime services among physical servers when necessary. By incorporating the mobile agent technology, distributed virtual machines can improve their resource utilization and service availability significantly. This paper focuses on finding the optimal migration policies for service and agent migrations for high throughput in reconfigurable distributed virtual machines. We analyze three issues of this decision problem: migration candidate determination, migration timing and destination server selection. The service migration timing and destination server selection are formulated by two optimization models. We derive the optimal migration policy for distributed and heterogenous systems based on stochastic optimization theories. Renewal processes are applied to model the dynamics of migration. We solve the agent migration problem by dynamic programming and extend the optimal service migration decision by considering the interplay of the hybrid mobility. We verify the accuracy of our migration decision policy in simulations.     

Matrix based proactive resource provisioning in mobile cloud environment

Mobile cloud computing is a dynamic, virtually scalable and network based computing environment where mobile device acts as a thin client and applications run on remote cloud servers. Mobile cloud computing resources required by different users depend on their respective personalized applications. Therefore, efficient resource provisioning in mobile clouds is an important aspect that needs special attention in order to make the mobile cloud computing a highly optimized entity. This paper proposes an adaptive model for efficient resource provisioning in mobile clouds by predicting and storing resource usages in a two dimensional matrix termed as resource provisioning matrix. These resource provisioning matrices are further used by an independent authority to predict future required resources using artificial neural network. Independent authority also checks and verifies resource usage bill computed by cloud service provider using resource provisioning matrices. It provides cost computation reliability for mobile customers in mobile cloud environment. Proposed model is implemented on Hadoop using three different applications. Results indicate that proposed model provides better mobile cloud resources utilization as well as maintains quality of service for mobile customer. Proposed model increases battery life of mobile device and decreases data usage cost for mobile customer.     

OpenStack Neat: a framework for dynamic and energy‐efficient consolidation of virtual machines in OpenStack clouds

Dynamic consolidation of virtual machines (VMs) is an efficient approach for improving the utilization of physical resources and reducing energy consumption in cloud data centers. Despite the large volume of research published on this topic, there are very few open‐source software systems implementing dynamic VM consolidation. In this paper, we propose an architecture and open‐source implementation of OpenStack Neat, a framework for dynamic VM consolidation in OpenStack clouds. OpenStack Neat can be configured to use custom VM consolidation algorithms and transparently integrates with existing OpenStack deployments without the necessity of modifying their configuration. In addition, to foster and encourage further research efforts in the area of dynamic VM consolidation, we propose a benchmark suite for evaluating and comparing dynamic VM consolidation algorithms. The proposed benchmark suite comprises OpenStack Neat as the base software framework, a set of real‐world workload traces, performance metrics and evaluation methodology. As an application of the proposed benchmark suite, we conduct an experimental evaluation of OpenStack Neat and several dynamic VM consolidation algorithms on a five‐node testbed, which shows significant benefits of dynamic VM consolidation resulting in up to 33% energy savings. Copyright © 2014 John Wiley & Sons, Ltd.     

Hybrid shuffled frog leaping algorithm for energy-efficient dynamic consolidation of virtual machines in cloud data centers

Cloud computing aims to provide dynamic leasing of server capabilities as scalable virtualized services to end users. However, data centers hosting cloud applications consume vast amounts of electrical energy, thereby contributing to high operational costs and carbon footprints. Green cloud computing solutions that can not only minimize the operational costs but also reduce the environmental impact are necessary. This study focuses on the Infrastructure as a Service model, where custom virtual machines (VMs) are launched in appropriate servers available in a data center. A complete data center resource management scheme is presented in this paper. The scheme can not only ensure user quality of service (through service level agreements) but can also achieve maximum energy saving and green computing goals. Considering that the data center host is usually tens of thousands in size and that using an exact algorithm to solve the resource allocation problem is difficult, the modified shuffled frog leaping algorithm and improved extremal optimization are employed in this study to solve the dynamic allocation problem of VMs. Experimental results demonstrate that the proposed resource management scheme exhibits excellent performance in green cloud computing.     

Deadline-driven provisioning of resources for scientific applications in hybrid clouds with Aneka

Scientific applications require large computing power, traditionally exceeding the amount that is available within the premises of a single institution. Therefore, clouds can be used to provide extra resources whenever required. For this vision to be achieved, however, requires both policies defining when and how cloud resources are allocated to applications and a platform implementing not only these policies but also the whole software stack supporting management of applications and resources. Aneka is a cloud application platform capable of provisioning resources obtained from a variety of sources, including private and public clouds, clusters, grids, and desktops grids. In this paper, we present Aneka’s deadline-driven provisioning mechanism, which is responsible for supporting quality of service (QoS)-aware execution of scientific applications in hybrid clouds composed of resources obtained from a variety of sources. Experimental results evaluating such a mechanism show that Aneka is able to efficiently allocate resources from different sources in order to reduce application execution times.     

Power-efficient assignment of virtual machines to physical machines

Motivated by current trends in cloud computing, we study a version of the generalized assignment problem where a set of virtual processors has to be implemented by a set of identical processors. For literature consistency, we say that a set of virtual machines (VMs) is assigned to a set of physical machines (PMs). The optimization criterion is to minimize the power consumed by all the PMs. We term the problem Virtual Machine Assignment (VMA). Crucial differences with previous work include a variable number of PMs, that each VM must be assigned to exactly one PM (i.e., VMs cannot be implemented fractionally), and a minimum power consumption for each active PM. Such infrastructure may be strictly constrained in the number of PMs or in the PMs’ capacity, depending on how costly (in terms of power consumption) it is to add a new PM to the system or to heavily load some of the existing PMs. Low usage or ample budget yields models where PM capacity and/or the number of PMs may be assumed unbounded for all practical purposes. We study four VMA problems depending on whether the capacity or the number of PMs is bounded or not. Specifically, we study hardness and online competitiveness for a variety of cases. To the best of our knowledge, this is the first comprehensive study of the VMA problem for this cost function.     

Empirical prediction models for adaptive resource provisioning in the cloud

Cloud computing allows dynamic resource scaling for enterprise online transaction systems, one of the key characteristics that differentiates the cloud from the traditional computing paradigm. However, initializing a new virtual instance in a cloud is not instantaneous; cloud hosting platforms introduce several minutes delay in the hardware resource allocation. In this paper, we develop prediction-based resource measurement and provisioning strategies using Neural Network and Linear Regression to satisfy upcoming resource demands.Experimental results demonstrate that the proposed technique offers more adaptive resource management for applications hosted in the cloud environment, an important mechanism to achieve on-demand resource allocation in the cloud.     

Grid Economics: A Selective Discussion of Two Research Problems

The last 5 years have seen considerable discussion of various types of Grids—compute Grids, storage Grids, and data Grids. Using the checklist given in Foster (, 2002) to define a Grid, two important problems that arise in the context of resource sharing in Grid computing environments are discussed. First, the well documented problem in compute Grid environments that arises from the inability of consumers to accurately estimate their resource requirements is presented. This results in incorrect scheduling of requests for Grid resources and social welfare loss. To address this problem, two research proposals are briefly described. The first approach argues for the design of decision support tools to help users with resource estimation while the second approach studies the design of resource allocation mechanisms that can work with stochastic specifications of resource requirements. This is in contrast to the traditional point estimates of resource required by extant mechanisms. Next, resource provisioning and pricing problems that arise in data storage and retrieval Grids are described. These Grids differ fundamentally from compute Grids but share some economic characteristics with P2P file sharing networks. Drawing on this connection, pricing mechanisms and resource provisioning research is briefly discussed.