基于深度强化学习的自适应虚拟机整合方法

能耗限制的服务质量优化问题一直以来都是数据中心虚拟机资源管理所面临的巨大挑战之一.尽管现有的工作通过虚拟机整合技术一定程度上降低了能耗和提升了系统服务质量，但这些方法通常难以实现长期最优的管理目标，并且容易受到业务场景变化的影响，面临变更困难以及管理成本高等难题.针对数据中心虚拟机资源管理存在的能耗和服务质量长期最优难保证以及策略调整灵活性差的问题，提出了一种基于深度强化学习的自适应虚拟机整合方法(deep reinforcement learning-based adaptive virtual machine consolidation method, RA-VMC).该方法利用张量化状态表示、确定性动作输出、卷积神经网络和加权奖赏机制构建了从数据中心系统状态到虚拟机迁移策略的端到端决策模型；设计自动化状态生成机制和反向梯度限定机制以改进深度确定性策略梯度算法，加快虚拟机迁移决策模型的收敛速度并且保证近似最优的管理性能.基于真实虚拟机负载数据的仿真实验结果表明：与开源云平台中流行的虚拟机整合方法相比，该方法能够有效地降低能耗和提高系统的服务质量.

Adaptive Virtual Machine Consolidation Method Based on Deep Reinforcement Learning

The problem of service quality optimization with energy consumption restriction has always been one of the big challenges for virtual machine (VM) resource management in data centers. Although existing work has reduced energy consumption and improved system service quality to a certain extent through VM consolidation technology, these methods are usually difficult to achieve long-term optimal management goals. Moreover, their performance is susceptible to the change of application scenarios, such that they are difficult to be replaced and will produce much management cost. In view of the problem that VM resource management in data center is hard to achieve long-term optimal energy efficiency and service quality, and also has poor flexibility in policy adjustment, this paper proposes an adaptive VM consolidation method based on deep reinforcement learning. This method builds an end-to-end decision-making model from data center system state to VM migration strategy through state tensor representation, deterministic action output, convolution neural network and weighted reward mechanism; It also designs an automatic state generation mechanism and an inverting gradient limitation mechanism to improve deep deterministic strategy gradient algorithm, speed up the convergence speed of VM migration decision-making model, and guarantee the approximately optimal management performance. Simulation experiment results based on real VM load data show that compared with popular VM consolidation methods in open source cloud platforms, this method can effectively reduce energy consumption and improve system service quality.