基于Q学习算法的发电商报价策略模型

针对日前电力市场发电商利益最大化问题,利用Q学习算法构造了发电商策略报价模型。以发电商即时收益和市场相对占有率组成奖赏函数,使发电商策略收益最大化并同时达到提高市场占有率的目的。通过算例验证了模型的有效性,发现如果发电商试图提高市场占有率将选择低报价策略;考虑爬坡限制后使得24 h独立的Q学习联立,引起发电商报价策略变化。     

Self-learning energy management for plug-in hybrid electric bus considering expert experience and generalization performance

A self-learning energy management is proposed for plug-in hybrid electric bus, by combining Q-Learning (QL) and Pontryagin's minimum principle algorithms. Different from the existing strategies, the expert experience and generalization performance are focused in the proposed strategy. The expert experience is designed as the approximately optimal reference state-of-charge (SOC) trajectories, and the generalization performance is enhanced by a multiply driving cycle training method. In specific, an efficient zone of SOC is firstly designed based on the approximately optimal reference SOC trajectories. Then, the agent of the QL is trained off-line by taking the expert experience as reference SOC trajectories. Finally, an adaptive strategy is proposed based on the well-trained agent. Specially, two different reward functions are defined. That is, the reward function in the off-line training mainly considers the tracking performance between the expert experience and the SOC, while mainly considering the punishment in the adaptive strategy. Simulation results show that the proposed strategy has good generalization performance and can improve the fuel economy by 22.49%, compared to a charge depleting-charge sustaining (CDCS) strategy.     

非泊松工件流CSPS系统的Q学习算法适用性仿真研究

研究工件非泊松到达情况下, 传送带给料加工站(CSPS)系统无法建立成半马尔可夫决策过程(SMDP)模型 时, Q学习算法的适用性问题. 首先, 以马尔可夫调制泊松过程(MMPP)和半马尔可夫调制泊松过程(SMMPP)来模拟 非泊松工件流, 并在相同的平均到达率下, 仿真评估其Q学习算法性能, 并分别与泊松工件流情况下的Q学习算法性 能进行比较: 其次, 在非泊松工件流情况下, 观测以实时统计平均到达率作为工件标准泊松到达率的理论优化情况: 最后讨论在MMPP和SMMPP叠加混合非泊松工件流情况下CSPS 系统的Q学习算法性能. 实验表明, 在工件非泊松 到达情况下Q学习算法依然能学到较好的控制策略, 从而说明了CSPS系统中Q学习算法的适用性.     

CNR中基于多用户Q学习的联合信道选择和功率控制

针对认知无线网络中多用户资源分配时需要大量信道和功率策略信息交互,并且占用和耗费了大规模系统资源的问题,通过非合作博弈模型对用户的策略进行了研究,提出一种基于多用户Q学习的联合信道选择和功率控制算法。用户在自学习过程中将采用统一的策略,仅通过观察自己的回报来进行Q学习,并逐渐收敛到最优信道和功率分配的最优集合。仿真结果表明,该算法可以高概率地收敛到纳什均衡,用户通过信道选择得到的整体回报非常接近最大整体回报值。     

Fast Online Q(λ)

Q()-learning uses TD()-methods to accelerate Q-learning. The update complexity of previous online Q() implementations based on lookup tables is bounded by the size of the state/action space. Our faster algorithm's update complexity is bounded by the number of actions. The method is based on the observation that Q-value updates may be postponed until they are needed.     

Colearning in Differential Games

Game playing has been a popular problem area for research in artificial intelligence and machine learning for many years. In almost every study of game playing and machine learning, the focus has been on games with a finite set of states and a finite set of actions. Further, most of this research has focused on a single player or team learning how to play against another player or team that is applying a fixed strategy for playing the game. In this paper, we explore multiagent learning in the context of game playing and develop algorithms for co-learning in which all players attempt to learn their optimal strategies simultaneously. Specifically, we address two approaches to colearning, demonstrating strong performance by a memory-based reinforcement learner and comparable but faster performance with a tree-based reinforcement learner.     

基于Q学习算法的作业车间动态调度

近年来, 在基于Q学习算法的作业车间动态调度系统中, 状态-行动和奖励值靠人为主观设定, 导致学习效果不理想, 与已知最优解相比, 结果偏差较大. 为此, 基于作业车间调度问题的特质, 对Q学习算法的要素进行重新设计, 并用标准算例库进行仿真测试. 将结果先与已知最优解和混合灰狼优化算法、离散布谷鸟算法和量子鲸鱼群算法在近似程度、最小值方面进行比较分析. 实验结果表明, 与国内求解作业车间调度问题的Q学习算法相比, 该方法在最优解的近似程度上显著提升, 与群智能算法相比, 在大多数算例中, 寻优能力方面有显著提升.     

基于Q学习的单路口交通信号协调控制

Q学习通过与外部环境的交互来进行单路口的交通信号自适应控制。在城市交通愈加拥堵的时代背景下,为了缓解交通拥堵,提出一种结合SCOOT系统对绿信比优化方法的Q学习算法。本文将SCOOT系统中对绿信比优化的方法与Q学习相结合,即通过结合车均延误率以及停车次数等时间因素以及经济因素2方面,建立新的数学模型来作为本算法的成本函数并建立一种连续的奖惩函数,在此基础上详细介绍Q学习算法在单路口上的运行过程并且通过与Webster延误率和基于最小车均延误率的Q学习进行横向对比,验证了此算法优于定时控制以及基于车均延误的Q学习算法。相对于这2种算法,本文提出的算法更加适合单路口的绿信比优化。     

强化学习在自适应视频码率控制算法中的应用

视频码率自适应是提高视频服务质量的一种有效方法.现有视频码率自适应算法大多都试图将一套相对固定的模型规则应用于所有用户,无法确保所有用户都拥有良好的QoE.针对上述问题,本文提出一种基于强化学习的自适应视频码率调节方案,可以根据不同的网络条件实时调整视频码率算法的参数.该方案通过强化学习方法,提高了学习收敛速度,限制了次优选择,并且不会因为快速收敛而降低视频码率调节的效率.实验结果表明,基于强化学习的自适应视频码率调节方案与Bola,M PC等传统视频ABR算法相比,总体平均比特率提高了大约8.3%,这得益于本文提出的方案能够更好地在不同的网络状态下对视频码率调节行为进行细粒度定制和优化.     

基于最优初始值Q学习的电力信息网络防御策略学习算法

电力信息网络的安全与稳定是当今社会发展的重要保障,随着电力信息网络越来越庞大和复杂,如何高效合理地建立电力信息防护网络成为研究人员关注的重点之一。在自动化电力信息网络中,其防御策略通常缺乏统筹管理,只能针对少数设备进行防护,存在着更新速度慢、更新周期长、无法自动更新和资源分配不均等问题。本文提出一种基于最优初始值Q学习的电力信息网络防御策略学习算法,该算法以强化学习中的Q学习算法为框架,利用生成对抗网络思想,通过攻击智能体和防御智能体的模拟对抗学习安全策略。算法中的防御智能体使用Q学习方法更新其防御策略,利用历史防御经验在线改进防御策略,避免了人为手动操作。在训练中引入最优初始值极大加快了系统防御性能的训练速度。实验结果验证了算法的有效性。