基于强化学习的值迭代算法

强化学习(Reinforcement Learning)是学习环境状态到动作的一种映射,并且能够获得最大的奖赏信号。强化学习中有三种方法可以实现回报的最大化:值迭代、策略迭代、策略搜索。该文介绍了强化学习的原理、算法,并对有环境模型和无环境模型的离散空间值迭代算法进行研究,并且把该算法用于固定起点和随机起点的格子世界问题。实验结果表明,相比策略迭代算法,该算法收敛速度快,实验精度好。     

一种高斯过程的带参近似策略迭代算法

在大规模状态空间或者连续状态空间中,将函数近似与强化学习相结合是当前机器学习领域的一个研究热点;同时,在学习过程中如何平衡探索和利用的问题更是强化学习领域的一个研究难点.针对大规模状态空间或者连续状态空间、确定环境问题中的探索和利用的平衡问题,提出了一种基于高斯过程的近似策略迭代算法.该算法利用高斯过程对带参值函数进行建模,结合生成模型,根据贝叶斯推理,求解值函数的后验分布.在学习过程中,根据值函数的概率分布,求解动作的信息价值增益,结合值函数的期望值,选择相应的动作.在一定程度上,该算法可以解决探索和利用的平衡问题,加快算法收敛.将该算法用于经典的Mountain Car 问题,实验结果表明,该算法收敛速度较快,收敛精度较好.     

一种优先级扫描的Dyna结构优化算法

不确定环境的时序决策问题是强化学习研究的主要内容之一,agent的目标是最大化其与环境交互过程中获得的累计奖赏值.直接学习方法寻找最优策略的算法收敛效率较差,而采用Dyna结构将学习与规划并行集成,可提高算法的收敛效率.为了进一步提高传统Dyna结构的收敛速度和收敛精度,提出了Dyna-PS算法,并在理论上证明了其收敛性.该算法在Dyna结构规划部分使用优先级扫描算法的思想,对优先级函数值高的状态优先更新,剔除了传统值迭代、策略迭代过程中不相关和无更新意义的状态更新,提升了规划的收敛效率,从而进一步提升了Dyna结构算法的性能.将此算法应用于一系列经典规划问题,实验结果表明,Dyna-PS算法有更快的收敛速度和更高的收敛精度,且对于状态空间的增长具有较强的鲁棒性.     

基于分层强化学习及人工势场的多Agent路径规划方法

针对路径规划算法收敛速度慢及效率低的问题,提出了一种基于分层强化学习及人工势场的多Agent路径规划算法。首先,将多Agent的运行环境虚拟为一个人工势能场,根据先验知识确定每点的势能值,它代表最优策略可获得的最大回报;其次,利用分层强化学习方法的无环境模型学习以及局部更新能力将策略更新过程限制在规模较小的局部空间或维度较低的高层空间上,提高学习算法的性能;最后,针对出租车问题在栅格环境中对所提算法进行了仿真实验。为了使算法贴近真实环境,增加算法的可移植性,在三维仿真环境中对该算法进行验证,实验结果表明该算法收敛速度快,收敛过程稳定。     

基于鸽群的鲁棒强化学习算法

强化学习是一种人工智能算法,具有计算逻辑清晰、模型易扩展的优点,可以在较少甚至没有先验信息的前提下,通过和环境交互并最大化值函数,调优策略性能,有效地降低物理模型引起的复杂性。基于策略梯度的强化学习算法目前已成功应用于图像智能识别、机器人控制、自动驾驶路径规划等领域。然而强化学习高度依赖采样的特性决定了其训练过程需要大量样本来收敛,且决策的准确性易受到与仿真环境中不匹配的轻微干扰造成严重影响。特别是当强化学习应用于控制领域时,由于无法保证算法的收敛性,难以对其稳定性进行证明,为此,需要对强化学习进行改进。考虑到群体智能算法可通过群体协作解决复杂问题,具有自组织性及稳定性强的特征,利用其对强化学习进行优化求解是一个提高强化学习模型稳定性的有效途径。结合群体智能中的鸽群算法,对基于策略梯度的强化学习进行改进：针对求解策略梯度时存在迭代求解可能无法收敛的问题,提出了基于鸽群的强化学习算法,以最大化未来奖励为目的求解策略梯度,将鸽群算法中的适应性函数和强化学习结合估计策略的优劣,避免求解陷入死循环,提高了强化学习算法的稳定性。在具有非线性关系的两轮倒立摆机器人控制系统上进行仿真验证,实验结果表...     

一种基于Off-Policy的无模型输出数据反馈H∞ 控制方法

针对模型未知的线性离散系统在扰动存在条件下的调节控制问题, 提出了一种基于Off-policy的输入输出数据反馈的H_∞控制方法. 本文从状态反馈在线学习算法出发, 针对系统运行过程中状态数据难以测得的问题, 通过引入增广数据向量将状态反馈策略迭代在线学习算法转化为输入输出数据反馈在线学习算法. 更进一步, 通过引入辅助项的方法将输入输出数据反馈策略迭代在线学习算法转化为无模型输入输出数据反馈Off-policy学习算法. 该算法利用历史输入输出数据实现最优输出反馈策略的学习, 同时克服了On-policy算法需要频繁与实际环境进行交互这一缺点. 除此之外, 与On-policy算法相比, Off-policy学习算法具有克服学习噪声的影响, 使学习结果收敛于理论最优值这一优点. 最终, 通过仿真实验验证了学习算法的收敛性.     

非完全信息下基于PPO-CFR的扩展式博弈决策

非完全信息下的人机对抗通常可以通过双人零和博弈模型加以描述,反事实后悔最小化(counterfactual regret minimization, CFR)是处理非完全信息双人零和博弈的一种流行算法.然而现有CFR及其变体算法在迭代过程中使用固定的后悔值计算和策略更新类型,在非完全信息扩展式博弈下表现各有优劣,泛化性能薄弱.针对这一问题,本文将强化学习近端策略优化(proximal policy optimization,PPO)算法与CFR算法相结合,提出一种PPO-CFR算法,通过训练出理性的智能体,从而实现CFR迭代过程后悔值计算和策略更新类型的自适应选择,以提高算法的泛化性能,并实现非完全信息扩展式博弈的策略优化.本文采用通用的扑克博弈实验验证所提算法,并制定逐步奖励函数训练智能体的动作策略,实验结果表明,与现有方法相比, PPO-CFR算法具有更好的泛化性能和更低的可利用度,迭代策略更为逼近纳什均衡策略.     

双Q网络学习的迁移强化学习算法

深度强化学习在训练过程中会探索大量环境样本,造成算法收敛时间过长,而重用或传输来自先前任务(源任务)学习的知识,对算法在新任务(目标任务)的学习具有提高算法收敛速度的潜力.为了提高算法学习效率,提出一种双Q网络学习的迁移强化学习算法,其基于actor-critic框架迁移源任务最优值函数的知识,使目标任务中值函数网络对策略作出更准确的评价,引导策略快速向最优策略方向更新.将该算法用于Open AI Gym以及在三维空间机械臂到达目标物位置的实验中,相比于常规深度强化学习算法取得了更好的效果,实验证明提出的双Q网络学习的迁移强化学习算法具有较快的收敛速度,并且在训练过程中算法探索更加稳定.     

基于加权值函数分解的多智能体分层强化学习技能发现方法

针对目前大多数多智能体强化学习算法在智能体数量增多以及环境动态不稳定的情况下导致的维度爆炸和奖励稀疏的问题，提出了一种基于加权值函数分解的多智能体分层强化学习技能发现算法。首先，该算法将集中训练分散执行的架构与分层强化学习相结合，在上层采用加权值函数分解的方法解决智能体在训练过程中容易忽略最优策略而选择次优策略的问题；其次，在下层采用独立Q学习算法使其能够在多智能体环境中分散式地处理高维复杂的任务；最后，在底层独立Q学习的基础上引入技能发现策略，使智能体之间相互学习互补的技能。分别在简易团队运动和星际争霸Ⅱ两个仿真实验平台上对该算法与多智能体强化学习算法和分层强化学习算法进行对比，实验表明，该算法在奖励回报以及双方对抗胜率等性能指标上都有所提高，提升了整个多智能体系统的决策能力和收敛速度，验证了算法的可行性。     

基于联邦学习的无线网络节点能量与信息管理策略

在无线通信网络环境中,分布式客户端节点在用户隐私保护、数据传输效率、能量利用效率之间较难实现平衡。针对该问题,提出一种结合联邦学习与传统集中式学习的能量与信息管理优化策略。以覆盖性强、适用性广的移动信息采集设备作为学习服务器,将分布分散、资源受限的客户端节点作为学习参与者,通过构建马尔科夫决策模型分析客户端节点在移动信息采集过程中的状态变化和行为模式,同时采用值迭代算法和深度强化学习算法对该模型进行近似求解,获得客户端节点最优的信息传输与能量管理组合策略。仿真结果表明,相比MDP、GRE、RAN策略,该策略的长期效用较高且数据延迟较小,可实现客户端节点在信息传输过程中的数据隐私性、数据可用性与能量消耗之间的最优平衡。     

Preference-based reinforcement learning: a formal framework and a policy iteration algorithm

This paper makes a first step toward the integration of two subfields of machine learning, namely preference learning and reinforcement learning (RL). An important motivation for a preference-based approach to reinforcement learning is the observation that in many real-world domains, numerical feedback signals are not readily available, or are defined arbitrarily in order to satisfy the needs of conventional RL algorithms. Instead, we propose an alternative framework for reinforcement learning, in which qualitative reward signals can be directly used by the learner. The framework may be viewed as a generalization of the conventional RL framework in which only a partial order between policies is required instead of the total order induced by their respective expected long-term reward. Therefore, building on novel methods for preference learning, our general goal is to equip the RL agent with qualitative policy models, such as ranking functions that allow for sorting its available actions from most to least promising, as well as algorithms for learning such models from qualitative feedback. As a proof of concept, we realize a first simple instantiation of this framework that defines preferences based on utilities observed for trajectories. To that end, we build on an existing method for approximate policy iteration based on roll-outs. While this approach is based on the use of classification methods for generalization and policy learning, we make use of a specific type of preference learning method called label ranking. Advantages of preference-based approximate policy iteration are illustrated by means of two case studies.     

Two-loop reinforcement learning algorithm for finite-horizon optimal control of continuous-time affine nonlinear systems

This article proposes three novel time-varying policy iteration algorithms for finite-horizon optimal control problem of continuous-time affine nonlinear systems. We first propose a model-based time-varying policy iteration algorithm. The method considers time-varying solutions to the Hamiltonian–Jacobi–Bellman equation for finite-horizon optimal control. Based on this algorithm, value function approximation is applied to the Bellman equation by establishing neural networks with time-varying weights. A novel update law for time-varying weights is put forward based on the idea of iterative learning control, which obtains optimal solutions more efficiently compared to previous works. Considering that system models may be unknown in real applications, we propose a partially model-free time-varying policy iteration algorithm that applies integral reinforcement learning to acquiring the time-varying value function. Moreover, analysis of convergence, stability, and optimality is provided for every algorithm. Finally, simulations for different cases are given to verify the convenience and effectiveness of the proposed algorithms.     

Average Reward Reinforcement Learning: Foundations,Algorithms, and Empirical Results

This paper presents a detailed study of average reward reinforcement learning, an undiscounted optimality framework that is more appropriate for cyclical tasks than the much better studied discounted framework. A wide spectrum of average reward algorithms are described, ranging from synchronous dynamic programming methods to several (provably convergent) asynchronous algorithms from optimal control and learning automata. A general sensitive discount optimality metric calledn-discount-optimality is introduced, and used to compare the various algorithms. The overview identifies a key similarity across several asynchronous algorithms that is crucial to their convergence, namely independent estimation of the average reward and the relative values. The overview also uncovers a surprising limitation shared by the different algorithms while several algorithms can provably generategain-optimal policies that maximize average reward, none of them can reliably filter these to producebias-optimal (orT-optimal) policies that also maximize the finite reward to absorbing goal states. This paper also presents a detailed empirical study of R-learning, an average reward reinforcement learning method, using two empirical testbeds: a stochastic grid world domain and a simulated robot environment. A detailed sensitivity analysis of R-learning is carried out to test its dependence on learning rates and exploration levels. The results suggest that R-learning is quite sensitive to exploration strategies and can fall into sub-optimal limit cycles. The performance of R-learning is also compared with that of Q-learning, the best studied discounted RL method. Here, the results suggest that R-learning can be fine-tuned to give better performance than Q-learning in both domains.     

基于模型的强化学习在无人机路径规划中的应用

针对当前强化学习算法在无人机升空平台路径规划任务中样本效率低、算法鲁棒性较差的问题,提出一种基于模型的内在奖励强化学习算法。采用并行架构将数据收集操作和策略更新操作完全解耦,提升算法学习效率,并运用内在奖励的方法提高智能体对环境的探索效率,避免收敛到次优策略。在策略学习过程中,智能体针对模拟环境的动态模型进行学习,从而在有限步内更好地预测状态、奖励等信息。在此基础上,通过结合有限步的规划计算以及神经网络的预测,提升价值函数的预测精准度,以利用较少的经验数据完成智能体的训练。实验结果表明,相比同样架构的无模型强化学习算法,该算法达到相同训练水平所需的经验数据量减少近600幕数据,样本效率和算法鲁棒性都有大幅提升,相比传统的非强化学习启发类算法,分数提升接近8 000分,与MVE等主流的基于模型的强化学习算法相比,平均分数可以提升接近2 000分,且在样本效率和稳定性上都有明显提高。     

H∞ tracking control for linear discrete‐time systems via reinforcement learning

In this paper, the H_∞ tracking control of linear discrete‐time systems is studied via reinforcement learning. By defining an improved value function, the tracking game algebraic Riccati equation with a discount factor is obtained, which is solved by iteration learning algorithms. In particular, Q‐learning based on value iteration is presented for H_∞ tracking control, which does not require the system model information and the initial allowable control policy. In addition, to improve the practicability of algorithm, the convergence analysis of proposed algorithm with a discount factor is given. Finally, the feasibility of proposed algorithms is verified by simulation examples.     

基于CMAC的非参数化近似策略迭代增强学习

为解决在线近似策略迭代增强学习计算复杂度高、收敛速度慢的问题，引入CMAC结构作为值函数逼近器，提出一种基于CMAC的非参数化近似策略迭代增强学习（NPAPI-CMAC）算法。算法通过构建样本采集过程确定CMAC泛化参数，利用初始划分和拓展划分确定CMAC状态划分方式，利用量化编码结构构建样本数集合定义增强学习率，实现了增强学习结构和参数的完全自动构建。此外，该算法利用delta规则和最近邻思想在学习过程中自适应调整增强学习参数，利用贪心策略对动作投票器得到的结果进行选择。一级倒立摆平衡控制的仿真实验结果验证了算法的有效性、鲁棒性和快速收敛能力。     

移动机器人路径规划强化学习的初始化

针对现有机器人路径规划强化学习算法收敛速度慢的问题,提出了一种基于人工势能场的移动机器人强化学习初始化方法.将机器人工作环境虚拟化为一个人工势能场,利用先验知识确定场中每点的势能值,它代表最优策略可获得的最大累积回报.例如障碍物区域势能值为零,目标点的势能值为全局最大.然后定义Q初始值为当前点的立即回报加上后继点的最大折算累积回报.改进算法通过Q值初始化,使得学习过程收敛速度更快,收敛过程更稳定.最后利用机器人在栅格地图中的路径对所提出的改进算法进行验证,结果表明该方法提高了初始阶段的学习效率,改善了算法性能.     

Natural actor-critic algorithms

We present four new reinforcement learning algorithms based on actor-critic, natural-gradient and function-approximation ideas, and we provide their convergence proofs. Actor-critic reinforcement learning methods are online approximations to policy iteration in which the value-function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent. Methods based on policy gradients in this way are of special interest because of their compatibility with function-approximation methods, which are needed to handle large or infinite state spaces. The use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates. The use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases. Our results extend prior two-timescale convergence results for actor-critic methods by Konda and Tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients. Our results extend prior empirical studies of natural actor-critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms.     

基于生成对抗网络的最大熵逆强化学习

针对逆强化学习算法在训练初期由于专家样本稀疏所导致的学习速率慢的问题,提出一种基于生成对抗网络（Generative Adversarial Networks,GAN）的最大熵逆强化学习算法。在学习过程中,结合专家样本训练优化生成对抗网络,以生成虚拟专家样本,在此基础上利用随机策略生成非专家样本,构建混合样本集,结合最大熵概率模型,对奖赏函数进行建模,并利用梯度下降方法求解最优奖赏函数。基于所求解的最优奖赏函数,利用正向强化学习方法求解最优策略,并在此基础上进一步生成非专家样本,重新构建混合样本集,迭代求解最优奖赏函数。将所提出的算法与MaxEnt IRL算法应用于经典的Object World与Mountain Car问题,实验表明,该算法在专家样本稀疏的情况下可以较好地求解奖赏函数,具有较好的收敛性能。