多步R学习算法

目的讨论平均准则下控制马氏链的强化学习算法,在事先未知状态转移矩阵及报酬函数的条件下,通过试凑法寻找使得长期每阶段期望平均报酬最大的最优控制策略.方法结合平均报酬问题的一步学习算法和即时差分学习算法,提出了一种多步强化学习算法--R(λ)学习算法.结果与结论新算法使得已有的R学习算法成为其λ=0时的特例.同时它也是折扣报酬Q(λ)学习算法到平均报酬问题的自然推广.仿真结果表明λ取中间值的R(λ)学习算法明显优于一步的R学习算法.

Incremental Multi-Step R-Learning

Aim To investigate the model-free multi-step average reward reinforcement learning algorithm. Methods By combining the R-learning algorithms with the temporal-difference learning (TD(λ)-learning) algorithms for average reward problems, a novel incremental algorithm, called R(λ)-learning, was proposed. Results and Conclusion The proposed algorithm is a natural extension of the Q(λ)-learning, the multi-step discounted reward reinforcement learning algorithm, to the average reward cases. Simulation results show that the R(λ)-learning with intermediate λ values makes significant performance improvement over the simple R-learning.