用于连续时间中策略梯度算法的动作稳定更新算法

在强化学习中，策略梯度法经常需要通过采样将连续时间问题建模为离散时间问题。为了建模更加精确，需要提高采样频率，然而过高的采样频率可能会使动作改变频率过高，从而降低训练效率。针对这个问题，提出了动作稳定更新算法。该方法使用策略函数输出的改变量计算动作重复的概率，并根据该概率随机地重复或改变动作。在理论上分析了算法性能。之后在九个不同的环境中评估算法的性能，并且将它和已有方法进行了比较。该方法在其中六个环境下超过了现有方法。实验结果表明，动作稳定更新算法可以有效提高策略梯度法在连续时间问题中的训练效率。

Action stable updating algorithm for policy gradient methods in continuous time

In reinforcement learning, the policy gradient algorithm often needs to model the continuous-time process as a discrete-time process through sampling. To model the problem more accurately, it improved the sampling frequency. However, the excessive sampling frequency may reduce the training efficiency. To solve this problem, this paper proposed action stable updating algorithm. This method calculated the probability of action repetition using the change of the output of the policy function, and randomly repeated or changed the action based on this probability. This paper theoretically analyzed the performance of this method. This paper evaluated the performance of this method in 9 different environments and compared it with the existing methods. This method surpassed existing methods in six of these environments. The experimental results show that this method can improve the training efficiency of the policy gradient algorithm in continuous-time problems.