采用分类经验回放的深度确定性策略梯度方法EI北大核心CSCD

深度确定性策略梯度(Deep deterministic policy gradient,DDPG)方法在连续控制任务中取得了良好的性能表现.为进一步提高深度确定性策略梯度方法中经验回放机制的效率,提出分类经验回放方法,并采用两种方式对经验样本分类:基于时序差分误差样本分类的深度确定性策略梯度方法(DDPG with temporal difference-error classification,TDCDDPG)和基于立即奖赏样本分类的深度确定性策略梯度方法(DDPG with reward classification,RC-DDPG).在TDCDDPG和RC-DDPG方法中,分别使用两个经验缓冲池,对产生的经验样本按照重要性程度分类存储,网络模型训练时通过选取较多重要性程度高的样本加快模型学习.在连续控制任务中对分类经验回放方法进行测试,实验结果表明,与随机选取经验样本的深度确定性策略梯度方法相比,TDC-DDPG和RC-DDPG方法具有更好的性能.

Deep Deterministic Policy Gradient With Classified Experience Replay

The deep deterministic policy gradient (DDPG) algorithm achieves good performance in continuous control tasks. In order to further improve the efficiency of the experience replay mechanism in the DDPG algorithm, a method of classifying the experience replay is proposed, where transitions are classified in two branches: deep deterministic policy gradient with temporal difference-error classification (TDC-DDPG) and deep deterministic policy gradient with reward classification (RC-DDPG). In both methods, two replay buffers are introduced respectively to classify the transitions according to the degree of importance. Learning can be speeded up in network model training period by selecting a greater number of transitions with higher importance. The classification experience replay method has been tested in a series of continuous control tasks and experimental results show that the TDC-DDPG and RC-DDPG methods have better performance than the DDPG method with random selection of transitions.