基于MetrOPOlis准则的Q-学习算法研究

探索与扩张是Q－学习算法中动作选取的关键问题，一味地扩张使智能体很快地陷入局部最优，虽然探索可以跳出局部最优并加速学习，而过多的探索将影响算法的性能，通过把Q－学习中寻求成策略表示为组合优化问题中最优解的搜索，将模拟退火算法的Mketropolis准则用于Q－学习中探索和扩张之间的折衷处理，提出基于Metropolis准则的Q－学习算法SA－Q-learning，通过比较，它具有更快的收敛速度，而且避免了过多探索引起的算法性能下降。

RESEARCH ON Q-LEARNING ALGORITHM BASED ON METROPOLIS CRITERION

The balance between exploration and exploitation is one of the key problems when action selection is performed in Q-learning. Pure exploitations will cause the agent to reach the local optimization quickly, whereas excessive explorations will degenerate the performance of the Q-learning algorithm even if they can accelerate learning process and can avoid the local optimization. In this paper, finding the optimum policy in Q-learning is described as searching optimum solution in combinatorial optimization. Then Metropolis criterion of simulated annealing algorithm is introduced in the balance between exploration and exploitation of Q-learning, and the Q-learning algorithm based on Metropolis criterion, SA-Q-learning, is correspondingly presented. Finally, tests show that SA-Q-learning converges more quickly than Q-learning, and can avoid the degeneracy in performance due to excessive explorations.