基于AdaBelief的Heavy-Ball动量方法

同时使用动量和自适应步长技巧的自适应矩估计(Adaptive Moment Estimation,Adam)型算法广泛应用于深度学习中.针对此方法不能同时在理论和实验上达到最优这一问题,文中结合AdaBelief灵活调整步长提高实验性能的技巧,以及仅采用指数移动平均(Exponential Moving Average,EMA)策略调整步长的Heavy-Ball动量方法加速收敛的优点,提出基于AdaBelief的Heavy-Ball动量方法.借鉴AdaBelief和Heavy-Ball动量方法收敛性分析的技巧,巧妙选取时变步长、动量系数,并利用添加动量项和自适应矩阵的方法,证明文中方法对于非光滑一般凸优化问题具有最优的个体收敛速率.最后,在凸优化问题和深度神经网络上的实验验证理论分析的正确性,并且证实文中方法可在理论上达到最优收敛性的同时提高性能.

AdaBelief Based Heavy-Ball Momentum Method

Adaptive moment estimation algorithms with momentum and adaptive step techniques are widely applied in deep learning.However,these algorithms cannot achieve the optimal performance in both theory and experiment.To solve the problem,an AdaBelief based heavy-ball momentum method,AdaBHB,is proposed.The AdaBelief technique of adjusting step size flexibly is introduced to improve the algorithm performance in experiments.The heavy ball momentum method with step size adjusted by exponential moving average strategy is employed to accelerate convergence.According to the convergence analysis techniques of AdaBelief and Heavy-ball momentum methods,time-varying step size and momentum coefficient are selected skillfully and the momentum term and adaptive matrix are added.It is proved that AdaBHB gains the optimal individual convergence rate for non-smooth general convex optimization problems.Finally,the correctness of the theoretical analysis of the proposed algorithm is verified by experiments on convex optimization problems and deep neural networks,and AdaBHB is validated to obtain the optimal convergence in theory with performance improved.