Model-based reinforcement learning for nonlinear optimal control with practical asymptotic stability guarantees

We propose a new reinforcement learning approach for nonlinear optimal control where the value function is updated as restricted to control Lyapunov function (CLF) and the policy is improved using a variation of Sontag's formula. The practical asymptotic stability of the closed-loop system is guaranteed during the training and at the end of training without requiring an additional actor network and its update rule. For a single-layer neural network (NN) with exact basis functions, the approximate function converges to the optimal value function, resulting in the optimal controller. When a deep NN is used, the level set shapes of the trained NN become similar to those of the optimal value function. Because Sontag's formula with CLF is equivalent to the optimal controller when the given CLF has the same level set shapes as the optimal value function, Sontag's formula with the trained NN provides a nearly optimal controller.