In this paper, we present a kernel-based least squares policy iteration (KLSPI) algorithm for reinforcement learning (RL) in large or continuous state spaces, which can be used to realize adaptive feedback control of uncertain dynamic systems. By using KLSPI, near-optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants. In KLSPI, Mercer kernels are used in the policy evaluation of a policy iteration process, where a new kernel-based least squares temporal-difference algorithm called KLSTD-Q is proposed for efficient policy evaluation. To keep the sparsity and improve the generalization ability of KLSTD-Q solutions, a kernel sparsification procedure based on approximate linear dependency (ALD) is performed. Compared to the previous works on approximate RL methods, KLSPI makes two progresses to eliminate the main difficulties of existing results. One is the better convergence and (near) optimality guarantee by using the KLSTD-Q algorithm for policy evaluation with high precision. The other is the automatic feature selection using the ALD-based kernel sparsification. Therefore, the KLSPI algorithm provides a general RL method with generalization performance and convergence guarantee for large-scale Markov decision problems (MDPs). Experimental results on a typical RL task for a stochastic chain problem demonstrate that KLSPI can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration (LSPI) algorithm. Furthermore, the KLSPI method was also evaluated on two nonlinear feedback control problems, including a ship heading control problem and the swing up control of a double-link underactuated pendulum called acrobot. Simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems. It is also demonstrated that KLSPI can be applied to online learning control by incorporating an initial controller to ensure online performance. 相似文献
Multimedia Tools and Applications - The graph convolutional network (GCN), which can handle graph-structured data, is enjoying great interest in recent years. However, while GCN achieved remarkable... 相似文献
With superior biocompatibility and unique magnetic properties, iron-based nanoparticles (IBNP) are commonly encapsulated in cells and extracellular vesicles (EV) to allow for magnetic force controlled drug delivery and non-invasive tracking. Based on their natural source and similar morphology, we classify both cells and EVs as being natural lipid encapsulations (NLEs), distinguishing them from synthetic liposomes. Both their imaging contrast and drug effects are dominated by the amount of iron encapsulated in each NLE, demonstrating the importance of magnetic labeling efficiency. It is known that the membranes function as barriers to ensure that substances pass in and out in an orderly manner. The most important issue in increasing the cellular uptake of IBNPs is the interaction between the NLE membrane and IBNPs, which has been found to be affected by properties of the IBNPs as well as NLE heterogeneity. Two aspects are important for effective magnetic labelling: First, how to effectively drive membrane wrapping of the nanoparticles into the NLEs, and second, how to balance biosafety and nanoparticle uptake. In this review, we will provide a systematic overview of the magnetic labeling of NLEs with IBNPs. This article provides a summary of the applications of magnetically labeled NLEs and the labeling methods used for IBNPs. The review also analyzes the role of IBNPs physicochemical properties, especially their magnetic properties, and the heterogeneity of NLEs in the internalization pathway. At the same time, the future development of magnetically labeled NLEs is also discussed.
