连续状态自适应离散化基于K-均值聚类的强化学习方法

使用聚类算法对连续状态空间进行自适应离散化．得到了基于K-均值聚类的强化学习方法．该方法的学习过程分为两部分：对连续状态空间进行自适应离散化的状态空间学习,使用K-均值聚类算法;寻找最优策略的策略学习．使用替代合适迹Sarsa学习算法．对连续状态的强化学习基准问题进行仿真实验,结果表明该方法能实现对连续状态空间的自适应离散化,并最终学习到最优策略．与基于CMAC网络的强化学习方法进行比较．结果表明该方法具有节省存储空间和缩短计算时间的优点．     

Kernel-based least squares policy iteration for reinforcement learning.

In this paper, we present a kernel-based least squares policy iteration (KLSPI) algorithm for reinforcement learning (RL) in large or continuous state spaces, which can be used to realize adaptive feedback control of uncertain dynamic systems. By using KLSPI, near-optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants. In KLSPI, Mercer kernels are used in the policy evaluation of a policy iteration process, where a new kernel-based least squares temporal-difference algorithm called KLSTD-Q is proposed for efficient policy evaluation. To keep the sparsity and improve the generalization ability of KLSTD-Q solutions, a kernel sparsification procedure based on approximate linear dependency (ALD) is performed. Compared to the previous works on approximate RL methods, KLSPI makes two progresses to eliminate the main difficulties of existing results. One is the better convergence and (near) optimality guarantee by using the KLSTD-Q algorithm for policy evaluation with high precision. The other is the automatic feature selection using the ALD-based kernel sparsification. Therefore, the KLSPI algorithm provides a general RL method with generalization performance and convergence guarantee for large-scale Markov decision problems (MDPs). Experimental results on a typical RL task for a stochastic chain problem demonstrate that KLSPI can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration (LSPI) algorithm. Furthermore, the KLSPI method was also evaluated on two nonlinear feedback control problems, including a ship heading control problem and the swing up control of a double-link underactuated pendulum called acrobot. Simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems. It is also demonstrated that KLSPI can be applied to online learning control by incorporating an initial controller to ensure online performance.     

基于多项式函数和隐私保护策略的多关键字检索算法

针对一般检索算法在云计算环境下对大量加密数据不能有效进行多关键字检索的问题,提出了一种多项式函数结合隐私保护策略的检索算法,该算法可以进行有效的多关键字排列检索。首先,提出了一种基本策略,利用多项式函数隐藏加密的关键字;然后,寻找一种可以进行有效多关键字排列检索的模式。为了增强检索的保密性,提出了一种隐私保护策略,利用安全内积方法保护检索多关键字的私密性。最后,使用Enron电子邮件数据集作为实验数据库,分别在索引时间成本、暗门生成时间成本和查询时间方面评估了本文算法的检索和保密性能,实验结果表明,本文算法可在云计算中进行高效的加密多关键字排列检索。     

两方零和马尔科夫博弈下的策略梯度算法

在两方零和马尔科夫博弈中,由于玩家策略会受到另一个玩家策略的影响,传统的策略梯度定理只适用于交替训练两个玩家的策略.为了实现同时训练两个玩家的策略,文中给出两方零和马尔科夫博弈下的策略梯度定理.然后,基于该策略梯度定理,提出基于额外梯度的REINFORCE算法,可使玩家的联合策略收敛到近似纳什均衡.文中从多个维度分析算法的优越性.首先,在同时移动博弈游戏上的对比实验表明,文中算法的收敛性和收敛速度较优.其次,分析文中算法得到的联合策略的特点,并验证这些联合策略达到近似纳什均衡.最后,在不同难度等级的同时移动博弈游戏上的对比实验表明,文中算法在更大的难度等级下仍能保持不错的收敛速度.     

A phased reinforcement learning algorithm for complex control problems

In this article, a phased reinforcement learning algorithm for controlling complex systems is proposed. The key element of the proposed algorithm is a shaping function defined on a novel position–direction space. The shaping function is autonomously constructed once the goal is reached, and constrains the exploration area to optimize the policy. The efficiency of the proposed shaping function was demonstrated by using a complex control problem of positioning a 2-link planar underactuated manipulator.     

基于值函数估计的参数探索策略梯度算法

策略梯度估计方差大是策略梯度算法存在的普遍问题,基于参数探索的策略梯度算法（PGPE）通过使用确定性策略有效缓解了这一问题。然而,PGPE算法基于蒙特卡罗方法进行策略梯度的估计,需要大量学习样本才能保证梯度估计相对稳定,因此,梯度估计方差大阻碍了其在现实问题中的实际应用。为进一步减小PGPE算法策略梯度估计的方差,提出了基于值函数估计的参数探索策略梯度算法（PGPE-FA）,该算法在PGPE算法中引入Actor-Critic框架。具体地,提出的方法使用价值函数估计策略梯度,代替了PGPE方法使用轨迹样本估计策略梯度的方式,从而减小梯度估计方差。最后,通过实验验证了所提算法能够减小梯度估计的方差。     

Bayes Theorem Based Virtual Machine Scheduling for Optimal Energy Consumption

This paper proposes an algorithm for scheduling Virtual Machines (VM) with energy saving strategies in the physical servers of cloud data centers. Energy saving strategy along with a solution for productive resource utilization for VM deployment in cloud data centers is modeled by a combination of “Virtual Machine Scheduling using Bayes Theorem” algorithm (VMSBT) and Virtual Machine Migration (VMMIG) algorithm. It is shown that the overall data center’s consumption of energy is minimized with a combination of VMSBT algorithm and Virtual Machine Migration (VMMIG) algorithm. Virtual machine migration between the active physical servers in the data center is carried out at periodical intervals as and when a physical server is identified to be under-utilized. In VM scheduling, the optimal data centers are clustered using Bayes Theorem and VMs are scheduled to appropriate data center using the selection policy that identifies the cluster with lesser energy consumption. Clustering using Bayes rule minimizes the number of server choices for the selection policy. Application of Bayes theorem in clustering has enabled the proposed VMSBT algorithm to schedule the virtual machines on to the physical server with minimal execution time. The proposed algorithm is compared with other energy aware VM allocations algorithms viz. “Ant-Colony” optimization-based (ACO) allocation scheme and “min-min” scheduling algorithm. The experimental simulation results prove that the proposed combination of ‘VMSBT’ and ‘VMMIG’ algorithm outperforms other two strategies and is highly effective in scheduling VMs with reduced energy consumption by utilizing the existing resources productively and by minimizing the number of active servers at any given point of time.     

Policy Improvements for Probabilistic Pursuit-Evasion Game

This paper focuses on a pursuit-evasion game (PEG) which involves two teams: one side consists of pursuers trying to minimize the time required to capture evaders, and the other side consists of evaders trying to maximize the capture time by escaping the pursuers. In this paper, we propose a hybrid pursuit policy for a probabilistic PEG, which possesses the combined merits of local-max and global-max pursuit policies proposed in previous literature. A method to find optimal pursuit and evasion polices for two competitive parties of the pursuers and evaders is also proposed. For this, we employ an episodic parameter optimization (EPO) algorithm to learn good values for the weighting parameters of a hybrid pursuit policy and an intelligent evasion policy. The EPO algorithm is performed during the numerous repeated simulation runs of the PEG and the reward of each episode is updated using reinforcement learning, and the optimal weighting parameters are selected by using particle swarm optimization. We analyze the trend of the optimal parameter values with respect to the number of the pursuers and evaders. The proposed strategy is validated both in simulations and experiments with small ground robots.     

基于多动作并行异步深度确定性策略梯度的选矿运行指标决策方法

为了解决深度确定性策略梯度算法探索能力不足的问题,提出一种多动作并行异步深度确定性策略梯度(MPADDPG)算法,并用于选矿运行指标强化学习决策.该算法使用多个actor网络,进行不同的初始化和训练,不同程度地提升了探索能力,同时通过扩展具有确定性策略梯度结构的评论家体系,揭示了探索与利用之间的关系.该算法使用多个DDPG代替单一DDPG,可以减轻一个DDPG性能不佳的影响,提高学习稳定性;同时通过使用并行异步结构,提高数据利用效率,加快了网络收敛速度;最后, actor通过影响critic的更新而得到更好的策略梯度.通过选矿过程运行指标决策的实验结果验证了所提出算法的有效性.     

Extended distributed learning automata

In this paper, a new structure for cooperative learning automata called extended learning automata (eDLA) is introduced. Based on the new structure, an iterative randomized heuristic algorithm using sampling is proposed for finding an optimal subgraph in a stochastic edge-weighted graph. Stochastic graphs are graphs in which the weights of edges have an unknown probability distribution. The proposed algorithm uses an eDLA to find a policy that leads to a subgraph that satisfy some restrictions such as minimum or maximum weight (length). At each stage of the proposed algorithm, the eDLA determines which edges should be sampled. The proposed eDLA-based sampling method may reduce unnecessary samples and hence decrease the time required for finding an optimal subgraph. It is shown that the proposed method converges to an optimal solution, the probability of which can be made arbitrarily close to 1 by using a sufficiently small learning parameter. A new variance-aware threshold value is also proposed that can significantly improve the convergence rate of the proposed eDLA-based algorithm. It is further shown that our algorithm is competitive in terms of the quality of the solution.     

基于滚动时域强化学习的智能车辆侧向控制算法

针对智能车辆的高精度侧向控制问题, 提出一种基于滚动时域强化学习(Receding horizon reinforcement learning, RHRL)的侧向控制方法. 车辆的侧向控制量由前馈和反馈两部分构成, 前馈控制量由参考路径的曲率以及动力学模型直接计算得出; 而反馈控制量通过采用滚动时域强化学习算法求解最优跟踪控制问题得到. 提出的方法结合滚动时域优化机制, 将无限时域最优控制问题转化为若干有限时域控制问题进行求解. 与已有的有限时域执行器−评价器学习不同, 在每个预测时域采用时间独立型执行器−评价器网络结构学习最优值函数和控制策略. 与模型预测控制(Model predictive control, MPC)方法求解开环控制序列不同, RHRL控制器的输出是一个显式状态反馈控制律, 兼具直接离线部署和在线学习部署的能力. 此外, 从理论上证明了RHRL算法在每个预测时域的收敛性, 并分析了闭环系统的稳定性. 在仿真环境中完成了结构化道路下的车辆侧向控制测试. 仿真结果表明, 提出的RHRL方法在控制性能方面优于现有先进算法, 最后, 以红旗E-HS3电动汽车作为实车平台, 在封闭结构化城市测试道路和乡村起伏砂石道路下进行了侧向控制实验. 实验结果显示, RHRL在结构化城市道路中的侧向控制性能优于预瞄控制, 在乡村道路中具有较强的路面适应能力和较好的控制性能.     

基于灵敏度分析的含比例型手续费的投资组合优化

研究含比例型手续费的离散时间投资组合优化问题. 基于马尔可夫决策过程模型和性能灵敏度分析方法, 推导两个不同投资策略之间的资产长期平均增值率的差分公式, 利用差分公式的结构特点, 证明了最优性方程, 并设计出可在线应用的策略迭代算法. 仿真实例验证了所提出算法的有效性.

     

基于PSR模型的规划算法

提出一种基于PSR模型的规划算法.首先提出了状态经历的概念与发现方法,并进一步用此概念来描述系统的PSR状态.在此基础上,讨论了如何用判别分析方法,确定任意经历下的PSR状态以及如何在该过程中同时获取系统的PSR模型.从而可引入Q学习算法,用于决策当前的最优策略.算法被应用于一些标准的POMDP问题,实验结果验证了所提方法的有效性.     

基于遗传算法的信息流安全标记自动化挖掘

分布式信息流模型可以细粒度地控制信息的访问和传播,但其灵活性也增加了系统安全需求表达和策略管理实施的复杂性,严重制约了模型在云计算等分布式系统中的应用。对此,设计了一种基于安全断言的策略描述语言以表达安全需求;形式化定义了信息流安全标记挖掘问题,分析并证明了该问题是NP完全问题;提出了基于遗传算法的标记挖掘近似最优化算法。实验结果表明,算法可自动挖掘出策略配置最优解方案,有效提高了分布式信息流模型在实际系统中的可用性。     

基于遗传算法的顾客购买行为特征提取

提出一种基于遗传算法的顾客行为特征提取算法。首先,采用Tanimoto 相似度来度量顾客间购买行为,并设计遗传聚类算法对顾客群体进行划分,把具有相似购买行为顾客聚集为一类。然后,针对不同顾客群体的购买行为特征,设计一种基于遗传算法的多种群特征提取方法,从各个子群体中发现顾客的购买行为的知识。为了增强种群内部协同进化能力和规则质量,我们采用最近邻替代遗传策略和局部搜索策略。使用实际零售数据集对整个算法进行验证,并与经典的Apriori算法进行比较。实验结果表明该算法在不需要产生频繁项集的情况下,可较高效生成精简规则集,在规则形式方面也更加灵活。最后,对实验结果进行详细分析。     

基于目标的域随机化方法在机器人操作方面的研究

使用强化学习解决机器人操作问题有着诸多优势,然而传统的强化学习算法面临着奖励稀疏的困难,且得到的策略难以直接应用到现实环境中。为了提高策略从仿真到现实迁移的成功率,提出了基于目标的域随机化方法：使用基于目标的强化学习算法对模型进行训练,可以有效地应对机器人操作任务奖励稀疏的情况,得到的策略可以在仿真环境下良好运行,与此同时在算法中还使用了目标驱动的域随机化方法,在提高策略泛用性以及克服仿真和现实环境之间的差距上有着良好的效果,仿真环境下的策略容易迁移到现实环境中并成功执行。结果表明,使用了基于目标的域随机化方法的强化学习算法有助于提高策略从仿真到现实迁移的成功率。     

一种基于AS关系的BGP路由策略冲突检测算法

因特网边界网关协议BGP是基于策略的路由协议，它为路由选择提供了灵活性，然而，实践表面BGP路由会产生振荡，其原因是路由策略配置冲突，通过分析AS关系，提出一种基于AS关系的路由策略冲突检测算法，该算法根据AS关系下的路由策略配置原则检测路由策略配置错误。     

策略梯度的超启发算法求解带容量约束车辆路径问题

有容量车辆路径问题是组合优化问题中比较热门的问题, 它属于经典的NP-hard问题并且时间复杂度高.本文提出了一种基于策略梯度的超启发算法, 将强化学习中的确定性策略梯度算法引入到超启发算法的高层策略中的底层算法选择策略, 确定性策略梯度算法采用Actor-Critic框架, 另外为了能够在后续计算和神经网络参数更新中引用历史经验数据, 在确定性策略梯度算法中设计了经验池用于存储状态转移数据. 在超启发算法解的接受准则方面, 文中通过实验对比了3种接受准则的效果, 最终选择了自适应接受准则作为高层策略中解的接受准则. 通过对有容量车辆路径问题标准算例的计算, 并将求解结果与其他算法对比, 验证了所提算法在该问题求解上的有效性和稳定性.     

基于策略迭代算法的连续时间线性Markov跳变系统非零和微分反馈Nash控制

针对一类连续时间线性Markov跳变系统,本文提出了一种新的策略迭代算法用于求解系统的非零和微分反馈Nash控制问题.通过求解耦合的数值迭代解,以获得具有线性动力学特性和无限时域二次成本的双层非零和微分策略的Nash均衡解.在每一个策略层,采用策略迭代算法来计算与每一组给定的反馈控制策略相关联的最小无限时域值函数.然后,通过子系统分解将Markov跳变系统分解为N个并行的子系统,并将该算法应用于跳变系统.本文提出的策略迭代算法可以很容易求解非零和微分策略所对应的耦合代数Riccati方程,且对高维系统有效.最后通过仿真示例证明了本文设计方法的有效性和可行性.     

一类非线性动态系统基于强化学习的最优控制制

提出一类非线性不确定动态系统基于强化学习的最优控制方法. 该方法利用欧拉强化学习算法估计对象的未知非线性函数, 给出了强化学习中回报函数和策略函数迭代的在线学习规则. 通过采用向前欧拉差分迭代公式对学习过程中的时序误差进行离散化, 实现了对值函数的估计和控制策略的改进. 基于值函数的梯度值和时序误差指标值, 给出了该算法的步骤和误差估计定理. 小车爬山问题的仿真结果表明了所提出方法的有效性.