基于Q学习和规划的传感器节点任务调度算法*

为了改善节点的学习策略,提高节点的应用性能,以数据收集为应用建立任务模型,提出基于Q学习和规划的传感器节点任务调度算法,包括定义状态空间、延迟回报、探索和利用策略等基本元素.根据无线传感器网络(WSN)特性,建立基于优先级机制和过期机制的规划过程,使节点可以有效利用经验知识,改善学习策略.实验表明,文中算法具备根据当前WSN环境进行动态任务调度的能力.相比其它任务调度算法,文中算法能量消耗合理且获得较好的应用性能.     

基于Q-learning算法的煤矿井下移动机器人路径规划

如何针对煤矿井下环境的不确定性规划机器人的路径是其中的一个难点。文章提出了一种基于Q-learning算法的移动机器人路径规划,希望对提高机器人救援的避障能力的提升,起到一定的促进作用。     

基于反馈机制的自愈算法

提出了一种网络自愈算法,当网络中的节点发生故障或链路出现拥塞时,该算法利用Q学习的反馈机制、多QoS约束的评价函数和基于Boltzmann-Gibbs分布的路径选择策略,自适应地选择恢复路径,降低了选择发生故障和拥塞路径的概率,从而实现了自愈。仿真结果表明,该算法在恢复率、区分业务能力和网络资源优化等方面,表现出了良好的性能。     

Path Selection in Disaster Response Management Based on Q-learning

Suitable rescue path selection is very important to rescue lives and reduce the loss of disasters,and has been a key issue in the field of disaster response management.In this paper,we present a path selection algorithm based on Q-learning for disaster response applications.We assume that a rescue team is an agent,which is operating in a dynamic and dangerous environment and needs to find a safe and short path in the least time.We first propose a path selection model for disaster response management,and deduce that path selection based on our model is a Markov decision process.Then,we introduce Q-learning and design strategies for action selection and to avoid cyclic path.Finally,experimental results show that our algorithm can find a safe and short path in the dynamic and dangerous environment,which can provide a specific and significant reference for practical management in disaster response applications.     

Reactive fuzzy controller design by Q-learning for mobile robot navigation

In this paper a learning mechanism for reactive fuzzy controller design of a mobile robot navigating in unknown environments is proposed. The fuzzy logical controller is constructed based on the kinematics model of a real robot. The approach to learning the fuzzy rule base by relatively simple and less computational Q-learning is described in detail. After analyzing the credit assignment problem caused by the rules collision, a remedy is presented. Furthermore, time-varying parameters are used to increase the learning speed. Simulation results prove the mechanism can learn fuzzy navigation rules successfully only using scalar reinforcement signal and the rule base learned is proved to be correct and feasible on real robot platforms.     

Creating Advice-Taking Reinforcement Learners

Learning from reinforcements is a promising approach for creating intelligent agents. However, reinforcement learning usually requires a large number of training episodes. We present and evaluate a design that addresses this shortcoming by allowing a connectionist Q-learner to accept advice given, at any time and in a natural manner, by an external observer. In our approach, the advice-giver watches the learner and occasionally makes suggestions, expressed as instructions in a simple imperative programming language. Based on techniques from knowledge-based neural networks, we insert these programs directly into the agent's utility function. Subsequent reinforcement learning further integrates and refines the advice. We present empirical evidence that investigates several aspects of our approach and shows that, given good advice, a learner can achieve statistically significant gains in expected reward. A second experiment shows that advice improves the expected reward regardless of the stage of training at which it is given, while another study demonstrates that subsequent advice can result in further gains in reward. Finally, we present experimental results that indicate our method is more powerful than a naive technique for making use of advice.     

Learning Intelligent Behavior in a Non-stationary and Partially Observable Environment

Individual learning in an environment where more than one agent exist is a chal-lengingtask. In this paper, a single learning agent situated in an environment where multipleagents exist is modeled based on reinforcement learning. The environment is non-stationaryand partially accessible from an agents' point of view. Therefore, learning activities of anagent is influenced by actions of other cooperative or competitive agents in the environment.A prey-hunter capture game that has the above characteristics is defined and experimentedto simulate the learning process of individual agents. Experimental results show that thereare no strict rules for reinforcement learning. We suggest two new methods to improve theperformance of agents. These methods decrease the number of states while keeping as muchstate as necessary.     

一种空间仿生柔性机器人设计与智能规划仿真方法

针对传统空间刚体机器人存在的自由度有限和环境适应性差等缺陷,基于生物体结构提出了一种受“尺蠖”与“蛇”启发的适用于空间在轨服务的柔性机器人。首先,搭建了柔性机器人原型样机,研究了镍钛形状记忆合金（SMA）驱动器的驱动特性,设计了可视化控制界面并通过实物实验验证了机器人原型样机的可操控性。然后,设计了一种基于所提柔性机器人结构的Q学习算法和相应的奖励函数,搭建了柔性机器人仿真模型并在仿真环境中完成了基于Q学习的机器臂自主学习规划仿真实验。实验结果显示机器臂能够在较短时间内收敛到稳定状态并自主完成规划任务,表明所提出算法具有有效性和可行性,强化学习方法在柔性机器人的智能规划与控制中具有良好的应用前景。     

无人机自组网中基于Q-learning算法的及时稳定路由策略

无人机自组网凭借其抗干扰能力强、适用于复杂地形、智能化程度高和成本较低的优点,近年来受到广泛关注,该网络中路由协议的设计与优化一直是核心研究问题。针对无人机自组网中因节点快速移动造成节点本地存储的路由未及时更新而失效的问题,提出一种基于Q-learning算法的动态感知优化链路状态路由协议(DSQ-OLSR)。该协议首先充分考虑了无人机自组网节点高速移动的特点,在选取多点中继(MPR)节点时添加了链路稳定性和链路存在时间这两个指标,使得选出的MPR节点集更稳定、合理;其次,结合Q-learning算法对TC消息的发送间隔进行自适应调整,使得在网络拓扑变动较小时增大TC发送间隔以减小控制开销,而在拓扑变动较大时减小TC发送间隔用于达到快速感知并构建网络拓扑的要求,进而实现数据的及时路由。仿真结果表明,与DT-OLSR协议相比,该协议在端到端时延、吞吐量、成功率和网络生存时间性能上分别提高了12.61%、9.28%、7.69%和5.86%,由此验证了其有效性。     

基于因子分解机用于安全探索的Q表初始化方法

针对强化学习的大多数探索/利用策略在探索过程中忽略智能体随机选择动作带来的风险的问题,提出一种基于因子分解机（FM）用于安全探索的Q表初始化方法。首先,引入Q表中已探索的Q值作为先验知识;然后,利用FM建立先验知识中状态和行动间潜在的交互作用的模型;最后,基于该模型预测Q表中的未知Q值,从而进一步引导智能体探索。在OpenAI Gym的网格强化学习环境Cliffwalk中进行的A/B测试里,基于所提方法的Boltzmann和置信区间上界（UCB）探索/利用策略的不良探索幕数分别下降了68.12%和89.98%。实验结果表明,所提方法提高了传统策略的探索安全性,同时加快了收敛。