Backward Q-learning: The combination of Sarsa algorithm and Q-learning

Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning and the Sarsa algorithms, but they possess different characteristics. Generally speaking, the Sarsa algorithm has faster convergence characteristics, while the Q-learning algorithm has a better final performance. However, Sarsa algorithm is easily stuck in the local minimum and Q-learning needs longer time to learn. Most literatures investigated the action selection policy. Instead of studying an action selection strategy, this paper focuses on how to combine Q-learning with the Sarsa algorithm, and presents a new method, called backward Q-learning, which can be implemented in the Sarsa algorithm and Q-learning. The backward Q-learning algorithm directly tunes the Q-values, and then the Q-values will indirectly affect the action selection policy. Therefore, the proposed RL algorithms can enhance learning speed and improve final performance. Finally, three experimental results including cliff walk, mountain car, and cart–pole balancing control system are utilized to verify the feasibility and effectiveness of the proposed scheme. All the simulations illustrate that the backward Q-learning based RL algorithm outperforms the well-known Q-learning and the Sarsa algorithm.     

基于秩的Q-路由选择算法

如何在动态变化的复杂网络中实现高效的路由选择是当前的研究热点之一。Q-学习是一种常用的强化学习算法,通过与环境的不断交互来解决未知环境中最优控制问题,能有效地完成在线式学习任务。本文提出一种基于秩的Q-路由选择(Rank-based Q-routing, RQ routing)算法。RQ routing算法在Q-学习的框架下,保留了Q-路由选择(Q-routing)算法的高效性,引入能动态计算的秩函数,用于表示当前状态在场景中的优先级,用以求解路由选择的最优解,避免等待队列过长,减少网络拥堵,提高传输速度。RQ routing算法中的秩函数具有灵活性,使用不同的秩函数即可满足各种场景的需求,保证了算法具有更好的泛化能力,克服了传统Q-routing应用场景单一的不足。实验验证了本文算法的有效性。     

A survey of routing algorithm for mesh Network-on-Chip

With the rapid development of semiconductor industry, the number of cores integrated on chip increases quickly, which brings tough challenges such as bandwidth, scalability and power into on-chip interconnection. Under such background, Network-on-Chip (NoC) is proposed and gradually replacing the traditional on-chip interconnections such as sharing bus and crossbar. For the convenience of physical layout, mesh is the most used topology in NoC design. Routing algorithm, which decides the paths of packets, has significant impact on the latency and throughput of network. Thus routing algorithm plays a vital role in a wellperformed network. This study mainly focuses on the routing algorithms of mesh NoC. By whether taking network information into consideration in routing decision, routing algorithms of NoC can be roughly classified into oblivious routing and adaptive routing. Oblivious routing costs less without adaptiveness while adaptive routing is on the contrary. To combine the advantages of oblivious and adaptive routing algorithm, half-adaptive algorithms were proposed. In this paper, the concepts, taxonomy and features of routing algorithms of NoC are introduced. Then the importance of routing algorithms in mesh NoC is highlighted, and representative routing algorithms with respective features are reviewed and summarized. Finally, we try to shed light upon the future work of NoC routing algorithms.     

基于Q学习的蚁群优化水声网络协议

针对水声通信中数据传输延时高且动态适应性弱的问题, 提出了一种基于Q学习优化的蚁群智能水声网络路由协议(Q-learning ant colony optimization, QACO). 协议包括路由行为和智能决策部分, 在路发现和维护阶段, 依靠网络智能蚂蚁进行网络拓扑环境的构建和节点之间的信息交换以及网络的维护. 在Q学习阶段, 通过定量化节点能量和深度以及网络传输延时学习特征作为折扣因子和学习率, 以延长网络的生命周期, 降低系统能耗和延时. 最后通过水声网络环境进行仿真, 实验结果表明QACO在能耗、延迟和网络生命周期方面都优于基于Q学习辅助的蚁群算法(Q-learning aided ant colony routing protocol, QLACO)和基于 Q-learning 的节能和生命周期感知路由协议(Q-learning-based energy-efficient and lifetime-aware routing protocol, QELAR)和基于深度路由协议 (depth-based routing, DBR)算法.     

Data-driven adaptive dynamic programming for partially observable nonzero-sum games via Q-learning method

This paper concerns with a class of discrete-time linear nonzero-sum games with the partially observable system state. As is known, the optimal control policy for the nonzero-sum games relies on the full state measurement which is hard to fulfil in partially observable environment. Moreover, to achieve the optimal control, one needs to know the accurate system model. To overcome these deficiencies, this paper develops a data-driven adaptive dynamic programming method via Q-learning method using measurable input/output data without any system knowledge. First, the representation of the unmeasurable inner system state is built using historical input/output data. Then, based on the representation state, a Q-function-based policy iteration approach with convergence analysis is introduced to approximate the optimal control policy iteratively. A neural network (NN)-based actor-critic framework is applied to implement the developed data-driven approach. Finally, two simulation examples are provided to demonstrate the effectiveness of the developed approach.     

微网控制系统中多AgentQ学习算法的研究

本文首先介绍了微网控制系统的多Agent结构以及各Agent的工作流程,然后提出了应用于微网控制系统的多Agent结构的协作学习算法,该算法在Q学习算法的基础上进行了改进,使之适用于混合环境中。最后将IEEE9节点系统作为微网模拟系统并在其中进行了仿真,结果显示该算法可以在微网功率发生波动时快速地使功率恢复到稳定状态。     

基于生成模型的Q-learning二分类算法

对于二分类问题,基于判别模型的分类器一般都是寻找一条最优判决边界,容易受到数据波动的影响。针对该问题提出一种基于生成模型的Q-learning二分类算法（BGQ-learning）,将状态和动作分开编码,得到对应各类的判决函数,增加了决策空间的灵活性,同时在求解参数时,采用最小二乘时序差分（TD）算法和半梯度下降法的组合优化方法,加速了参数的收敛速度。设计实验对比了BGQ-learning算法与三种经典分类器以及一种新颖的分类器的分类性能,在UCI数据库七个数据集上的测试结果表明,该算法有着优良的稳定性以及良好的分类精确度。     

基于Q学习的供应链分销系统最优订货策略研究

研究由一个制造商和多个分销商组成的分销系统的最优订货策略问题.在外部顾客需求不断变化的情况下,以不断提高分销系统双方合作绩效为目标,基于Q学习算法来确定每个分销商的最优订货批量.实例结果表明,在外部需求不断变化的条件下,该算法能简便地解决供应链企业分销系统合作中的最优订货批量问题.     

PRDT（2，1）架构的NoC系统容错路由算法

链路和节点的故障会导致网络中许多节点无法相互通讯,因此容错性是NoC系统设计中的一个重要问题。基于一种新的NoC网络拓扑结构PRDT（2,1）,提出一种PRDT（2,1）容错路由算法以及相应的节点失效算法。节点失效算法通过使较少数量的无故障节点失效来构造矩形故障区域,PRDT（2,1）容错路由算法仅使用了最小数量的虚拟通道并提供足够的自适应性以实现无死锁容错路由。只要故障区域没有断开网络,这一算法能够保证路由的连通性。算法在不同故障率的PRDT（2,1）网络中仿真,结果显示这一算法具有良好的平滑降级使用特性。     

MMNNN: A tree-based Multicast Mechanism for NoC-based deep Neural Network accelerators

Network-on-Chip (NoC) devices have been widely used in multiprocessor systems. In recent years, NoC-based Deep Neural Network (DNN) accelerators have been proposed to connect neural computing devices using NoCs. Such designs dramatically reduce off-chip memory accesses of these platforms. However, the large number of one-to-many packet transfers significantly degrade performance with traditional unicast channels. We propose a multicast mechanism for a NoC-based DNN accelerator called Multicast Mechanism for NoC-based Neural Network accelerator (MMNNN). To do so, we propose a tree-based multicast routing algorithm with excellent scalability and the ability to minimize the number of packets in the network. We also propose a router architecture for single-flit packets. Our proposed router transfers flits to multiple destinations in a single process and has no head-of-line blocking issue, offering higher throughput and lower latency than traditional wormhole router architectures. Simulation results show that our proposed multicast mechanism offers excellent performance in classification latency, average packet latency, and energy consumption.     

模糊Q学习的足球机器人双层协作模型

针对传统的足球机器人3层决策模型存在决策不连贯的问题和缺乏适应性与学习能力的缺点,提出了一种基于模糊Q学习的足球机器人双层协作模型.该模型使协调决策和机器人运动成为2个功能独立的层次,使群体意图到个体行为的过度变为一个直接的过程,并在协调层通过采用Q学习算法在线学习不同状态下的最优策略,增强了决策系统的适应性和学习能力.在Q学习中通过把状态繁多的系统状态映射为为数不多的模糊状态,大大减少了状态空间的大小,避免了传统Q学习在状态空间和动作空间较大的情况下收敛速度慢,甚至不能收敛的缺点,提高了Q学习算法的收敛速度.最后,通过在足球机器人SimuroSot仿真比赛平台上进行实验,验证了双层协作模型的有效性.     

状态翻转控制下布尔控制网络的可镇定性和Q学习算法

在给定一个子集的条件下, 本文研究了在状态翻转控制下布尔控制网络的全局镇定问题. 对于节点集的给定子 集, 状态翻转控制可以将某些节点的值从1 (或0)变成0 (或1). 将翻转控制作为控制之一, 本文研究了状态翻转控制下的 布尔控制网络. 将控制输入和状态翻转控制结合, 提出了联合控制对和状态翻转转移矩阵的概念. 接着给出了状态翻转 控制下布尔控制网络全局稳定的充要条件. 镇定核是最小基数的翻转集合, 本文提出了一种寻找镇定核的算法. 利用可 达集的概念, 给出了一种判断全局镇定和寻找联合控制对序列的方法. 此外, 如果系统是一个大型网络, 则可以利用一 种名为Q学习算法的无模型强化学习方法寻找联合控制对序列. 最后给出了一个数值例子来说明本文的理论结果.     

基于多智能体Q学习的异构车载网络选择方法

异构车载网络环境下如何选择接入网络对于车载终端用户的服务体验而言至关重要,目前基于Q学习的网络选择方法利用智能体与环境的交互来迭代学习网络选择策略,从而实现较优的网络资源分配.然而该类方法通常存在状态空间过大引起迭代效率低下和收敛速度较慢的问题,同时由于Q值表更新产生的过高估计现象容易导致网络资源利用不均衡.针对上述问...     

一种基于分簇的车联网对向协助下载方法

针对目前车联网中提出的协助下载方法的吞吐量低和延迟较长的问题,提出了一种基于动态相邻间距分簇算法的对向协助下载方法.根据道路上车辆的密度不同,动态改变车辆相邻间距使簇保持一个合适的大小;利用分簇算法为车辆分簇;使用分簇的车辆为用户提供协助下载服务.该方法充分利用了道路上行驶的车辆,使更多的车辆加入到协助下载中,提高了用户获取数据吞吐量的同时降低了用户下载数据的延迟.仿真结果表明:使用基于分簇的对向协助下载方法比使用单个车辆对向协助下载时,在盲区的吞吐量提高50％.     

基于自组织模糊RBF网络的连续空间Q学习

针对连续空间下的强化学习控制问题,提出了一种基于自组织模糊RBF网络的Q学习方法．网络的输入为状态,输出为连续动作及其Q值,从而实现了“连续状态—连续动作”的映射关系．首先将连续动作空间离散化为确定数目的离散动作,采用完全贪婪策略选取具有最大Q值的离散动作作为每条模糊规则的局部获胜动作．然后采用命令融合机制对获胜的离散动作按其效用值进行加权,得到实际作用于系统的连续动作．另外,为简化网络结构和提高学习速度,采用改进的RAN算法和梯度下降法分别对网络的结构和参数进行在线自适应调整．倒立摆平衡控制的仿真结果验证了所提Q学习方法的有效性．     

Lightweight Transactional Memory systems for NoCs based architectures: Design,implementation and comparison of two policies

Hardware Transactional Memory (HTM) is an attractive design concept which simplifies parallel programming by shifting the problem of correct synchronization between threads to the underlying hardware memory system.     

一种适用于2D Mesh片上网络的可重构容错路由算法

适用于2D Mesh片上网络的可重构容错路由算法,在芯片某些区域由于制造缺陷、使用老化等原因拓扑结构变得不再规整的时候,可以对网络节点重新进行配置,从而保证健康节点间的正常通信.基于SystemC的平台仿真表明该算法相对于传统算法可以获得更佳的网络性能.该算法是免于死锁的,同时对其可重构机制也给出了详细的论述.它还具有良好的扩展性,当系统规模增大的时候每个路由器的硬件开销保持恒定,而其容错能力也得到了增强.     

离散多智能体有限时间Q学习协同输出调节

针对离散多智能体系统输出调节,提出了一种基于Q学习的最优控制策略。对于传统多智能体系统的输出调节,获取系统的精确动力学模型并依此求得其HJB方程的解为主要障碍。该策略通过智能体之间的局部通信,在不依赖系统动态模型的前提下实现了对每个智能体输出的全局最优控制。为实现对系统响应速率的优化,提出了一种新的有限时间局部误差公式,不仅保证了算法原有的全局最优性能,而且将输出同步时间缩短了近50%,并对所提算法的稳定性进行了分析。仿真结果表明,该策略在避免建立复杂系统模型和求解离散HJB方程的前提下实现了对系统的最优控制,采用更新后的有限时间局部误差公式有效缩短了收敛时间。     

基于最优初始值Q学习的电力信息网络防御策略学习算法

电力信息网络的安全与稳定是当今社会发展的重要保障,随着电力信息网络越来越庞大和复杂,如何高效合理地建立电力信息防护网络成为研究人员关注的重点之一。在自动化电力信息网络中,其防御策略通常缺乏统筹管理,只能针对少数设备进行防护,存在着更新速度慢、更新周期长、无法自动更新和资源分配不均等问题。本文提出一种基于最优初始值Q学习的电力信息网络防御策略学习算法,该算法以强化学习中的Q学习算法为框架,利用生成对抗网络思想,通过攻击智能体和防御智能体的模拟对抗学习安全策略。算法中的防御智能体使用Q学习方法更新其防御策略,利用历史防御经验在线改进防御策略,避免了人为手动操作。在训练中引入最优初始值极大加快了系统防御性能的训练速度。实验结果验证了算法的有效性。