事件驱动的强化学习多智能体编队控制

针对经典强化学习的多智能体编队存在通信和计算资源消耗大的问题,本文引入事件驱动控制机制,智能体的动作决策无须按固定周期进行,而依赖于事件驱动条件更新智能体动作。在设计事件驱动条件时,不仅考虑智能体的累积奖赏值,还引入智能体与邻居奖赏值的偏差,智能体间通过交互来寻求最优联合策略实现编队。数值仿真结果表明,基于事件驱动的强化学习多智能体编队控制算法,在保证系统性能的情况下,能有效降低多智能体的动作决策频率和资源消耗。     

数据丢包下事件驱动的非线性多智能体迭代学习控制

针对具有随机链路丢包、通信带宽受限以及模型未知的非线性多智能体一致性问题, 提出一种事件驱动的分布式无模型迭代学习控制策略. 首先建立系统的事件驱动决策机制, 给出基于输出信息的通信触发条件, 当该条件满足时触发事件, 各智能体间进行通信, 不满足条件时则不通信, 从而能够有效减少智能体间的大量通信和能量耗散. 其次, 使用伪偏导数将非线性系统沿迭代轴动态线性化, 借助邻居在前一步事件触发时的输出信息设计随机链路丢包补偿机制, 再结合事件驱动通信机制设计分布式控制协议. 在此基础上, 使用压缩映射原理分析算法收敛性能, 仿真结果表明随着迭代次数的增加, 事件触发间隔变大, 所有的智能体将完成对期望轨迹的跟踪.     

通信时滞下事件驱动多智能体系统环形编队控制

针对一阶具有通信时滞的多智能体系统环形编队存在通信和计算资源消耗大的问题,引入事件驱动控制机制,综合设计适用于任意环形编队的控制律分别耦合状态相关和状态无关两类事件驱动条件。事件触发函数基于状态误差建立,使得个体之间的信息通信和控制信号更新仅在事件触发时刻进行。从理论上严格证明了系统在控制律作用下的收敛性,并通过数值仿真验证了控制算法的有效性。仿真结果也表明,在获得系统期望性能的前提下,降低了控制器输入的更新频率和减少了智能体的资源消耗。     

部分可观测下基于RGMAAC算法的多智能体协同

多智能体深度强化学习(MADRL)将深度强化学习的思想和算法应用到多智能体系统的学习和控制中,是开发具有群智能体的多智能体系统的重要方法.现有的MADRL研究主要基于环境完全可观测或通信资源不受限的假设展开算法设计,然而部分可观测性是多智能体系统实际应用中客观存在的问题,例如智能体的观测范围通常是有限的,可观测的范围外不包括完整的环境信息,从而对多智能体间协同造成困难.鉴于此,针对实际场景中的部分可观测问题,基于集中式训练分布式执行的范式,将深度强化学习算法Actor-Critic扩展到多智能体系统,并增加智能体间的通信信道和门控机制,提出recurrent gated multi-agent Actor-Critic算法(RGMAAC).智能体可以基于历史动作观测记忆序列进行高效的通信交流,最终利用局部观测、历史观测记忆序列以及通过通信信道显式地由其他智能体共享的观察进行行为决策;同时,基于多智能体粒子环境设计多智能体同步且快速到达目标点任务,并分别设计2种奖励值函数和任务场景.实验结果表明,当任务场景中明确出现部分可观测问题时,RGMAAC算法训练后的智能体具有很好的表现,在稳定性...     

基于意图的多智能体深度强化学习运动规划方法

现有的多智能体运动规划任务存在缺乏有效合作方法、通信依赖要求高以及缺乏信息筛选机制等问题。针对这些问题，提出了一种基于意图的多智能体深度强化学习运动规划方法，该方法可以帮助智能体在无需显式通信的条件下无碰撞地到达目标点。首先，将意图概念引入多智能体运动规划问题，将智能体的视觉图像和历史地图相结合以预测智能体的意图，使智能体可以对其他智能体的动作做预判，从而有效协作；其次，设计了一个基于注意力机制的卷积神经网络架构，并利用该网络预测智能体的意图、选择智能体的动作，在筛选出有用的视觉输入信息的同时，减少了多智能体合作对通信的依赖；最后提出了一种基于价值的深度强化学习算法来学习运动规划策略，通过改进目标函数和Q值计算方式使策略更加稳定。在PyBullet仿真平台的6种不同的仿真场景中进行了测试，实验结果表明，相较于其他先进的多智能体运动规划方法，所提方法使多智能体团队的合作效率平均提高了10.74%,具有显著的性能优势。     

车联网中基于联邦深度强化学习的任务卸载算法

随着车联网应用服务体系日益丰富，计算资源有限的车辆难以处理这些计算密集和时延敏感的车联网应用。计算卸载作为移动边缘计算中的一种关键技术可以解决这一难题。对于车联网中动态的多车辆多路侧单元的任务卸载环境，提出了一种基于联邦深度强化学习的任务卸载算法。该算法将每辆车都看作是智能体，采用联邦学习的框架训练各智能体，各智能体分布式决策卸载方案，以最小化系统的平均响应时间。设置评估实验，在多种动态变化的场景下对提出的算法的性能进行对比分析。实验结果显示，提出的算法求解出的系统平均响应时间短于基于规则的算法和多智能体深度强化学习算法，接近于理想方案，且求解时间远短于理想方案。实验结果表明，所提算法能够在可接受的算法执行时间内求解出接近于理想方案的系统平均响应时间。     

基于加权值函数分解的多智能体分层强化学习技能发现方法

针对目前大多数多智能体强化学习算法在智能体数量增多以及环境动态不稳定的情况下导致的维度爆炸和奖励稀疏的问题，提出了一种基于加权值函数分解的多智能体分层强化学习技能发现算法。首先，该算法将集中训练分散执行的架构与分层强化学习相结合，在上层采用加权值函数分解的方法解决智能体在训练过程中容易忽略最优策略而选择次优策略的问题；其次，在下层采用独立Q学习算法使其能够在多智能体环境中分散式地处理高维复杂的任务；最后，在底层独立Q学习的基础上引入技能发现策略，使智能体之间相互学习互补的技能。分别在简易团队运动和星际争霸Ⅱ两个仿真实验平台上对该算法与多智能体强化学习算法和分层强化学习算法进行对比，实验表明，该算法在奖励回报以及双方对抗胜率等性能指标上都有所提高，提升了整个多智能体系统的决策能力和收敛速度，验证了算法的可行性。     

多智能体分层强化学习综述

作为机器学习和人工智能领域的一个重要分支,多智能体分层强化学习以一种通用的形式将多智能体的协作能力与强化学习的决策能力相结合,并通过将复杂的强化学习问题分解成若干个子问题并分别解决,可以有效解决空间维数灾难问题。这也使得多智能体分层强化学习成为解决大规模复杂背景下智能决策问题的一种潜在途径。首先对多智能体分层强化学习中涉及的主要技术进行阐述,包括强化学习、半马尔可夫决策过程和多智能体强化学习;然后基于分层的角度,对基于选项、基于分层抽象机、基于值函数分解和基于端到端等4种多智能体分层强化学习方法的算法原理和研究现状进行了综述;最后介绍了多智能体分层强化学习在机器人控制、博弈决策以及任务规划等领域的应用现状。     

基于事件驱动的多智能体有限时间分群一致控制

针对二阶线性多智能体系统的分群一致控制问题,考虑智能体通信拓扑同时包含协作和对抗关系,提出一种基于事件驱动控制的有限时间分布式领航跟随分群一致性算法,该算法可使多智能体系统在有限时间内实现分群一致,即各子组内的智能体实现状态一致,不同子组收敛至不同一致状态.采用事件驱动控制机制,设计事件驱动函数及事件触发条件,降低智能体控制器更新频率,减少系统能耗.基于代数图论和李雅普诺夫稳定性理论推导出系统的有限时间稳定性条件,通过巧妙构造Lyapunov函数,给出系统有限收敛时间的显式估计,同时证明在所提出的事件驱动机制下,每个智能体相邻触发时间间隔有严格的正下界,即避免了芝诺行为.仿真实验验证了所提出的有限时间事件驱动分群一致控制算法的有效性.     

基于事件驱动的二次凸优化问题分布式优化算法

针对多智能体系统中等式约束下的二次凸优化问题,给出一种事件驱动机制下的分布式优化算法.该算法可以降低每个智能体控制协议的更新频率以及智能体之间的通信负担.基于图论和李雅普诺夫函数方法给出两种不同的事件触发条件,其中第2种事件触发条件不需要拉普拉斯矩阵的最大特征根的信息,可实现算法全分布式实施.两种事件触发条件均可实现算法渐近收敛到优化值,避免智能体控制协议的连续更新以及智能体之间的连续通信,同时保证每个智能体相邻事件触发时刻的时间间隔大于0,避免持续事件触发.将所提出的算法应用于Matlab仿真环境中进行仿真验证,仿真结果验证了所提出算法的有效性.     

Distributed event-triggered control of multi-agent systems with combinational measurements

This paper studies the distributed rendezvous problem of multi-agent systems with novel event-triggered controllers. We have proposed a combinational measurement approach to event design and developed the basic event-triggered control algorithm. As a result, control of agents is only triggered at their own event time, which reduces the amount of communication and lowers the frequency of controller updates in practice. Furthermore, based on the convergence analysis of the basic algorithm, we have proposed a new iterative event-triggered algorithm where continuous measurement of the neighbor states is avoided. It is noted that the amount of communication among agents has been significantly reduced without obvious negative effects on the control performances. The effectiveness of the proposed strategies is illustrated by numerical examples in 3D spaces.     

组合连通拓扑下基于事件触发的多智能体快速一致性算法

针对组合连通拓扑下多智能体系统控制过程中存在通信和计算资源损耗大以及系统收敛速度慢等问题,提出一种新的具有状态预测器的事件触发一致性控制协议,通过设计状态预测器使每个智能体都能对其邻居智能体的未来状态作出预测;同时,对于智能个体给出了基于状态信息的事件触发条件,当状态误差满足该条件才触发事件.在该控制策略下多智能体系统可在节约通信和计算资源的同时具有更快的收敛速度.利用Lyapunov稳定性理论和代数图论,证明了所提事件触发控制策略能够有效实现组合连通拓扑结构下的平均一致性,且不存在Zeno行为.仿真实例进一步验证了理论结果的有效性.     

Agent-based time delay margin in consensus of multi-agent systems by an event-triggered control method: Concept and computation

This paper deals with defining the concept of agent-based time delay margin and computing its value in multi-agent systems controlled by event-triggered based controllers. The agent-based time delay margin specifying the time delay tolerance of each agent for ensuring consensus in event-triggered controlled multi-agent systems can be considered as complementary for the concept of (network) time delay margin, which has been previously introduced in some literature. In this paper, an event-triggered control method for achieving consensus in multi-agent systems with time delay is considered. It is shown that the Zeno behavior is excluded by applying this method. Then, in a multi-agent system controlled by the considered event-triggered method, the concept of agent-based time delay margin in the presence of a fixed network delay is defined. Moreover, an algorithm for computing the value of the time delay margin for each agent is proposed. Numerical simulation results are also provided to verify the obtained theoretical results.     

Sampled-data based distributed convex optimization with event-triggered communication

This paper studies the distributed convex optimization problem for multi-agent systems over undirected and connected networks. Motivated by practical considerations, we propose a new distributed optimization algorithm with event-triggered communication. The proposed event detection is decentralized, sampled-data and not requires periodic communications among agents to calculate the threshold. Based on Lyapunov approaches, we show that the proposed algorithm is asymptotically converge to the unknown optimizer if the design parameters are chosen properly. We also give an upper bound on the convergence rate. Finally, we illustrate the effectiveness of the proposed algorithm by a numerical simulation.     

Decentralized Event-Triggered Average Consensus for Multi-Agent Systems in CPSs with Communication Constraints

The paper investigates decentralized event-triggered average consensus problem for multi-agent systems in cyberphysical systems (CPSs) with communication constraints. To reduce communication burden and improve the communication efficiency of multi-agent systems in CPSs, event-trigger is distributed at subsystem/agent level. A multi-agent system is then modeled as a reduced dimension hybrid system by taking into account decentralized event-triggered mechanism, communication delays and data dropouts within one framework. Some sufficient conditions for average consensus of each agent and an upper bound of communication delay and maximal allowable number of successive data dropouts (MANSD) are obtained, which can conveniently provide the relationship between the triggering parameters, communication constraints and the system stability. Specially, the quantitative relationship between the triggering parameters, MANSD and the system stability is derived. Finally, simulation results are given to illustrate the effectiveness of the proposed method.      

Bipartite Byzantine-resilient event-triggered consensus control of heterogeneous multi-agent systems

This article studies the bipartite resilient event-triggered consensus control for a class of the heterogeneous multi-agent systems. Due to the external cyberattacks, some agents may become the Byzantine agents and will affect the behavior of the other agents. To improve the security of the multi-agent systems against the Byzantine agents, a novel bipartite event-triggered heterogeneous mean-subsequence-reduced algorithm is designed. First, to handle the heterogeneous multi-agent systems, a state transformation is carefully designed, to facilitate the design and analysis of the bipartite resilient consensus algorithm. Based upon the designed state transformation, the bipartite resilient control inputs are constructed, where the structural balance analysis shows that the resulting effective signed graph and the equivalent signed graph are both structurally balanced, if the signed graph of the multi-agent systems is structurally balanced. In addition, a dynamic event-triggered mechanism is proposed, where a set of dynamic factors are introduced into the event-triggered functions to prevent the usage of the global topology information. By virtue of the designed algorithm, it is guaranteed that the heterogeneous multi-agent systems can achieve the bipartite consensus in the presence of the Byzantine agents, and the communication burden among the agents can be reduced. The numerical simulations are conducted to verify the effectiveness of the proposed algorithm.     

基于过滤机制筛选信息的多智能体策略方法

多智能体系统在进行协作或竞争时,会面临联合信息空间扩大、智能体间信息提取效率降低的问题.对此,采用增加过滤机制来筛选信息的多智能体强化学习策略方法(FMAC),以增强智能体间信息交流能力.该方法通过找到彼此相关联的智能体,根据相关性计算智能体的信息贡献,过滤掉无关智能体信息,从而实现在合作、竞争或者混合环境下智能体间有...     

基于间歇事件触发控制二阶多智能体网络的一致性*

针对二阶多智能体网络在无向连通拓扑结构下的间歇控制一致性问题,引入了合适的事件触发控制算法,该算法能够有效的减少控制器的更新次数。同时,模型中的每个智能体具有非线性项,使得多智能体网络更为实际。采用Lyapunov-Krasovskii泛函方法,对网络进行了一致性分析,给出了事件触发控制算法下二阶多智能体网络趋于一致的充分条件,得到了网络通讯间隔的要求,并排除了Zeno现象。最后,数值仿真结果进一步验证了理论分析的正确性。