Q-learning算法及其在囚徒困境问题中的实现

Ｑ－ｌｅａｒｎｉｎｇ是一种优良的强化学习算法。该文首先阐述了Ｑ－ｌｅａｒｎｉｎｇ的基本学习机制,然后以囚徒困境问题为背景,分析、对比Ｔ Ｑ－ｌｅａｒｎｉｎｇ算法与ＴＦＴ算法,验证了 Ｑ－ｌｅａｒｎｉｎｇ算法的优良特性。     

基于Markov对策的多Agent强化学习模型及算法研究

在ＭＤＰ,单Ａｇｅｎｔ可以通过强化学习来寻找问题的最优解。但在多Ａｇｅｎｔ系统中,ＭＤＰ模型不再适用。同样极小极大Ｑ算法只能解决采用零和对策模型的ＭＡＳ学习问题。文中采用非零和Ｍａｒｋｏｖ对策作为多Ａｇｅｎｔ系统学习框架,并提出元对策强化学习的学习模型和元对策Ｑ算法。理论证明元对策Ｑ算法收敛在非零和Ｍａｒｋｏｖ对策的元对策最优解。     

仿真机器人足球中的强化学习

本文总结当前仿真机器人足球中强化学习的研究进展,系统阐述在仿真机器人足球不同决策层次中使用强化学习的不同方法,针对仿真机器人足球的特点讨论当前使用的几种对环境状态空间进行泛化的方法,并展望今后强化学习在仿真机器人足球中的主要应用方向.     

远程教学的Agent模型

利用网络进行远程教学有很多优点,本文讨论如何把Ａｇｅｎｔ技术应用到远程教学中,以增强教学效果,在多Ａｇｅｎｔ通信语言ＫＱＭＬ的基础上,提出了远程教学的Ａｇｅｎｔ模型,设计了Ａｇｅｎｔ的能力编码,通过对搜索教师Ａｇｅｎｔ算法的时间复杂度分析,说明在该模型中搜索教师Ａｇｅｎｔ是可行的和有效的。     

基于Agent的分形供应链自适应协同计算

应用分形、智能Ａｇｅｎｔ和神经网络自适应控制技术,研究分形供应链适应环境变化的结构模式和策略模式．探讨了分形供应链Ａｇｅｎｔ关联结构,提出了分形供应链双层自适应协同计算模式,论述了资源Ａｇｅｎｔ,信息协调Ａｇｅｎｔ,人机交互Ａｇｅｎｔ和领域计算Ａｇｅｎｔ之间的相互作用关系．以一个分形模块的策略协同为分析对象,研究了领域单元的自适应协同计算模式,分析了分形模块的成本模型,并对基于Ａｇｅｎｔ交互的神经网络模型部分进行了算例仿真．     

机器人足球比赛策略仿真系统的开发

多智能体系统（Ｍｕｌｔｉ－ＡｇｅｎｔＳｙｓｔｅｍ）是近来在智能机器人领域兴起的一个新课题。它主要研究多机器人在各种不利的环境条件下,如何相互配合和合作来达到某一目的。微机器人世界杯足球比赛（ＭＩＲＯＳＯＴ）为研究多智能体系统提供既经济又典型的实验场地。本文主要讨论机器人足球比赛所必需的比赛策略及其计算机仿真。本文首先描述了机器人足球比赛几何建模与动态建模,其次提出足球机器人的基本行为与动作仿真,最后讨论了机器人足球比赛策略及其计算机仿真。     

强化学习理论、算法及应用

强化学习（ｒｅｉｎｆｏｒｃｅｍｅｎｔ ｌｅａｒｎｉｎｇ）一词来自于行为心理学,这一理论把行为学习看成是反复试验的过程,从而把环境状态映射成相应的动作。首先全面地介绍了强化学习理论的主要算法,即瞬时差分法、Ｑ－学习算法及自适应启发评价算法;然后介绍了强化学习的应用情况;最后讨论了强化学习目前所要研究的问题。     

基于对象／Agent的计算机集成过程运行系统建模研究

运用面向对象方法学、ＫＡＤＳ原理以及分布式人工智能领域的Ａｇｅｎｔ概念,用对象描述过程领域,用Ａｇｅｎｔ描述过程运行决策系统,建立一种基于对象／Ａｇｅｎｔ的计算机集成过程运行（ＣＩＰＯ）系统模型。通过开发实例系统ＴＥ－ＣＩＰＯＳ,说明了该模型的应用及优点。     

基于Agent的分形供应链自适应协同计算

应用分形、智能Ａｇｅｎｔ和神经网络自适应控制技术,研究分形供应链适应环境变化的结构模式和策略模式．探讨了分形供应链Ａｇｅｎｔ关联结构,提出了分形供应链双层自适应协同计算模式,论述了资源Ａｇｅｎｔ,信息协调Ａｇｅｎｔ,人机交互Ａｇｅｎｔ和领域计算Ａｇｅｎｔ之间的相互作用关系．以一个分形模块的策略协同为分析对象,研究了领
域单元的自适应协同计算模式,分析了分形模块的成本模型,并对基于Ａｇｅｎｔ交互的神经网络模型部分进行了算例仿真．

     

自主机器人自组织结构IRASO的仿真研究

自主机器人车辆具有智能性和快速反应的特点,而在精确推理和实时性之间录示最佳折衷是体系结构的关键文中提出基于分布式多Ａｇｅｎｔ系统的自组织体系结构ＩＲＡＳＯ系统的自组织体系结构ＩＲＡＳＯ,Ａｇｅｎｔ之间动态组合以适应环境变化,公告板系统评估环境势态和指导Ａｇｅｎｔ组织,同时设计了Ａｇｅｎｔ协调工作的空间和时间模型,基于ＴＣＰ／ＩＰ的计算联勇域为异质分布式多Ａｇｅｎｔ的协作运行提供支持,仿真结果表明该     

分层增强学习在足球机器人比赛中的应用

足球机器人的研究是一项挑战性的研究领域，为了设计出智能型的球员必须涉及到计算机、人工智能、视觉及机械学等方面的研究。球员的学习能力是体现其智能的主要标志。如何在不断改变的外界环境中选取合适的动作技巧是在机器人足球比赛中的一个关键问题。该文介绍了马尔可夫决策过程，在半马尔可夫决策模型下，利用分层增强学习算法对不同层次的动作学习和选取同时进行学习。在仿真平台上进行实验，结果表明该学习方法是非常有效的。     

基于UML的多智能体建模分析方法研究

以多智能体系统机器人足球仿真比赛为研究背景 ,对Agent理论和结构模型进行分析 ,探讨了使用UML语言建立Agent模型与结构之间映射关系的方法 .通过对Agent的内部结构与行为的描述 ,实现了A gent建模与其软件建模的统一 .并在开发机器人足球仿真比赛队伍AFU2 0 0 4Agent软件的过程中成功地加以应用 ,为Agent的建模和设计实现开启了新的思路 .     

Training Multiagent Systems by Q‐Learning: Approaches and Empirical Results

Multiagent systems are increasingly present in computational environments. However, the problem of agent design or control is an open research field. Reinforcement learning approaches offer solutions that allow autonomous learning with minimal supervision. The Q‐learning algorithm is a model‐free reinforcement learning solution that has proven its usefulness in single‐agent domains; however, it suffers from dimensionality curse when applied to multiagent systems. In this article, we discuss two approaches, namely TRQ‐learning and distributed Q‐learning, that overcome the limitations of Q‐learning offering feasible solutions. We test these approaches in two separate domains. The first is the control of a hose by a team of robots. The second is the trash disposal problem. Computational results show the effectiveness of Q‐learning solutions to multiagent systems’ control.     

机器人足球赛中基于增强学习的任务分工

本文研究了机器人足球赛中利用增强学习进行角色分工的问题,通过仿真试验 和理论分析,指出文［1］中采取无限作用范围衰减奖励优化模型(infinite horizon disc ou nted model)的Q学习算法对该任务不合适,并用平均奖励模型(average reward model)对 算法进行了改进,实验表明改进后学习的收敛速度以及系统的性能都提高了近一倍．     

基于Q学习Robocup前锋的射门训练

3D仿真机器人是当今人工智能领域里一个极富挑战性的高技术密集型项目。在Robocup 3D比赛中,由于现今球队的人数较少,一个球队的进攻能力往往取决于前锋的个人能力,因此增强前锋的射门能力就显得十分重要。Q学习是一种重要的强化学习方法,将Q学习用到Robocup仿真环境中,使智能体通过在线学习获得射门技巧,并且通过实际比赛证明了算法的有效性。     

机器学习在RoboCup中的应用研究

RoboCup is a particularly good domain for studying multi-agent systems.A wide variety of MAS issues can be studied in robotic soccer,in which the theory,algorithm and architecture of agent system can be evaluated.Because of the inherent complexity of MAS,there are many interests in using machine learning techniques to handle it.This paper investigates and discusses the machine-learning techniques used in RoboCup.The background is firstly presented and the application of machine learning in RoboCup is lately demonstrated with some top simulation teams.The machine-learning system in NDSocTeam is also introduced.Finally some open issues in this field are pointed out.     

On-the-fly generation of multi-robot team formation strategies based on game conditions

This paper describes a new model to automatically generating dynamic formation strategies for robotic soccer applications based on game conditions, regarded to as favorable or unfavorable for a robotic team. Decisions are distributedly computed by the players of a multi-agent team. A game policy is defined and applied by a human coach who establishes the attitude of the team for defending or attacking. A simple neural net model is applied using current and previous game experience to classify the game’s parameters so that the new game conditions can be determined so that a robotic team can modify its strategy on-the-fly. Experiments and results of the proposed model for a robotic soccer team show the promise of the approach.     

增强Q学习在非确定马尔可夫系统寻优问题中的应用

增强学习属于机器学习的一种,它通过与环境的交互获得策略的改进,其在线学习和自适应学习的特点使其成为解决策略寻优问题有力的工具。多智能体系统是人工智能领域的一个研究热点,对于多智能体学习技术的研究需要建立在系统环境模型的基础之上,由于多个智能体的存在,智能体之间的相互影响使得多智能体系统高度复杂,多智能体系统环境属于非确定马尔可夫模型,因此直接把基于马尔可夫模型的增强学习技术引入多智能体系统是不合适的。论文基于智能体间独立的学习机制,提出了一种改进的多智能体Q学习算法,使其适用于非确定马尔可夫环境,并对该学习技术在多智能体系统RoboCup中的应用进行了研究,实验证明了该学习技术的有效性与泛化能力,最后简要给出了多智能体增强学习研究的方向及进一步的工作。     

Cooperative behavior acquisition for mobile robots in dynamically changing real worlds via vision-based reinforcement learning and development

In this paper, we first discuss the meaning of physical embodiment and the complexity of the environment in the context of multi-agent learning. We then propose a vision-based reinforcement learning method that acquires cooperative behaviors in a dynamic environment. We use the robot soccer game initiated by RoboCup (Kitano et al., 1997) to illustrate the effectiveness of our method. Each agent works with other team members to achieve a common goal against opponents. Our method estimates the relationships between a learner's behaviors and those of other agents in the environment through interactions (observations and actions) using a technique from system identification. In order to identify the model of each agent, Akaike's Information Criterion is applied to the results of Canonical Variate Analysis to clarify the relationship between the observed data in terms of actions and future observations. Next, reinforcement learning based on the estimated state vectors is performed to obtain the optimal behavior policy. The proposed method is applied to a soccer playing situation. The method successfully models a rolling ball and other moving agents and acquires the learner's behaviors. Computer simulations and real experiments are shown and a discussion is given.     

The CMUnited-98 champion small-robot team

Robotic soccer presents a large spectrum of challenging research opportunities. In this article, we present the main research and technical contributions of our champion CMUnited-98 small-robot team. The team is a multiagent robotic system with global perception, and distributed cognition and action. We introduce our new robot motion algorithm that reactively generates motion control to account for the target point, the desired robot orientation and obstacle avoidance. Our robots exhibit successful collision-free motion in the highly dynamic robotic soccer environment. At the strategic and decision-making level, we present the role-based behaviors of the CMUnited-98 robotic agents. Team collaboration is remarkably achieved through a new algorithm that allows for team agents to anticipate possible collaboration opportunities. Robots position themselves strategically in open positions that increase passing opportunities. The article terminates with a summary of the results of the RoboCup-98 games in which the CMUnited-98 small-robot team scored a total of 25 goals and suffered 6 goals in the five games that it played.