A hybrid agent architecture integrating desire,intention and reinforcement learning

This paper presents a hybrid agent architecture that integrates the behaviours of BDI agents, specifically desire and intention, with a neural network based reinforcement learner known as Temporal Difference-Fusion Architecture for Learning and COgNition (TD-FALCON). With the explicit maintenance of goals, the agent performs reinforcement learning with the awareness of its objectives instead of relying on external reinforcement signals. More importantly, the intention module equips the hybrid architecture with deliberative planning capabilities, enabling the agent to purposefully maintain an agenda of actions to perform and reducing the need of constantly sensing the environment. Through reinforcement learning, plans can also be learned and evaluated without the rigidity of user-defined plans as used in traditional BDI systems. For intention and reinforcement learning to work cooperatively, two strategies are presented for combining the intention module and the reactive learning module for decision making in a real time environment. Our case study based on a minefield navigation domain investigates how the desire and intention modules may cooperatively enhance the capability of a pure reinforcement learner. The empirical results show that the hybrid architecture is able to learn plans efficiently and tap both intentional and reactive action execution to yield a robust performance.     

Group planning with time constraints

Embedding planning systems in real-world domains has led to the necessity of Distributed Continual Planning (DCP) systems where planning activities are distributed across multiple agents and plan generation may occur concurrently with plan execution. A key challenge in DCP systems is how to coordinate activities for a group of planning agents. This problem is compounded when these agents are situated in a real-world dynamic domain where the agents often encounter differing, incomplete, and possibly inconsistent views of their environment. To date, DCP systems have only focused on cases where agents’ behavior is designed to optimize a global plan. In contrast, this paper presents a temporal reasoning mechanism for self-interested planning agents. To do so, we model agents’ behavior based on the Belief-Desire-Intention (BDI) theoretical model of cooperation, while modeling dynamic joint plans with group time constraints through creating hierarchical abstraction plans integrated with temporal constraints network. The contribution of this paper is threefold: (i) the BDI model specifies a behavior for self interested agents working in a group, permitting an individual agent to schedule its activities in an autonomous fashion, while taking into consideration temporal constraints of its group members; (ii) abstract plans allow the group to plan a joint action without explicitly describing all possible states in advance, making it possible to reduce the number of states which need to be considered in a BDI-based approach; and (iii) a temporal constraints network enables each agent to reason by itself about the best time for scheduling activities, making it possible to reduce coordination messages among a group. The mechanism ensures temporal consistency of a cooperative plan, enables the interleaving of planning and execution at both individual and group levels. We report on how the mechanism was implemented within a commercial training and simulation application, and present empirical evidence of its effectiveness in real-life scenarios and in reducing communication to coordinate group members’ activities.     

A plan-based agent architecture for interpreting natural language dialogue

This paper describes a plan-based agent architecture for modelling NL cooperative dialogue; in particular, the paper focuses on the interpretation of dialogue and on the explanation of its coherence by means of the recognition of the speakers' underlying intentions. The approach we propose makes it possible to analyze an explain in a uniform way several apparently unrelated linguistic phenomena, which have been often studied separately and treated via ad-hoc methods in the models of dialogue presented in the literature. Our model of linguistic interaction is based on the idea that dialogue can be seen as any other interaction among agents: therefore, domain-level and linguistic actions are treated in a similar way.Our agent architecture is based on a two-level representation of the knowledge about acting: at the metalevel, the agent modelling (AM) plans describe the recipes for plan formation and execution (they are a declarative representation of a reactive planner); at the object level, the domain and communicative actions are defined. The AM plans are used to identify the goals underlying the actions performed by an observed agent; the recognized plans constitute the dialogue context, where the intentions of all participants are stored in a structured way, in order to be used in the interpretation of the subsequent dialogue turns.     

Agent主动目标的形式化模型

在agent结构中,主动目标是一个功能上自含且有自己独立控制流的实体.给出了主动目标相关的语法定义以及主动目标运行的操作语义,而且主动目标驱动下的BDI agent结构也被形式化地定义出来.区别于以往的一些BDI agent结构,目标不是隐含地表示而是作为实体显式地表示在agent结构中,使agent结构很自然地支持并行的目标,这被认为是agent理性行为的一个重要方面.此外,对目标的显式定义也为agent在动态的环境中对承诺的重新考虑带来了方便.     

Roboskeleton: An architecture for coordinating robot soccer agents

SkeletonAgent is an agent framework whose main feature is to integrate different artificial intelligent skills, like planning or learning, to obtain new behaviours in a multi-agent environment. This framework has been previously instantiated in a deliberative domain (electronic tourism), where planning was used to integrate Web information in a tourist plan. RoboSkeleton results from the instantiation of the same framework, SkeletonAgent, in a very different domain, the robot soccer. This paper shows how this architecture is used to obtain collaborative behaviours in a reactive domain. The paper describes how the different modules of the architecture for the robot soccer agents are designed, directly showing the flexibility of our framework.     

Cooperative, hybrid agent architecture for real-time traffic signal control

This paper presents a new hybrid, synergistic approach in applying computational intelligence concepts to implement a cooperative, hierarchical, multiagent system for real-time traffic signal control of a complex traffic network. The large-scale traffic signal control problem is divided into various subproblems, and each subproblem is handled by an intelligent agent with a fuzzy neural decision-making module. The decisions made by lower-level agents are mediated by their respective higher-level agents. Through adopting a cooperative distributed problem solving approach, coordinated control by the agents is achieved. In order for the multiagent architecture to adapt itself continuously to the dynamically changing problem domain, a multistage online learning process for each agent is implemented involving reinforcement learning, learning rate and weight adjustment as well as dynamic update of fuzzy relations using an evolutionary algorithm. The test bed used for this research is a section of the Central Business District of Singapore. The performance of the proposed multiagent architecture is evaluated against the set of signal plans used by the current real-time adaptive traffic control system. The multiagent architecture produces significant improvements in the conditions of the traffic network, reducing the total mean delay by 40% and total vehicle stoppage time by 50%.     

An architecture for a self-improving instructional planner for intelligent tutoring systems

Machine instructional planners use changing and uncertain data to incrementally configure plans and control the execution and dynamic refinement of these plans. Current instructional planners cannot adequately plan, replan, and monitor the delivery of instruction. This is due in part to the fact that current instructional planners are incapable of planning in a global context, developing competing plans in parallel, monitoring their planning behavior, and dynamically adapting their control behavior. In response to these and other deficiencies of instructional planners a generic system architecture based on the blackboard model was implemented. This self-improving instructional planner (SUP) dynamically creates instructional plans, requests execution of these plans, replans, and improves its planning behavior based on a student's responses to tutoring. Global planning was facilitated by explicitly representing decisions about past, current, and future plans on a global data structure called the plan blackboard. Planning in multiple worlds is facilitated by labeling plan decisions by the context in which they were generated. Plan monitoring was implemented as a set of monitoring knowledge sources. The flexible control capability for instructional planner was adapted from the blackboard architecture BB1. The explicit control structure of SUP enabled complex and flexible planning behavior while maintaining a simple planning architecture.     

Inferring threats in urban environments with uncertain and approximate data: an agent-based approach

In this article we discuss the problem of inferring threats in an urban environment, where the knowledge of the environment involves multiple types of intelligence and infrastructure data, and is by nature uncertain or approximate. We use a collection of situation-aware agents to infer potential threats in such environments, where agents are responsible for event correlation and situation assessment. We review the weaknesses of a current approach to threat assessment in Homeland Security and then describe our agent-based approach. The key innovations of our agent-based approach are: an ontological commitment to events and situations, fuzzy event correlation, fuzzy situation assessment, adaptability and learning during threat assessment operations, and an enhancement of traditional belief-desire-intention (BDI) agents with situation awareness. We describe the properties of situation-aware BDI agents and discuss the implementation of them on a variety of BDI agent platforms. Lastly, we discuss the interoperability of these platforms and address the issue of scalability through coupling to large-scale peer-to-peer overlays.     

一种基于强化学习的学习Agent

强化学习通过感知环境状态和从环境中获得不确定奖赏值来学习动态系统的最优行为策略,是构造智能Agent的核心技术之一,在面向Agent的开发环境AODE中扩充BDI模型,引入策略和能力心智成分,采用强化学习技术实现策略构造函数,从而提出一种基于强化学习技术的学习Agent,研究AODE中自适应Agent物结构和运行方式,使智能Agent具有动态环境的在线学习能力,有效期能够有效地满足Agent各种心智要求。     

Extending BDI plan selection to incorporate learning from experience

An important drawback to the popular Belief, Desire, and Intentions (BDI) paradigm is that such systems include no element of learning from experience. We describe a novel BDI execution framework that models context conditions as decision trees, rather than boolean formulae, allowing agents to learn the probability of success for plans based on experience. By using a probabilistic plan selection function, the agents can balance exploration and exploitation of their plans. We extend earlier work to include both parameterised goals and recursion and modify our previous approach to decision tree confidence to include large and even non-finite domains that arise from such consideration. Our evaluation on a pre-existing program that relies heavily on recursion and parametrised goals confirms previous results that naive learning fails in some circumstances, and demonstrates that the improved approach learns relatively well.     

情感等级BDI主体模型研究

信念、愿望和意图(BDI)模型是近年来影响最为深远的主体技术之一。文中把命题动态逻辑和无穷值的ukasiewicz逻辑进行融合后对情感等级BDI主体模型进行了形式化。为通过信念度、愿望度、意图度、害怕度、焦虑度和自信度对不确定性行为进行表示和推理,把相应的公理添加到ukasiewicz逻辑中。文中的情感等级BDI主体模型的行为是通过添加具体条件的每种背景的不同测度来决定,清晰地表示主体的心理状态和情感状态的不确定性。文中对情感等级BDI模型进行公理化,并说明它们对主体行为的影响。此模型可较轻易地向包括其它心理状态和情感状态的主体进行推广。文中在给出情感等级BDI模型的语言、语义及公理和演绎规则后,证明此逻辑系统的可靠性和完全性。随后给出情感等级BDI主体模型的不同背景之间的相互关系,并对该主体的买房行动进行实例分析。本研究立足于不确定性的表示和推理,旨在为分布式人工智能提供形式支持。     

Declarative planning in procedural agent architectures

Practical agent languages and their corresponding architectures have often relied on a static plan library with more or less direct trigger-response activation mechanisms as a source for agent behaviours for the sake of runtime efficiency. Although efficient, such a language design choice severely limits an agent’s ability to reason about its goals and adapt to unforeseen circumstances after being deployed. This effectively delegates the task of planning to the designers themselves, who must design plan libraries able to cope with every foreseeable situation an agent might find itself in by designing plans to deal with any contingency. In this paper we develop a formal conversion process from traditional BDI agent languages into declarative planning. Using this conversion process, we show how to integrate domain independent planning algorithms into the BDI interpreter, allowing a designer to program an agent not only through the trigger-response mechanism used in traditional languages, but also in terms of declarative goals. Our contribution here is twofold: firstly we increase an agent’s ability to cope with unforeseen situations and secondly we unburden an agent designer from having to define multiple plan combinations that could be easily generated by a planner.     

An abstract knowledge based approach to diagnosis and recovery of plan failure in multi-agent systems

Open multi-agent systems (MAS) are decentralised and distributed systems that consist of a large number of loosely coupled autonomous agents. In the absence of centralised control they tend to be difficult to manage, especially in an open environment, which is dynamic, complex, distributed and unpredictable. This dynamism and uncertainty in an open environment gives rise to unexpected plan failures. In this paper we present an abstract knowledge based approach for the diagnosis and recovery of plan action failures. Our approach associates a sentinel agent with each problem solving agent in order to monitor the problem solving agent’s interactions. The proposed approach also requires the problem solving agents to be able to report on the status of a plan’s actions.Once an exception is detected the sentinel agents start an investigation of the suspected agents. The sentinel agents collect information about the status of failed plan abstract actions and knowledge about agents’ mental attitudes regarding any failed plan. The sentinel agent then uses this abstract knowledge and the agents’ mental attitudes, to diagnose the underlying cause of the plan failure. The sentinel agent may ask the problem solving agent to retry their failed plan based on the diagnostic result.     

Introducing a novel model of belief–desire–intention agent for urban land use planning

Land use planning is a potentially demanding search and optimization task that has been challenged by numerous researchers in the field of spatial planning. Agent and multi-agent systems are examples of the modern concepts, which have been gaining more attention in challenging spatial issues recently. Although the efficiency of belief, desire, and intention (BDI) architecture of agents is validated in varieties of sciences, its uses in Geospatial Information Systems (GIS) and specifically among spatial planners is still burgeoning. In this paper, we attempted to integrate the concepts of BDI agent architecture into spatial issues; as a result, a novel spatial agent model is designed and implemented to analyze the urban land use planning. The proposed approach was checked in urban land use planning problems using a case study in a municipal area. The result of implementation showed the effects of spatial agents' behaviors such as intention, commitment, and interaction on their decision.     

Architectural Methodology Based on Intentional Configuration of Behaviors

Intelligence has been an object of study for a long time. Different architectures try to capture and reproduce these aspects into artificial systems (or agents), but there is still no agreement on how to integrate them into a general framework. With this objective in mind, we propose an architectural methodology based on the idea of intentional configuration of behaviors. Behavior‐producing modules are used as basic control components that are selected and modified dynamically according to the intentions of the agent. These intentions are influenced by the situation perceived, knowledge about the world, and internal variables that monitor the state of the agent. The architectural methodology preserves the emergence of functionality associated with the behavior‐based paradigm in the more abstract levels involved in configuring the behaviors. Validation of this architecture is done using a simulated world for mobile robots, in which the agent must deal with various goals such as managing its energy and its well‐being, finding targets, and acquiring knowledge about its environment. Fuzzy logic, a topologic map learning algorithm, and activation variables with a propagation mechanism are used to implement the architecture for this agent.     

Cooperation and Competition in ALIAS: A Logic Framework for Agents that Negotiate

This paper presents ALIAS, an agent architecture based on intelligent logic agents, where the main form of agent reasoning is abduction. The system is particularly suited for solving problems where knowledge is incomplete, where agents may need to make reasonable hypotheses about the problem domain and other agents, and where the raised hypotheses have to be consistent for the overall set of agents. ALIAS agents are pro-active, exhibiting a goal-directed behavior, and autonomous, since each one can solve problems using its own private knowledge base. ALIAS agents are also social, because they are able to interact with other agents, in order to cooperatively solve problems. The coordination mechanisms are modeled by means of LAILA, a logic-based language which allows to express intra-agent reasoning and inter-agent coordination. As an application, we show how LAILA can be used to implement inter-agent dialogues, e.g., for negotiation. In particular, LAILA is well-suited to coordinate the process of negotiation aimed at exchanging resources between agents, thus allowing them to execute the plans to achieve their goals.     

ON THE ROLE OF BDI MODELING FOR INTEGRATED CONTROL AND COORDINATED BEHAVIOR IN AUTONOMOUS AGENTS

This paper describes an architecture for controlling and coordinating autonomous agents, building on previous work addressing reactive and deliberative control methods. The proposed multilayered hybrid architecture allows a rationally bounded, goal-directed agent to reason predictively about potential conflicts by constructing knowledge level models that explain other agents' observed behaviors and hypothesize their beliefs, desires, and intentions; at the same time, it enables the agent to operate autonomously, to react promptly to changes in its real-time environment, and to coordinate its actions effectively with other agents. A principal aim of this research is to understand the role dzfferent functional capabilities play in constraining an agent5 behavior under varying environmental conditions. To this end, an experimental test bed has been constructed comprising a simulated multi-agent world in which a variety of agent configurations and behaviors have been investigated. A number of experimentalfindings are reported.     

一种实时Agent体系结构

提出了一种应用于实时环境的Agent体系结构,集成了慎思式和反应式Agent的主要行为特征,支持多Agent之间的协同求解,并在内部机制上采用集中控制和规划缓冲等技术满足系统对实时性和稳定性的需要。最后给出了该体系结构在实时监控系统中的一个具体应用,论证了其设计的合理性。     

REACTIVE AGENT DESIGN FOR INTELLIGENT TUTORING SYSTEMS

This paper presents a new approach to the analysis and design of intelligent tutoring systems (ITS), based on reactive principles and cognitive models, this way leading to multiagent architecture. In these kinds of models, the analysis problem is treated bottom-up, as opposed to that of traditional artificial intelligence (AI), i.e., top down. We present one ITS example called Makatsina (meaning tutor in TOTONACA, a Mexican pre-Columbian language), constructed according to this approach, which teaches the skills necessary to solve the truss analysis problem by the method of joints. This learning domain is an integration skill. The classical ITS work is based on explicit goals and an internal representation of the environment. The new approach has reactive agents which have no representation of their environment and act using a stimulus response behavior type. In this way they can respond to the present state of the environment in which they are embedded. With these elements, errors, and teaching plans, each agent behaves as an expert assistant that is able to handle different teaching methods. Reactive agent programming is found to be simple because agents have simple behaviors. The difficulty lies in the interaction mechanism analysis and design between the environment and the intelligent reactive system.