Learning in groups of traffic signals

Computer science in general, and artificial intelligence and multiagent systems in particular, are part of an effort to build intelligent transportation systems. An efficient use of the existing infrastructure relates closely to multiagent systems as many problems in traffic management and control are inherently distributed. In particular, traffic signal controllers located at intersections can be seen as autonomous agents. However, challenging issues are involved in this kind of modeling: the number of agents is high; in general agents must be highly adaptive; they must react to changes in the environment at individual level while also causing an unpredictable collective pattern, as they act in a highly coupled environment. Therefore, traffic signal control poses many challenges for standard techniques from multiagent systems such as learning. Despite the progress in multiagent reinforcement learning via formalisms based on stochastic games, these cannot cope with a high number of agents due to the combinatorial explosion in the number of joint actions. One possible way to reduce the complexity of the problem is to have agents organized in groups of limited size so that the number of joint actions is reduced. These groups are then coordinated by another agent, a tutor or supervisor. Thus, this paper investigates the task of multiagent reinforcement learning for control of traffic signals in two situations: agents act individually (individual learners) and agents can be “tutored”, meaning that another agent with a broader sight will recommend a joint action.     

A layered approach to learning coordination knowledge in multiagent environments

Multiagent learning involves acquisition of cooperative behavior among intelligent agents in order to satisfy the joint goals. Reinforcement Learning (RL) is a promising unsupervised machine learning technique inspired from the earlier studies in animal learning. In this paper, we propose a new RL technique called the Two Level Reinforcement Learning with Communication (2LRL) method to provide cooperative action selection in a multiagent environment. In 2LRL, learning takes place in two hierarchical levels; in the first level agents learn to select their target and then they select the action directed to their target in the second level. The agents communicate their perception to their neighbors and use the communication information in their decision-making. We applied 2LRL method in a hunter-prey environment and observed a satisfactory cooperative behavior. Guray Erus received the B.S. degree in computer engineering in 1999, and the M.S. degree in cognitive sciences, in 2002, from Middle East Technical University (METU), Ankara, Turkey. He is currently a teaching and research assistant in Rene“ Descartes University, Paris, France, where he prepares a doctoral dissertation on object detection on satellite images, as a member of the intelligent perception systems group (SIP-CRIP5). His research interests include multi-agent systems and image understanding. Faruk Polat is a professor in the Department of Computer Engineering of Middle East Technical University, Ankara, Turkey. He received his B.Sc. in computer engineering from the Middle East Technical University, Ankara, in 1987 and his M.S. and Ph.D. degrees in computer engineering from Bilkent University, Ankara, in 1989 and 1993, respectively. He conducted research as a visiting NATO science scholar at Computer Science Department of University of Minnesota, Minneapolis in 1992–93. His research interests include artificial intelligence, multi-agent systems and object oriented data models.     

TRAMMAS: A tracing model for multiagent systems

Agent's flexibility and autonomy, as well as their capacity to coordinate and cooperate, are some of the features which make multiagent systems useful to work in dynamic and distributed environments. These key features are directly related to the way in which agents communicate and perceive each other, as well as their environment and surrounding conditions. Traditionally, this has been accomplished by means of message exchange or by using blackboard systems. These traditional methods have the advantages of being easy to implement and well supported by multiagent platforms; however, their main disadvantage is that the amount of social knowledge in the system directly depends on every agent actively informing of what it is doing, thinking, perceiving, etc. There are domains, for example those where social knowledge depends on highly distributed pieces of data provided by many different agents, in which such traditional methods can produce a great deal of overhead, hence reducing the scalability, efficiency and flexibility of the multiagent system. This work proposes the use of event tracing in multiagent systems, as an indirect interaction and coordination mechanism to improve the amount and quality of the information that agents can perceive from both their physical and social environment, in order to fulfill their goals more efficiently. In order to do so, this work presents an abstract model of a tracing system and an architectural design of such model, which can be incorporated to a typical multiagent platform.     

Zeus: A toolkit for building distributed multiagent systems

The multiagent systems approach of knowledge- level cooperation between autonomous agents promises significant benefits to distributed systems engineering, such as enhanced interoperability, scalability, and reconfigurability. However, thus far, because of the innate difficulty of constructing multiagent systems, this promise has been largely unrealized. Hence there is an emerging desire among agent developers to move away from developing point solutions to point problems in favor of developing methodologies and toolkits for building distributed multiagent systems. This philosophy led to the development of the ZEUS Agent Building Toolkit, which facilitates the rapid development of collaborative agent applications through the provision of a library of agent- level components and an environment to support the agent-building process. The ZEUS toolkit is a synthesis of established agent technologies with some novel solutions to provide an integrated collaborative agent-building environment.     

An experts approach to strategy selection in multiagent meeting scheduling

In the multiagent meeting scheduling problem, agents negotiate with each other on behalf of their users to schedule meetings. While a number of negotiation approaches have been proposed for scheduling meetings, it is not well understood how agents can negotiate strategically in order to maximize their users’ utility. To negotiate strategically, agents need to learn to pick good strategies for negotiating with other agents. In this paper, we show how agents can learn online to negotiate strategically in order to better satisfy their users’ preferences. We outline the applicability of experts algorithms to the problem of learning to select negotiation strategies. In particular, we show how two different experts approaches, plays [3] and Exploration–Exploitation Experts (EEE) [10] can be adapted to the task. We show experimentally the effectiveness of our approach for learning to negotiate strategically.     

Infrastructures for the environment of multiagent systems

The notion of environment is receiving an increasing attention in the development of multiagent applications. This is witnessed by the emergence of a number of infrastructures providing agent designers with useful means to develop the agent environment, and thus to structure an effective multiagent application. In this paper we analyse the role and features of such infrastructures, and survey some relevant examples. We endorse a general viewpoint where the environment of a multiagent system is seen as a set of basic bricks we call environment abstractions, which (i) provide agents with services useful for achieving individual and social goals, and (ii) are supported by some underlying software infrastructure managing their creation and exploitation. Accordingly, we focus the survey on the opportunities that environment infrastructures provide to system designers when developing multiagent applications.     

Environment as a first class abstraction in multiagent systems

The current practice in multiagent systems typically associates the environment with resources that are external to agents and their communication infrastructure. Advanced uses of the environment include infrastructures for indirect coordination, such as digital pheromones, or support for governed interaction in electronic institutions. Yet, in general, the notion of environment is not well defined. Functionalities of the environment are often dealt with implicitly or in an ad hoc manner. This is not only poor engineering practice, it also hinders engineers to exploit the full potential of the environment in multiagent systems. In this paper, we put forward the environment as an explicit part of multiagent systems.We give a definition stating that the environment in a multiagent system is a first-class abstraction with dual roles: (1) the environment provides the surrounding conditions for agents to exist, which implies that the environment is an essential part of every multiagent system, and (2) the environment provides an exploitable design abstraction for building multiagent system applications. We discuss the responsibilities of such an environment in multiagent systems and we present a reference model for the environment that can serve as a basis for environment engineering. To illustrate the power of the environment as a design abstraction, we show how the environment is successfully exploited in a real world application. Considering the environment as a first-class abstraction in multiagent systems opens up new horizons for research and development in multiagent systems.     

Analyzing and visualizing multiagent rewards in dynamic and stochastic domains

The ability to analyze the effectiveness of agent reward structures is critical to the successful design of multiagent learning algorithms. Though final system performance is the best indicator of the suitability of a given reward structure, it is often preferable to analyze the reward properties that lead to good system behavior (i.e., properties promoting coordination among the agents and providing agents with strong signal to noise ratios). This step is particularly helpful in continuous, dynamic, stochastic domains ill-suited to simple table backup schemes commonly used in TD(λ)/Q-learning where the effectiveness of the reward structure is difficult to distinguish from the effectiveness of the chosen learning algorithm. In this paper, we present a new reward evaluation method that provides a visualization of the tradeoff between the level of coordination among the agents and the difficulty of the learning problem each agent faces. This method is independent of the learning algorithm and is only a function of the problem domain and the agents’ reward structure. We use this reward property visualization method to determine an effective reward without performing extensive simulations. We then test this method in both a static and a dynamic multi-rover learning domain where the agents have continuous state spaces and take noisy actions (e.g., the agents’ movement decisions are not always carried out properly). Our results show that in the more difficult dynamic domain, the reward efficiency visualization method provides a two order of magnitude speedup in selecting good rewards, compared to running a full simulation. In addition, this method facilitates the design and analysis of new rewards tailored to the observational limitations of the domain, providing rewards that combine the best properties of traditional rewards.     

Locality-sensitive task allocation and load balancing in networked multiagent systems: Talent versus centrality

With the development of large scale multiagent systems, agents are always organized in network structures where each agent interacts only with its immediate neighbors in the network. Coordination among networked agents is a critical issue which mainly includes two aspects: task allocation and load balancing; in traditional approach, the resources of agents are crucial to their abilities to get tasks, which is called talent-based allocation. However, in networked multiagent systems, the tasks may spend so much communication costs among agents that are sensitive to the agent localities; thus this paper presents a novel idea for task allocation and load balancing in networked multiagent systems, which takes into account both the talents and centralities of agents. This paper first investigates the comparison between talent-based task allocation and centrality-based one; then, it explores the load balancing of such two approaches in task allocation. The experiment results show that the centrality-based method can reduce the communication costs for single task more effectively than the talent-based one, but the talent-based method can generally obtain better load balancing performance for parallel tasks than the centrality-based one.     

Finding influential agent groups in complex multiagent software systems based on citation network analyses

Current complex engineering software systems are often composed of many components and can be built based on a multiagent approach, resulting in what are called complex multiagent software systems. In a complex multiagent software system, various software agents may cite the operation results of others, and the citation relationships among agents form a citation network; therefore, the importance of a software agent in a system can be described by the citations from other software agents. Moreover, the software agents in a system are often divided into various groups, and each group contains the agents undergoing similar tasks or having related functions; thus, it is necessary to find the influential agent group (not only the influential individual agent) that can influence the system outcome utilities more than the others. To solve such a problem, this paper presents a new model for finding influential agent groups based on group centrality analyses in citation networks. In the presented model, a concept of extended group centrality is presented to evaluate the impact of an agent group, which is collectively determined by both direct and indirect citations from other agents outside the group. Moreover, the presented model addresses two typical types of agent groups: one is the adjacent group where agents of a group are adjacent in the citation network, and the other is the scattering group where agents of a group are distributed separately in the citation network. Finally, we present case studies and simulation experiments to prove the effectiveness of the presented model.     

Consensus of multiagent systems based on disturbance observer

The objectives of this work are the development and design of disturbance observers (DO’s) for a team of agents that accomplish consensus on agents’ states in the presence of exogenous disturbances. A pinning control strategy is designed for a part of agents of the multiagent systems without disturbances, and this pinning control can bring multiple agents’ states to reaching an expected consensus value. Under the effect of the disturbances, nonlinear disturbance observers are developed for disturbances generated by an exogenous system to estimate the disturbances. Asymptotical consensus of the multiagent systems with disturbances under the composite controller can be achieved. Finally, by applying an example of multiagent systems with switching topologies and exogenous disturbances, the design of the parameters of DO’s are illuminated.     

Intelligent multiagent application system in an AI system

This article presents an intelligent multiagent application system in AI. The research trend into multiagents is changing from a centralized computing environment to a distributed computing environment. Also, the research into multiagents can be changed to a mobile environment. Initially, the study of multiagents is from research into human modeling. Therefore, we fi rst present a brief concept of a mobile multiagent, and then we present some application areas for mobile multiagents, especially in elearning, bioinformatics, control, and information retrieval, etc. Finally, we present the research theme of multiagents in AI. This work was presented in part at the 12th International Symposium on Articial Life and Robotics, Oita, January 25–27, 2007     

Design of coordination strategies in multiagent systems via genetic fuzzy systems

This paper suggests an evolutionary approach to design coordination strategies for multiagent systems. Emphasis is given to auction protocols since they are of utmost importance in many real world applications such as power markets. Power markets are one of the most relevant instances of multiagent systems and finding a profitable bidding strategy is a key issue to preserve system functioning and improve social welfare. Bidding strategies are modeled as fuzzy rule-based systems due to their modeling power, transparency, and ability to naturally handle imprecision in input data, an essential ingredient to a multiagent system act efficiently in practice. Specific genetic operators are suggested in this paper. Evolution of bidding strategies uncovers unknown and unexpected agent behaviors and allows a richer analysis of auction mechanisms and their role as a coordination protocol. Simulation experiments with a typical power market using actual thermal plants data show that the evolutionary, genetic-based design approach evolves strategies that enhance agents profitability when compared with the marginal cost-based strategies commonly adopted     

Introduction to the special issue on normative multiagent systems

This special issue contains four selected and revised papers from the second international workshop on normative multiagent systems, for short NorMAS07 (Boella et al. (eds) Normative multiagent systems. Dagstuhl seminar proceedings 07122, 2007), held at Schloss Dagstuhl, Germany, in March 2007. At the workshop a shift was identified in the research community from a legal to an interactionist view on normative multiagent systems. In this editorial we discuss the shift, examples, and 10 new challenges in this more dynamic setting, which we use to introduce the papers of this special issue.     

Two distribution methods for multiagent traffic simulations

Modeling and simulation play an important role in transportation networks analysis. With the widespread use of personalized real-time information sources, the behavior of the simulation depends heavily on individual travelers reactions to the received information. As a consequence, it is relevant for the simulation model to be individual-centered, and multiagent simulation is the most promising paradigm in this context. However, representing the movements of realistic numbers of travelers within reasonable execution times requires significant computational resources. It also requires relevant methods, architectures and algorithms that respect the characteristics of transportation networks. In this paper, we define two multiagent simulation models representing the existing sequential multiagent traffic simulations. The first model is fundamental diagram-based model, in which travelers do not interact directly and use a fundamental diagram of traffic flow to continuously compute their speeds. The second model is car-following based, in which travelers interact with their neighbors to adapt their speeds to their surrounding environment. Then we define patterns to distribute these simulations in a high-performance environment. The first is agent-based and distributes agents equally between available computation units. The second pattern is environment-based and partitions the environment over the different units. The results show that agent-based distribution is more efficient with fundamental diagram-based model simulations while environment-based distribution is more efficient with car following-based simulations.     

Approximately adaptive neural cooperative control for nonlinear multiagent systems with performance guarantee

This paper studies the cooperative control problem for a class of multiagent dynamical systems with partially unknown nonlinear system dynamics. In particular, the control objective is to solve the state consensus problem for multiagent systems based on the minimisation of certain cost functions for individual agents. Under the assumption that there exist admissible cooperative controls for such class of multiagent systems, the formulated problem is solved through finding the optimal cooperative control using the approximate dynamic programming and reinforcement learning approach. With the aid of neural network parameterisation and online adaptive learning, our method renders a practically implementable approximately adaptive neural cooperative control for multiagent systems. Specifically, based on the Bellman's principle of optimality, the Hamilton–Jacobi–Bellman (HJB) equation for multiagent systems is first derived. We then propose an approximately adaptive policy iteration algorithm for multiagent cooperative control based on neural network approximation of the value functions. The convergence of the proposed algorithm is rigorously proved using the contraction mapping method. The simulation results are included to validate the effectiveness of the proposed algorithm.     

Multiagent systems and information retrieval our experience with X.MAS

Autonomous agents and multiagent systems have been successfully applied to a number of problems and have been largely used in different application fields. In particular, in this paper we are interested in information retrieval. In fact, in this field multiagent solutions are very useful and effective since they decouple the problem in a network of software agents that interact to solve problems that are beyond the individual capabilities or knowledge. In so doing, multiagent systems allow to overwhelm typical problems of single agent and centralized approaches. To discuss the lesson learnt in using the multiagent technology in the field of information retrieval, in this paper, we present our experience in using X.MAS, a generic multiagent architecture aimed at retrieving, filtering and reorganizing information according to user interests. To this end, after presenting X.MAS, we illustrate six applications built upon it. Our conclusion is that multiagent technology is quite effective to design and realize concrete information retrieval applications.     

Design and implementation of a multiagent stock trading system

Stock trading is one of the key items in an economy and estimating its behavior and taking the best decision in it are among the most challenging issues. Solutions based on intelligent agent systems are proposed to cope with those challenges. Agents in a multiagent system (MAS) can share a common goal or they can pursue their own interests. That nature of MASs exactly fits the requirements of a free market economy. Although existing studies include noteworthy proposals on agent‐based market simulation and researchers discuss theoretical design issues of agent‐based stock exchange systems, unfortunately only a very few of the studies consider exact development and implementation of multiagent stock trading systems within the software engineering perspective and guides to the software engineers for constructing such software systems starting from scratch. To fill this gap, in this paper, we discuss the development of a multiagent‐based stock trading system by taking into consideration software design according to a well‐defined agent oriented software engineering methodology and implementation with a widely‐used MAS software development framework. Each participant in the system is first designed as belief–desire–intention agents with their facts, goals, and plans, and then belief–desire–intention reasoning and behavioral structure of the designed agents are implemented. Lessons learned during design and development within the software engineering perspective and evaluation of the implemented multiagent stock exchange system are also reported. Copyright © 2011 John Wiley & Sons, Ltd.     

Safety and Security in Multiagent Systems: Report on the 2nd SASEMAS workshop (SASEMAS'05)

As intelligent autonomous agents and multiagent systems' applications become more pervasive, it becomes increasingly more important to understand the risks associated with using these systems. Incorrect or inappropriate agent behaviour can have harmful effects including financial cost, loss of data, and injury to humans or systems. Thus, security and safety are two central issues when developing and deploying such systems.However, the process of developing safe and secure multiagent systems, and verifying and validating them, is much more difficult than for conventional software systems. This is due to many agent-related aspects, such as the complex and rich multiagent environments, the risks involved in such environments, and the characteristics that can be found in agent systems such as learning, dynamic reacting and adapting. Hence, new and different techniques and perspectives are required to assist with the development and deployment of such systems.The Safety and Security in Multiagent Systems (SASEMAS) workshop presents new developments, and lessons learned from real world cases, and it provides a forum for the exchange of ideas and discussion on areas related to security and safety in multiagent systems.