Opportunities for multiagent systems and multiagent reinforcement learning in traffic control

The increasing demand for mobility in our society poses various challenges to traffic engineering, computer science in general, and artificial intelligence and multiagent systems in particular. As it is often the case, it is not possible to provide additional capacity, so that a more efficient use of the available transportation infrastructure is necessary. This relates closely to multiagent systems as many problems in traffic management and control are inherently distributed. Also, many actors in a transportation system fit very well the concept of autonomous agents: the driver, the pedestrian, the traffic expert; in some cases, also the intersection and the traffic signal controller can be regarded as an autonomous agent. However, the “agentification” of a transportation system is associated with some challenging issues: the number of agents is high, typically agents are highly adaptive, they react to changes in the environment at individual level but cause an unpredictable collective pattern, and act in a highly coupled environment. Therefore, this domain poses many challenges for standard techniques from multiagent systems such as coordination and learning. This paper has two main objectives: (i) to present problems, methods, approaches and practices in traffic engineering (especially regarding traffic signal control); and (ii) to highlight open problems and challenges so that future research in multiagent systems can address them.     

Learning in groups of traffic signals

Computer science in general, and artificial intelligence and multiagent systems in particular, are part of an effort to build intelligent transportation systems. An efficient use of the existing infrastructure relates closely to multiagent systems as many problems in traffic management and control are inherently distributed. In particular, traffic signal controllers located at intersections can be seen as autonomous agents. However, challenging issues are involved in this kind of modeling: the number of agents is high; in general agents must be highly adaptive; they must react to changes in the environment at individual level while also causing an unpredictable collective pattern, as they act in a highly coupled environment. Therefore, traffic signal control poses many challenges for standard techniques from multiagent systems such as learning. Despite the progress in multiagent reinforcement learning via formalisms based on stochastic games, these cannot cope with a high number of agents due to the combinatorial explosion in the number of joint actions. One possible way to reduce the complexity of the problem is to have agents organized in groups of limited size so that the number of joint actions is reduced. These groups are then coordinated by another agent, a tutor or supervisor. Thus, this paper investigates the task of multiagent reinforcement learning for control of traffic signals in two situations: agents act individually (individual learners) and agents can be “tutored”, meaning that another agent with a broader sight will recommend a joint action.     

Self-organization and multiagent systems: II. Applications and the development technology

Major trends in the development of modern information technologies are to a large extend determined by important practical problems that arise in economics, ecology, safety of society and individuals, and in other fields. Even though these problems seem to be quite different and the requirements for their software implementation are also different, they have many common features, which imply the most stringent requirements for modern information technologies. These features were analyzed in the first part of the present paper. That analysis showed that the new requirements for the model and software implementation of such problems are best met by the multiagent self-organizing system model. In this paper, we consider examples of using this model in various applications and describe their architectures and software implementation; in particular, multiagent self-organization models as applied for flood forecasting and planning and operational enterprise management are described. New capabilities of multiagent self-organizing systems are demonstrated using a self-learning system for detecting intrusions into computer networks as an example. Here, the problem of self-configuration of an overlay network is actually solved. The capabilities of a multiagent self-organizing system in large-scale control in real time are demonstrated using adaptive traffic control in large cities. For the software implementation of multiagent self-organizing systems, special development tools that are different from the existing ones are needed because the conventional top-down development paradigm is inappropriate for self-organizing architectures. The cause is that the global behavior of a multiagent self-organizing system emerges due to local interactions; therefore, it cannot be predicted in advance. For that reason, the bottom-up development model is more appropriate for such systems. In this paper, we give a brief review of the models and approaches proposed for this purpose. One of the promising approaches based on the use of the so-called self-organization design patterns is described in more detail. Results of using the multiagent self-organization model are discussed and prospects of its practical application are estimated.     

Cooperative, hybrid agent architecture for real-time traffic signal control

This paper presents a new hybrid, synergistic approach in applying computational intelligence concepts to implement a cooperative, hierarchical, multiagent system for real-time traffic signal control of a complex traffic network. The large-scale traffic signal control problem is divided into various subproblems, and each subproblem is handled by an intelligent agent with a fuzzy neural decision-making module. The decisions made by lower-level agents are mediated by their respective higher-level agents. Through adopting a cooperative distributed problem solving approach, coordinated control by the agents is achieved. In order for the multiagent architecture to adapt itself continuously to the dynamically changing problem domain, a multistage online learning process for each agent is implemented involving reinforcement learning, learning rate and weight adjustment as well as dynamic update of fuzzy relations using an evolutionary algorithm. The test bed used for this research is a section of the Central Business District of Singapore. The performance of the proposed multiagent architecture is evaluated against the set of signal plans used by the current real-time adaptive traffic control system. The multiagent architecture produces significant improvements in the conditions of the traffic network, reducing the total mean delay by 40% and total vehicle stoppage time by 50%.     

TRAMMAS: A tracing model for multiagent systems

Agent's flexibility and autonomy, as well as their capacity to coordinate and cooperate, are some of the features which make multiagent systems useful to work in dynamic and distributed environments. These key features are directly related to the way in which agents communicate and perceive each other, as well as their environment and surrounding conditions. Traditionally, this has been accomplished by means of message exchange or by using blackboard systems. These traditional methods have the advantages of being easy to implement and well supported by multiagent platforms; however, their main disadvantage is that the amount of social knowledge in the system directly depends on every agent actively informing of what it is doing, thinking, perceiving, etc. There are domains, for example those where social knowledge depends on highly distributed pieces of data provided by many different agents, in which such traditional methods can produce a great deal of overhead, hence reducing the scalability, efficiency and flexibility of the multiagent system. This work proposes the use of event tracing in multiagent systems, as an indirect interaction and coordination mechanism to improve the amount and quality of the information that agents can perceive from both their physical and social environment, in order to fulfill their goals more efficiently. In order to do so, this work presents an abstract model of a tracing system and an architectural design of such model, which can be incorporated to a typical multiagent platform.     

Soccer server: A tool for research on multiagent systems

This article describes Soccer Server, a simulator of the game of soccer designed as a benchmark for evaluating multiagent systems and cooperative algorithms. In real life, successful soccer teams require many qualities, such as basic ball control skills, the ability to carry out strategies, and teamwork. We believe that simulating such behaviors is a significant challenge for computer science, artificial intelligence, and robotics technologies. It is to promote the development of such technologies, and to help define a new standard problem for research, that we have developed Soccer Server. We demonstrate the potential of Soccer Server by reporting an experiment that uses the system to compare the performance of a neural network architecture and a decision tree algorithm at learning the selection of soccer play plans. Other researchers using Soccer Server to investigate the nature of cooperative behavior in a multiagent environment will have the chance to assess their progress at RoboCup-97, an international competition of robotic soccer to be held in conjunction with IJCAI-97. Soccer Server has been chosen as the official server for this contest.     

Research directions for service-oriented multiagent systems

Today's service-oriented systems realize many ideas from the research conducted a decade or so ago in multiagent systems. Because these two fields are so deeply connected, further advances in multiagent systems could feed into tomorrow's successful service-oriented computing approaches. This article describes a 15-year roadmap for service-oriented multiagent system research.     

Preface

Networked sensing and control has attracted significant interest in recent years due to its wide applications. For example, sensor networks, especially wireless sensor networks, have found important applications in environmental monitoring, agriculture, building and industrial automation, machine condition monitoring, intelligent transportation systems, health care, surveillance, and defense. On the other hand, due to the flexibility and significant cost-saving, there has been an increasing trend for control systems to utilize digital communication networks for exchanging information between sensor and controller and/or controller and actuator, as well as between subsystems or systems sharing the same communication networks. Furthermore, networked multiagent systems, where the collective behavior of a system is realized through interactions among dynamically decoupled systems, have also been the focus of many recent research efforts. In networked systems, the network is an important component with constraints and uncertainties, such as the limited bandwidth, random transmission-delays, possible packet-losses, out-of-sequence measurements, etc. The uncertainties and constraints become more significant in wireless networks because of its limited resources of communication and computation, and the fading that changes the throughput of communication channels. As a result, there is an imperative need of new tools for analysis and synthesis, and new algorithms for control, estimation and decision-making, to take these uncertainties and constraints into consideration and handle the interplay among communication, computation and control. In the past few years, a lot of research efforts have been devoted to this field and significant advancement has been made. This special issue intends to reflect part of those achievements. Indeed, the special issue consists of 17 papers among which 8 discuss the control over communication networks, 1 investigates the traffic control in communication networks, 3 are concerned with multiagent systems, and 5 deal with wireless sensor networks. All papers were selected from numerous submissions and carefully reviewed.We hope that the special issue will promote the research interest in networked sensing, actuation and control systems, and foster new applications of networked systems. Special issue editors: Lihua Xie, Nanyang Technological University, Singapore. Frank L. Lewis, The University of Texas at Arlington, USA.     

Motorway traffic flow modelling,estimation and control with vehicle automation and communication systems

Traffic congestion on motorways is a serious threat for the economic and social life of modern society as well as for the environment, which calls for drastic and radical solutions. Conventional traffic management measures, currently applied, are valuable, but face limitations. During the last decades, a variety of vehicle automation and communication systems (VACS) have been developed and deployed, and many more are expected to appear in the near future. These systems provide a novel basis for a new generation of traffic management, which exploits emerging vehicle automation functions and connectivity channels to enable sensible traffic flow improvements in terms of efficiency and safety. A number of innovative concepts, tools and results that open up new horizons for traffic management research and practice in presence of VACS have been produced recently in the frame of TRAMAN21, an ERC Advanced Grant. This paper presents a collection of novel problems as well as of related traffic flow modelling, estimation and control developments for motorway traffic that can be used in the evolving traffic environment with VACS.     

Methods and features of mathematical simulation of air traffic management systems

Air traffic management (ATM) systems and their components??planning the use of air-space and air traffic control systems??are among the most complex modern systems that cannot be introduced without a large amount of preliminary research. Tools for the simulation of the procedures of air traffic management and control used in Europe and the USA are considered. The importance of simulation is caused by the fundamental modernization of the principles and methods of control, which requires anticipatory research. The field of application of the Russian simulation tool for ATM systems called KIM OrVD is determined. This tool is used by the Russian ATM State Corporation. Examples of using this tool for working out proposals concerning the improvements in the Russian ATM system and the evaluation of the efficiency of such proposals are discussed.     

An architectural framework for CKBS applications

This paper presents an architectural framework for cooperating knowledge based systems (CKBSs) with parallels drawn from the multiagent systems of DAI. A CKBS is distinguished from a multiagent system by its need to provide a workable approach for real-world distributed applications. The framework proposed considers only interagent activities in what is called transaction-oriented processing. The framework, based largely on well-tested computer science concepts, provides for a multilayered edifice with information transparency, and a multilevel schema to suit different user expertise. It permits dynamic definition of cooperation strategies for different tasks as required, in a high-level language providing relative ease of use. A particular novelty is the interpretation of actions as side-effects of update operations on action tuples transmitted among agents via what are called shadows. This provides the generality needed. Effectiveness, flexibility, and ease of use are some of the key considerations     

Security policies for sharing knowledge in virtual communities

Knowledge management exploits the new opportunities of sharing knowledge among members of virtual communities in distributed computer networks, and knowledge-management systems are therefore modeled and designed as multiagent systems. In this paper, normative multiagent systems for secure knowledge management based on access-control policies are studied. It is shown how distributed access control is realized by means of local policies of access-control systems for documents of knowledge providers, and by means of global community policies regulating these local policies. Moreover, it is shown how such a virtual community of multiple knowledge providers respects the autonomy of the knowledge providers.     

Multi-objective multiagent credit assignment in reinforcement learning and NSGA-II

Multiagent systems have had a powerful impact on the real world. Many of the systems it studies (air traffic, satellite coordination, rover exploration) are inherently multi-objective, but are often treated as single-objective problems within the research. A key concept within multiagent systems is that of credit assignment: quantifying an individual agent’s impact on the overall system performance. In this work,we extend the concept of credit assignment into multi-objective problems. We apply credit assignment through difference evaluations to two different policy selection paradigms to demonstrate their broad applicability. We first examine reinforcement learning, in which using difference evaluations improves performance by (i) increasing learning speed by up to 10\(\times \), (ii) producing solutions that dominate all solutions discovered by a traditional team-based credit assignment schema and (iii) losing only 0.61 % of dominated hypervolume in a scenario where 20 % of agents act in their own interests instead of the system’s interests (compared to a 43 % loss when using a traditional global reward in the same scenario). We then derive multiple methods for incorporating difference evaluations into a state-of-the-art multi-objective evolutionary algorithm, NSGA-II. Median performance of the NSGA-II considering credit assignment dominates best-case performance of NSGA-II not considering credit assignment in a multiagent multi-objective problem. Our results strongly suggest that in a multiagent multi-objective problem, proper credit assignment is at least as important to performance as the choice of multi-objective algorithm.     

A layered approach to network management control

Public telecommunications networks have many supporting operations systems to provide network management functions and operational support. These systems must be coordinated to provide a highly reliable network service and to make best use of network resources. With increasing automation of network control, these systems must evolve to provide total network support. One organizing principle for this evolution is the definition of control layers at which individual control functions are exercised. We describe a proposal for layered control, with particular emphasis on the role of network traffic management. Layering will be a significant study item for the next extensions to CCITT's Telecommunications Management Network (TMN) recommendations.     

LEARNING AND COORDINATION FOR AUTONOMOUS INTERSECTION CONTROL

Future urban road traffic management is an example of a socially relevant problem that can be modeled as a large-scale, open, distributed system, composed of many autonomous interacting agents, which need to be controlled in a decentralized manner. In this context, advanced, reservation-based, intersection control—where autonomous vehicles controlled entirely by agents interact with a coordination facility that controls an intersection, to avoid collisions and minimize delays—will be a possible scenario in the near future. In this article, we seize the opportunities for multiagent learning offered by such a scenario, studying i) how vehicles, when approaching a reservation-based intersection, can coordinate their actions in order to improve their crossing times, and therefore, speed up the traffic flow through the intersection, and ii) how a set of reservation-based intersections can cooperatively act over an entire network of intersections in order to minimize travel times.     

Three faces of human-computer interaction

Human-computer interaction is considered a core element of computer science. Yet it has not coalesced; many researchers who identify their focus as human-computer interaction reside in other fields. The author examines the origins and evolution of three HCI research foci: computer operation, information systems management, and discretionary use. The author describes efforts to find common ground and forces that have kept them apart.     

The information systems research cycle

What distinguishes information systems from closely aligned disciplines such as computer science, organizational science, management science, economics, or systems engineering? How does IS research balance the demands of relevance and rigor to make effective contributions to both theory and practice? As senior researchers in IS, the authors have engaged in many debates on these questions and have come to some conclusions about what makes this burgeoning field unique and how to properly plan, execute, and evaluate IS research as well as transition it into practice.     

Modeling and predicting mental workload in en route air traffic control: critical review and broader implications

OBJECTIVE: We perform a critical review of research on mental workload in en route air traffic control (ATC). We present a model of operator strategic behavior and workload management through which workload can be predicted within ATC and other complex work systems. BACKGROUND: Air traffic volume is increasing worldwide. If air traffic management organizations are to meet future demand safely, better models of controller workload are needed. METHOD: We present the theoretical model and then review investigations of how effectively traffic factors, airspace factors, and operational constraints predict controller workload. RESULTS: Although task demand has a strong relationship with workload, evidence suggests that the relationship depends on the capacity of the controllers to select priorities, manage their cognitive resources, and regulate their own performance. We review research on strategies employed by controllers to minimize the control activity and information-processing requirements of control tasks. CONCLUSION: Controller workload will not be effectively modeled until controllers' strategies for regulating the cognitive impact of task demand have been modeled. APPLICATION: Actual and potential applications of our conclusions include a reorientation of workload modeling in complex work systems to capture the dynamic and adaptive nature of the operator's work. Models based around workload regulation may be more useful in helping management organizations adapt to future control regimens in complex work systems.