When Does Communication Learning Need Hierarchical Multi-Agent Deep Reinforcement Learning

Abstract

Multi-agent systems need to communicate to coordinate a shared task. We show that a recurrent neural network (RNN) can learn a communication protocol for coordination, even if the actions to coordinate are performed steps after the communication phase. We show that a separation of tasks with different temporal scale is necessary for successful learning. We contribute a hierarchical deep reinforcement learning model for multi-agent systems that separates the communication and coordination task from the action picking through a hierarchical policy. We further on show, that a separation of concerns in communication is beneficial but not necessary. As a testbed, we propose the Dungeon Lever Game and we extend the Differentiable Inter-Agent Learning (DIAL) framework. We present and compare results from different model variations on the Dungeon Lever Game.     

A comprehensive solution for application‐level adaptation

Driven by the emergence of mobile and pervasive computing there is a growing demand for context‐aware software systems that can dynamically adapt to their run‐time environment. We present the results of project MADAM that has delivered a comprehensive solution for the development and operation of context‐aware, self‐adaptive applications. The main contributions of MADAM are (a) a sophisticated middleware that supports the dynamic adaptation of component‐based applications, and (b) an innovative model‐driven development methodology that is based on abstract adaptation models and corresponding model‐to‐code transformations. MADAM has demonstrated the viability of a general, integrated approach to application‐level adaptation. We discuss our experiences with two real‐world case studies that were built using the MADAM approach. Copyright © 2008 John Wiley & Sons, Ltd.     

Playing MUSIC — building context‐aware and self‐adaptive mobile applications

Although the idea of context‐awareness was introduced almost two decades ago, few mobile software applications are available today that can sense and adapt to their run‐time environment. The development of context‐aware and self‐adaptive applications is complex and few developers have experience in this area. On the basis of several demonstrators built by the joint European research project MUSIC, this paper describes typical context and adaptation features relevant for the development of context‐aware and self‐adaptive mobile applications. We explain how the demonstrators were realised using the open‐source platform MUSIC and present the feedback of the developers of these demonstrators. The main contribution of this paper is to show how the development complexity of context‐aware and self‐adaptive mobile applications can be mastered by using an adaptation framework such as MUSIC. Copyright © 2012 John Wiley & Sons, Ltd.     

A modelling language for cooperative plans in highly dynamic domains

Cooperative behaviour is one of the challenges most pronounced in the RoboCup Middle Size League. Especially the dynamic nature of the domain, which calls for swift adaptation by each robot and the team as a whole, is a distinctive property of the league. The ability to establish highly responsive teamwork while facing unreliable communication and sensory noise is a key to successful soccer teams. Moreover, modelling such responsive, cooperative behaviour is difficult. In this work, we specify a novel model for cooperative behaviour geared towards highly dynamic domains, focussing on the language syntax and semantics. In our approach, agents estimate each other’s decision and correct these estimations once they receive contradictory information. We provide a comprehensive approach for agent teamwork featuring intuitive modelling capabilities for multi-agent activities, abstractions over activities and agents, and a clear operational semantics. Moreover, we briefly present a graphical modeling tool for cooperative strategies, which is based directly on the theory laid out, together with a practical framework for executing said strategies. We show experimentally the responsiveness and coherence of the resulting team play.     

Model metamorphosis

Model driven architectures (MDA) will become practical only if transformations from platform-independent to platform-specific models can be largely automated. Kafka, an extensible rule-based transformation language, and the Kase modeling tool enable automated, customized transformations that support round-trip engineering. With Kafka and Kase, developers can construct transformations without detailed knowledge of metamodels.     

Middleware challenges ahead

Middleware research and development has reached the end of its first major phase, and new requirements are arising that are so fundamentally different that they will lead to new-generation middleware systems. Facing dynamic modifications in distributed systems technology, middleware developers are striving to support applications that meet the technical challenges of ubiquitous computing     

A Generic Model for Fault Isolation in Integrated Management Systems

Distributed systems in enterprises as well astelecommunication environments strongly demand moreautomated fault management. A single fault in thesecomplex systems might cause a huge number of symptomatic error messages and side effects to occur. Thecommon root faults for these symptoms have to beidentified to start fault removal procedures as soon aspossible and to decrease system down-time. This paper presents a methodology for fault isolation inintegrated management systems. A generic model isdescribed that unifies the view of the management systemon the managed environment. It integrates the relevant aspects of network, system, and servicemanagement layers in order to perform integrated faultisolation. Our approach is based on a general dependencygraph model. It captures the information that isrequired to determine the root cause of a fault on theone hand, and the set of fault affected services andcustomers on the other hand. The layered TMNarchitecture serves as an example for an integratedmanagement environment throughout this paper.     

Architectural Constraints in the Model-Driven Development of Self-Adaptive Applications

In component-based software development, a composition of connected components represents a software system's architecture. This component framework supports adaptation through application variability. The adaptation decision is made at runtime by resolving the variation points and computing the utility of all application variants. Following the application variability model, a huge number of application variants may arise, many of which are not feasible in reality. Architectural constraints may be applied to filter out such infeasible variants before checking their utilities. This article presents an approach to specify architectural constraints while following a model-driven development approach.     

Melatonin as a Signaling Molecule for Metabolism Regulation in Response to Hypoxia in the Crab Neohelice granulata

Melatonin has been identified in a variety of crustacean species, but its function is not as well understood as in vertebrates. The present study investigates whether melatonin has an effect on crustacean hyperglycemic hormone (CHH) gene expression, oxygen consumption (VO₂) and circulating glucose and lactate levels, in response to different dissolved-oxygen concentrations, in the crab Neohelice granulata, as well as whether these possible effects are eyestalk- or receptor-dependent. Melatonin decreased CHH expression in crabs exposed for 45 min to 6 (2, 200 or 20,000 pmol·crab⁻¹) or 2 mgO₂·L⁻¹ (200 pmol·crab⁻¹). Since luzindole (200 nmol·crab⁻¹) did not significantly (p > 0.05) alter the melatonin effect, its action does not seem to be mediated by vertebrate-typical MT1 and MT2 receptors. Melatonin (200 pmol·crab⁻¹) increased the levels of glucose and lactate in crabs exposed to 6 mgO₂·L⁻¹, and luzindole (200 nmol·crab⁻¹) decreased this effect, indicating that melatonin receptors are involved in hyperglycemia and lactemia. Melatonin showed no effect on VO₂. Interestingly, in vitro incubation of eyestalk ganglia for 45 min at 0.7 mgO₂·L⁻¹ significantly (p < 0.05) increased melatonin production in this organ. In addition, injections of melatonin significantly increased the levels of circulating melatonin in crabs exposed for 45 min to 6 (200 or 20,000 pmol·crab⁻¹), 2 (200 and 20,000 pmol·crab⁻¹) and 0.7 (200 or 20,000 pmol·crab⁻¹) mgO₂·L⁻¹. Therefore, melatonin seems to have an effect on the metabolism of N. granulata. This molecule inhibited the gene expression of CHH and caused an eyestalk- and receptor-dependent hyperglycemia, which suggests that melatonin may have a signaling role in metabolic regulation in this crab.     

Selbst-adaptive Software

Zusammenfassung Der Trend zum ,,Mobile“ und ,,Pervasive Computing“ führt zu einem wachsenden Bedarf an Softwaresystemen, die in der Lage sind, sich selbst an ihre sich dynamisch ver?ndernde Ablaufumgebung anzupassen. Dynamische Adaption findet zur Laufzeit der Anwendung aufgrund von ?nderungen des Kontexts oder Ressourcenzustands statt. Zum Beispiel k?nnte eine Anwendung sich an niedrigere übertragungsbandbreite, verminderte Batteriekapazit?t, neu hinzugekommene Ger?te und Dienste oder ver?nderte Benutzerpr?ferenzen anpassen. In diesem Beitrag geben wir einen überblick über die Technik selbst-adaptiver Softwaresysteme. Wir diskutieren zun?chst die verschiedenartigen Anforderungen an adaptive Software und stellen dann entsprechende Infrastrukturkonzepte vor, die für die Entwicklung adaptiver Systeme erforderlich sind. Auch Anforderungen an eine Entwicklungsmethodik für adaptive Anwendungen werden erl?utert. Als Beispiel für eine umfassende und allgemeingültige L?sung pr?sentieren wir die wichtigsten Ergebnisse des europ?ischen Forschungsprojektes MADAM (Mobility and ADaptation enAbling Middleware) und vergleichen diese mit verwandten Arbeiten.