When Does Communication Learning Need Hierarchical Multi-Agent Deep Reinforcement Learning

Abstract

Multi-agent systems need to communicate to coordinate a shared task. We show that a recurrent neural network (RNN) can learn a communication protocol for coordination, even if the actions to coordinate are performed steps after the communication phase. We show that a separation of tasks with different temporal scale is necessary for successful learning. We contribute a hierarchical deep reinforcement learning model for multi-agent systems that separates the communication and coordination task from the action picking through a hierarchical policy. We further on show, that a separation of concerns in communication is beneficial but not necessary. As a testbed, we propose the Dungeon Lever Game and we extend the Differentiable Inter-Agent Learning (DIAL) framework. We present and compare results from different model variations on the Dungeon Lever Game.     

Malic acid production from renewables: a review

Malic acid derived from fossil resources is currently applied in the food and beverage industries with a medium global production capacity. However, in the transition from a fossil-based to a bio-based economy, biotechnologically produced l -malic acid may become an important platform chemical with many new applications, especially in the field of biopolymers. In this review, currently used petrochemical production routes to dl -malic acid are outlined and insights into possible bio-based alternatives for microbial l -malic acid production are provided. Besides ecological reasons, the possibility to produce enantiopure l -malic acid by microbial fermentation is the biggest advantage over chemical synthesis. State-of-the-art and open challenges concerning production host engineering, substrate choice and downstream processing are addressed. With regard to production hosts, a literature overview is given covering the leading natural production strains of Aspergillus, Ustilago and Aureobasidium, as well as Escherichia coli as the most important engineered recombinant host. The utilization of renewable substrates as an alternative to glucose is emphasized in particular as a key aspect for a competitive bio-based production. Out of the alternative substrates discussed in this review, the industrial side-streams crude glycerol and molasses seem to be most promising for large-scale l -malic acid production. © 2019 The Authors. Journal of Chemical Technology & Biotechnology published by John Wiley & Sons Ltd on behalf of Society of Chemical Industry.     

Electronic Properties of the Layered Perovskite Ruthenates: Correlated Electron Physics Approaching the Low-Disorder Limit

We present a brief review of the physics that is now accessible in the layered perovskite ruthenates that have been the focus of our research for the past few years. The main theme of the work has been the discovery of novel ground states that have an extreme sensitivity to disorder. We discuss these developments and the lessons that they provide for the wider field of correlated electrons in solids.     

Ormosil coatings of high hardness

Organically modified silicates (ormosils) of two systems were coated on polyethyleneterephthalate (PET) substrates. One was the tetraethoxysilane (TEOS)-vinyltriethoxysilane (VTES) system and the other was the TEOS-tetraisopropyltitanate (TIPT)-VTES system. The reactions among the alkoxides were examined by liquid state ²⁹Si nuclear magnetic resonance (NMR) spectroscopy. The chemical bonds between TEOS and VTES, between TEOS and TIPT, and between VTES and TIPT are shown in the spectra and the reaction schemes are proposed. Vickers hardnesses of the ormosil coatings were obtained by using the models developed by Jönsson and Hogmark, and the theoretical models developed by the authors were valuable to predict the hardness values. The ormosil coatings obtained in the present study were much harder than the hardest transparent plastics, and thus are very useful for hard or protective coatings on organic polymers.     

Time dependent behavior of elements combining ultra-high performance fiber reinforced concretes (UHPFRC) and reinforced concrete

Structural elements combining Ultra-High Performance Fiber Reinforced Concretes (UHPFRC) and concrete offer a high potential in view of rehabilitation and modification of existing structures. The investigation of the time-dependent behavior of composite “UHPFRC-concrete” elements is a fundamental step in the determination of durability and serviceability. For this, an experimental program was conducted on large composite “UHPFRC-concrete” beams and a numerical model was validated with the test results. The experimental results and a parametric study performed with the numerical model showed that UHPFRC and normal strength reinforced concrete are compatible in the long-term and that the critical period of composite “UHPFRC-concrete” elements are the first 90 days after the casting of the UHPFRC layer. Thus, the high potential of such composite elements can be exploited also in the long term.     

Detection rates for abnormal cervical smears: what are we screening for?

We analysed rates of detection for smear abnormalities in 255,000 women served by the Bristol screening programme. The programme began in 1966 with the aim of eradicating the 30-40 deaths each year in Bristol from cervical cancer. Organisation has been good and population uptake has been high for the past 15 years. Records were computerised in 1977. During the 1988 to 1993 screening round, 225,974 women were tested. New smear abnormalities were found in 15,551, of whom nearly 6000 were referred for colposcopy. These numbers are excessively high in comparison with the incidence of the malignancy we are trying to prevent. The effect of screening on death rates in Bristol is too small to detect. Our conclusion is that despite good organisation of the service, much of our effort in Bristol is devoted to limiting the harm done to healthy women and to protecting our staff from litigation as cases of serious disease continue to occur. The real lesson from 30 years' cervical screening is that no matter how obvious the predicted benefit may seem for any screening test, introduction should never take place without adequate prior evaluation of both positive and negative effects in controlled trials.     

Workbench: an AI approach to materials modelling

The Workbench is an adaptable and efficient tool for performing thermodynamic and kinetic calculations. This computer program uses various artificial intelligence techniques to provide a more versatile modelling system than has been possible with conventional programs. The structure and operation of the program is described and a simplified example is worked through illustrating the Workbench environment. Practical applications are given of the use of the Workbench to predict magnetic microstructures in Nd-Fe-B permanent magnets and to study the long term structural stability of Co-Pd multilayers.     

Synthesis and Pharmacological In Vitro Investigations of Novel Shikonin Derivatives with a Special Focus on Cyclopropane Bearing Derivatives

Melanoma is the deadliest form of skin cancer and accounts for about three quarters of all skin cancer deaths. Especially at an advanced stage, its treatment is challenging, and survival rates are very low. In previous studies, we showed that the constituents of the roots of Onosma paniculata as well as a synthetic derivative of the most active constituent showed promising results in metastatic melanoma cell lines. In the current study, we address the question whether we can generate further derivatives with optimized activity by synthesis. Therefore, we prepared 31, mainly novel shikonin derivatives and screened them in different melanoma cell lines (WM9, WM164, and MUG-Mel2 cells) using the XTT viability assay. We identified (R)-1-(1,4-dihydro-5,8-dihydroxy-1,4-dioxonaphthalen-2-yl)-4-methylpent-3-enyl 2-cyclopropyl-2-oxoacetate as a novel derivative with even higher activity. Furthermore, pharmacological investigations including the ApoToxGlo^TM Triplex assay, LDH assay, and cell cycle measurements revealed that this compound induced apoptosis and reduced cells in the G1 phase accompanied by an increase of cells in the G2/M phase. Moreover, it showed hardly any effects on the cell membrane integrity. However, it also exhibited cytotoxicity against non-tumorigenic cells. Nevertheless, in summary, we could show that shikonin derivatives might be promising drug leads in the treatment of melanoma.     

Characterization of Apo-Form Selective Inhibition of Indoleamine 2,3-Dioxygenase**

Indoleamine-2,3-dioxygenase 1 (IDO1) is a heme-containing enzyme that catalyzes the rate-limiting step in the kynurenine pathway of tryptophan (TRP) metabolism. As it is an inflammation-induced immunoregulatory enzyme, pharmacological inhibition of IDO1 activity is currently being pursued as a potential therapeutic tool for the treatment of cancer and other disease states. As such, a detailed understanding of the mechanism of action of IDO1 inhibitors with various mechanisms of inhibition is of great interest. Comparison of an apo-form-binding IDO1 inhibitor (GSK5628) to the heme-coordinating compound, epacadostat (Incyte), allows us to explore the details of the apo-binding inhibition of IDO1. Herein, we demonstrate that GSK5628 inhibits IDO1 by competing with heme for binding to a heme-free conformation of the enzyme (apo-IDO1), whereas epacadostat coordinates its binding with the iron atom of the IDO1 heme cofactor. Comparison of these two compounds in cellular systems reveals a long-lasting inhibitory effect of GSK5628, previously undescribed for other known IDO1 inhibitors. Detailed characterization of this apo-binding mechanism for IDO1 inhibition might help design superior inhibitors or could confer a unique competitive advantage over other IDO1 inhibitors vis-à-vis specificity and pharmacokinetic parameters.