EXPERIMENTS WITH ONLINE REINFORCEMENT LEARNING IN REAL-TIME STRATEGY GAMES

Real-time strategy (RTS) games provide a challenging platform to implement online reinforcement learning (RL) techniques in a real application. Computer, as one game player, monitors opponents’ (human or other computers) strategies and then updates its own policy using RL methods. In this article, we first examine the suitability of applying the online RL in various computer games. Reinforcement learning application depends on both RL complexity and the game features. We then propose a multi-layer framework for implementing online RL in an RTS game. The framework significantly reduces RL computational complexity by decomposing the state space in a hierarchical manner. We implement an RTS game—Tank General—and perform a thorough test on the proposed framework. We consider three typical profiles of RTS game players and compare two basic RL techniques applied in the game. The results show the effectiveness of our proposed framework and shed light on relevant issues in using online RL in RTS games.     

Prognostic values of echocardiography compared with clinical variables in suspected heart disease. Multivariate analysis of long-term prognosprognosis in 456 patients

The aim of the study was to evaluate the prognostic significance of clinical and echocardiographic data in patients referred for echocardiography in a retrospective analysis. Four hundred and fifty-six patients from a district hospital were studied. Survival after three years was 64%. Multivariate analysis identified five factors with independent prognostic information (relative risks of death are shown in brackets): left ventricular wall motion index (WMI) < or = 1.2 by echocardiography (2.5), status as in-patient (2.1), age > 65 years (1.7), clinical heart failure (1.9) and atrial fibrillation (1.5). When information on age, hospitalisation status, heart failure and heart rhythm had already been entered in the Cox model, echocardiographic results such as decreased WMI and dilated right ventricle still gave further prognostic information. We conclude that among conventional clinical and echocardiographic data WMI was the strongest predictor of long-term survival, and, despite prior knowledge of major clinical features, echocardiography provided further prognostic information.     

Diagnosing renal artery stenosis: a comparison between conventional renography, captopril renography and ultrasound Doppler in a large consecutive series of patients with arterial hypertension

PROBLEM: Comparison and characterisation of different lymphocyte subsets in the endometrium of endometriosis patients and in healthy women on every day of the menstrual cycle with special emphasis on the CD4:CD8 ratio in the endometrium. METHOD: Immunohistochemical staining of 253 endometrial biopsies of infertile women with and without endometriosis with Anti-Leu4 (CD3), Anti-Leu3a (CD4), Anti-Leu2a (CD8), Anti-Leu7 and Anti-Human-B-cell (CD22) using the immune peroxidase reaction. Identification and counting of positive lymphocyte were performed on cryostat sections. RESULTS: Endometrial lymphocyte subsets show equal quantity and distribution in endometriosis patients and in the control group. After a peak in the early proliferative phase the absolute number of T lymphocytes decreases while a predominance of T-suppressor/cytotoxic T lymphocytes (CD8) compared to T-helper/inducer lymphocytes (CD4) occurs towards the end of the menstrual cycle. CONCLUSION: Endometrium as the potential parent epithelia of endometriosis lesions seems not to be altered in its lymphatic cell content compared to healthy women. Furthermore, endometrium is clearly characterised as part of the mucosa associated lymphatic tissue (MALT). T lymphocytes show specific quantitative changes due to different phases of the menstrual cycle.     

Chronic thromboembolic pulmonary hypertension

Acute pulmonary hypertension has a high mortality at the onset. Patients surviving the first phase will usually recanalize the pulmonary arteries through intrinsic thrombolytic mechanisms and medical treatment. However, in some cases there is insufficient resolution of the emboli with subsequent thrombotic and fibrotic reorganization, leading to a worsening of the pulmonary obstruction. In the open pulmonary arteries the disease may lead to hypertrophy of the media and intimal proliferation, thus leading to a further increase in the pulmonary vascular resistance. This again leads to hypertrophy of the right ventricle and ultimately to right-sided heart failure. Untreated, chronic thromboembolic pulmonary hypertension has a five-year mortality approaching 100%, but extensive pulmonary thrombendarterectomy using extracorporeal circulation and deep hypothermia has been shown to lower the pulmonary vascular resistance and thereby improve the prognosis significantly. Operative treatment can now be offered in Denmark, and the purpose of this review is to draw attention to the disease, its symptoms, diagnosis and the surgical treatment.     

Direct aerobic oxidation of primary alcohols to methyl esters catalyzed by a heterogeneous gold catalyst

Methyl esters can be produced in high yields by oxiding methanolic solutions of primary alcohols with dioxygen over a heterogeneous gold catalyst. The versatility of this new methodology is demonstrated by the fact that alkylic, benzylic and allylic alcohols, as well as alcohols containing an amine functionality are oxidized in good to excellent yields.     

Placement of two- and three-dimensional irregular shapes for inertia moment and balance

We present a heuristic for the problem of placing irregular shapes in two or three dimensions within a container, such that the placement of the shapes is optimized for balance and inertia moment and no two shapes overlap. The heuristic is based on a technique that iteratively removes overlap, which has previously proven successful for bin-packing problems with rectangular objects and strip-packing problems with irregular shapes. We extend this method and demonstrate its ability to optimize an objective function related to the individual position of each shape. The approach iteratively reduces an augmented objective function, which is the sum of balance, inertia moment and overlap and uses the metaheuristic Guided Local Search.     

Inhibition of larval development of the marine copepod Acartia tonsa by four synthetic musk substances

A nitro musk (musk ketone) and three polycyclic musks (Tonalide, Galaxolide and Celestolide) were tested for acute and subchronic effects on a marine crustacean, the calanoid copepod Acartia tonsa. Sublethal effects on A. tonsa larvae were investigated with a rapid and cost effective bioassay, which is based on the easily detectable morphological change from the last nauplius to the first copepodite stage during copepod larval development. The inhibition of larval development after 5 days exposure was a very sensitive endpoint, with 5-d-EC(50)-values as low as 0.026 mg/l (Tonalide), 0.059 mg/l (Galaxolide), 0.066 mg/l (musk ketone) and 0.160 mg/l (Celestolide), respectively. These values were generally more than one order of magnitude below the 48-h-LC(50)-values found for adults, which were 0.47 mg/l (Galaxolide), 0.71 mg/l (Celestolide), 1.32 mg/l (musk ketone) and 2.5 mg/l (Tonalide). Since the synthetic musks strongly inhibited larval development in A. tonsa at low nominal concentrations, they should be considered as very toxic. The larval development test with A. tonsa is able to provide important aquatic toxicity data for the evaluation of synthetic musks, for which there is little published ecotoxicological information available regarding Crustacea. It is suggested that subchronic and chronic copepod toxicity tests should be used more frequently for risk assessment of environmental pollutants.     

Book reviews

Telecommunications and the City: Electronic Spaces, Urban Places. S. Graham & S. Marvin. London and New York, Routledge, 1996, 434 pp., £15.99 pbk. ISBN: 0 415 11903 0.

Housing Policy in Scotland. T. Begg. John Donald, Edinburgh, 1996, £16 pbk. ISBN: 0 85976 433 8.

Developments in Housing Management and Ownership. Ricardo Pinto (Ed.). Manchester University Press, 1995, 267 pp., £45.00 hbk. ISBN: 0 7190 3713 1.

A New Social Atlas of Britain. Daniel Dorling. Chichester, John Wiley & Sons, 1995, 247pp., £35.00 hbk. ISBN: 0 4719 4868 3.