Recent Advances in Hierarchical Reinforcement Learning

Reinforcement learning is bedeviled by the curse of dimensionality: the number of parameters to be learned grows exponentially with the size of any compact encoding of a state. Recent attempts to combat the curse of dimensionality have turned to principled ways of exploiting temporal abstraction, where decisions are not required at each step, but rather invoke the execution of temporally-extended activities which follow their own policies until termination. This leads naturally to hierarchical control architectures and associated learning algorithms. We review several approaches to temporal abstraction and hierarchical organization that machine learning researchers have recently developed. Common to these approaches is a reliance on the theory of semi-Markov decision processes, which we emphasize in our review. We then discuss extensions of these ideas to concurrent activities, multiagent coordination, and hierarchical memory for addressing partial observability. Concluding remarks address open challenges facing the further development of reinforcement learning in a hierarchical setting.     

Multiplicative Operator Splittings in Nonlinear Diffusion: From Spatial Splitting to Multiple Timesteps

Operator splitting is a powerful concept used in many diversed fields of applied mathematics for the design of effective numerical schemes. Following the success of the additive operator splitting (AOS) in performing an efficient nonlinear diffusion filtering on digital images, we analyze the possibility of using multiplicative operator splittings to process images from different perspectives.We start by examining the potential of using fractional step methods to design a multiplicative operator splitting as an alternative to AOS schemes. By means of a Strang splitting, we attempt to use numerical schemes that are known to be more accurate in linear diffusion processes and apply them on images. Initially we implement the Crank-Nicolson and DuFort-Frankel schemes to diffuse noisy signals in one dimension and devise a simple extrapolation that enables the Crank-Nicolson to be used with high accuracy on these signals. We then combine the Crank-Nicolson in 1D with various multiplicative operator splittings to process images. Based on these ideas we obtain some interesting results. However, from the practical standpoint, due to the computational expenses associated with these schemes and the questionable benefits in applying them to perform nonlinear diffusion filtering when using long timesteps, we conclude that AOS schemes are simple and efficient compared to these alternatives.We then examine the potential utility of using multiple timestep methods combined with AOS schemes, as means to expedite the diffusion process. These methods were developed for molecular dynamics applications and are used efficiently in biomolecular simulations. The idea is to split the forces exerted on atoms into different classes according to their behavior in time, and assign longer timesteps to nonlocal, slowly-varying forces such as the Coulomb and van der Waals interactions, whereas the local forces like bond and angle are treated with smaller timesteps. Multiple timestep integrators can be derived from the Trotter factorization, a decomposition that bears a strong resemblance to a Strang splitting. Both formulations decompose the time propagator into trilateral products to construct multiplicative operator splittings which are second order in time, with the possibility of extending the factorization to higher order expansions. While a Strang splitting is a decomposition across spatial dimensions, where each dimension is subsequently treated with a fractional step, the multiple timestep method is a decomposition across scales. Thus, multiple timestep methods are a realization of the multiplicative operator splitting idea. For certain nonlinear diffusion coefficients with favorable properties, we show that a simple multiple timestep method can improve the diffusion process.     

Longitudinal analysis of the relationship between symptoms and quality of life in veterans treated for posttraumatic stress disorder.

This study examined how change in posttraumatic stress disorder (PTSD) symptoms relates to change in quality of life. The sample consisted of 325 male Vietnam veterans with chronic PTSD who participated in a randomized trial of group psychotherapy. Latent growth modeling was used to test for synchronous effects of PTSD symptom change on psychosocial and physical health-related quality of life within the same time period and lagged effects of initial PTSD symptom change on later change in quality of life. PTSD symptoms were associated with reduced quality of life before treatment. There were synchronous effects of symptom change on change in quality of life but no significant lagged effects. Results indicate the importance of measuring quality of life in future investigations of PTSD treatment. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Failures of retrieval and comparison constrain change detection in natural scenes.

In a change detection paradigm, a target object in a natural scene either rotated in depth, was replaced by another object token, or remained the same. Change detection performance was reliably higher when a target postcue allowed participants to restrict retrieval and comparison processes to the target object (Experiment 1). Change detection performance remained excellent when the target object was not attended at change (Experiment 2) and when a concurrent verbal working memory load minimized the possibility of verbal encoding. (Experiment 3). Together, these data demonstrate that visual representations accumulate in memory from attended objects as the eyes and attention are oriented within a scene and that change blindness derives, at least in part, from retrieval and comparison failure. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The role of environmental context in mapping invasive plants with hyperspectral image data

Lepidium latifolium (perennial pepperweed) is a noxious Eurasian weed invading riparian and wetland areas of the western US. Effective management of Lepidium requires detailed, accurate maps of its distribution, as may be provided by remote sensing, to contain existing infestations and eradicate incipient populations. We mapped Lepidium with 3 m spatial resolution, 128-band HyMap image data in three sites of California's San Francisco Bay/Sacramento–San Joaquin Delta Estuary (Rush Ranch in Suisun Marsh and the Greater Jepson Prairie Ecosystem and the Cosumnes River Preserve in the Delta). These sites are markedly different in terms of hydrology, salinity, species composition, and structural and landscape diversity. Aggregated classification and regression tree models (CART), incorporating the results of mixture tuned matched filter (MTMF) analyses and spectral physiological indexes, were used to map Lepidium at the three sites. This approach was sufficiently flexible and robust to detect Lepidium with similar accuracies (~ 90%) at both Rush Ranch and Jepson Prairie, but was unsuccessful at Cosumnes River Preserve. Comparisons of the behavior of the MTMFs and the CARTs between sites reveal the importance of environmental context in species mapping. Rush Ranch presents the simplest conditions for mapping Lepidium: it is the wettest and least diverse site and Lepidium is spectrally distinct from co-occurring species. At Jepson Prairie, several co-occurring species closely resemble Lepidium spectrally. Nevertheless, hyperspectral data provide sufficient spectral detail to resolve Lepidium even at this challenging site, which is facilitated by phenological separation from the matrix of annual grasses. At Cosumnes River Preserve, however, Lepidium is neither spectrally nor phenologically distinct, and consequently could not be mapped successfully. Evidence suggests that the success of a remote sensing analysis declines as site complexity increases (species, structural, and landscape diversity; spectral variability; etc.), although this relationship is complex, indirect, and may be phenology-dependent.     

Gene amplification as a prognostic factor in primary and secondary high-grade malignant gliomas

ND10 are recently characterized nuclear domains that are composed of 0.5 microm sized, precisely circumscribed dots in cultured human cell lines. To investigate the distribution and number of ND10 on various types of normal and neoplastic human tissues, we carried out immunostaining and immunoprecipitation analyses with monoclonal antibodies 138 and 1150. The number of ND10 varied from 1 to 10 or more in various tissues as did their size. ND10 were diffusely located in early embryonic and normal tissues, except for the exocrine and endocrine cells of the pancreas and for hepatocytes. In normal squamous mucosa, basal cells had more ND10 than did differentiated superficial squamous cells. The number and size of ND10 were markedly increased in malignant neoplasms but were similar in benign tumors and corresponding normal tissues. Sex hormone-related normal tissues, such as the endometrium or myometrium, and neoplasms strongly stained for ND10. The distribution pattern of ND10 in human tissues indicates that they are conserved nuclear substructures that are closely associated with cellular differentiation, hormonal stimulation, and oncogenesis.     

Paraneoplastic encephalomyelitis and seminoma: importance of testicular ultrasonography

We report two patients with paraneoplastic limbic and brainstem encephalitis associated with occult nonmetastatic testicular seminoma. In each patient, the neoplasm was detectable only by testicular ultrasonography. Male patients with this syndrome in whom lung cancer is not found should undergo testicular ultrasonography as part of the search for an extrapulmonary neoplasm. A normal clinical testicular examination is insufficient to exclude an occult seminoma.     

Representation and selection of relative position.

Part 1 concerns representation: Pop-out for a target that can be distinguished from the distractors only by the relative position of its components was demonstrated, thereby showing that simple shape information is represented preattentively. Findings are discussed in terms of early- vs late-selection debate and in terms of theories of search and texture segregation. Part 2 concerns selection: Strong learning in a varied-mapping paradigm was demonstrated, and it was shown that preattentive shape information can be used for selection. Finally, an account of the learning, namely the group scale hypothesis, is suggested and a final experiment to test it is presented. Results confirm and extend the emphasis placed by J. Duncan and G. W. Humphreys's (1989) attentional engagement theory on grouping processes in visual search. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Benzylation of thiophene using zinc and iron chloride modified montmorillinite clay

Montmorillinite clays, modified with either Zn(II) or Fe(III) chlorides have been used to catalyze the Friedel-Crafts alkylation of thiophene. High yields (ca. 80%) of 2- and 3-benzylthiophene mixtures were obtained using either catalyst in chlorobenzene or nitrobenzene solvent at 80°C. The ratio of 2–3-products was found to be controlled by their relative stability. Reactions were most rapid in nitrobenzene solution (<5 min) and resulted in formation of the 2-isomer as the only mono-substituted product. This is believed to be the first example in which an alkylation of thiophene has been limited to one regio-isomer.