Transfer in variable-reward hierarchical reinforcement learning

Transfer learning seeks to leverage previously learned tasks to achieve faster learning in a new task. In this paper, we consider transfer learning in the context of related but distinct Reinforcement Learning (RL) problems. In particular, our RL problems are derived from Semi-Markov Decision Processes (SMDPs) that share the same transition dynamics but have different reward functions that are linear in a set of reward features. We formally define the transfer learning problem in the context of RL as learning an efficient algorithm to solve any SMDP drawn from a fixed distribution after experiencing a finite number of them. Furthermore, we introduce an online algorithm to solve this problem, Variable-Reward Reinforcement Learning (VRRL), that compactly stores the optimal value functions for several SMDPs, and uses them to optimally initialize the value function for a new SMDP. We generalize our method to a hierarchical RL setting where the different SMDPs share the same task hierarchy. Our experimental results in a simplified real-time strategy domain show that significant transfer learning occurs in both flat and hierarchical settings. Transfer is especially effective in the hierarchical setting where the overall value functions are decomposed into subtask value functions which are more widely amenable to transfer across different SMDPs.     

Learning first-order probabilistic models with combining rules

Many real-world domains exhibit rich relational structure and stochasticity and motivate the development of models that combine predicate logic with probabilities. These models describe probabilistic influences between attributes of objects that are related to each other through known domain relationships. To keep these models succinct, each such influence is considered independent of others, which is called the assumption of “independence of causal influences” (ICI). In this paper, we describe a language that consists of quantified conditional influence statements and captures most relational probabilistic models based on directed graphs. The influences due to different statements are combined using a set of combining rules such as Noisy-OR. We motivate and introduce multi-level combining rules, where the lower level rules combine the influences due to different ground instances of the same statement, and the upper level rules combine the influences due to different statements. We present algorithms and empirical results for parameter learning in the presence of such combining rules. Specifically, we derive and implement algorithms based on gradient descent and expectation maximization for different combining rules and evaluate them on synthetic data and on a real-world task. The results demonstrate that the algorithms are able to learn both the conditional probability distributions of the influence statements and the parameters of the combining rules.     

The first learning track of the international planning competition

The International Planning Competition is a biennial event organized in the context of the International Conference on Automated Planning and Scheduling. The 2008 competition included, for the first time, a learning track for comparing approaches for improving automated planners via learning. In this paper, we describe the structure of the learning track, the planning domains used for evaluation, the participating systems, the results, and our observations. Towards supporting the goal of domain-independent learning, one of the key features of the competition was to disallow any code changes or parameter tweaks after the training domains were revealed to the participants. The competition results show that at this stage no learning for planning system outperforms state-of-the-art planners in a domain independent manner across a wide range of domains. However, they appear to be close to providing such performance. Evaluating learning for planning systems in a blind competition raises important questions concerning criteria that should be taken into account in future competitions.     

Drive level acoustic defectoscopy for head disc interface (HDI) integration and manufacturing

A short product development cycle for modern hard disc drive (HDD) depends entirely on early defect characterization. However, deviations from desired HDI mechanical performance are not always manifested in electromagnetic read back or servo control signals. The best non-destructive tool to monitor HDI dynamics at the HDD level is a passive acoustic technique that tracks equilibrium of the head gimbal assembly and air bearing (AB) modes. This technique is capable of active head protrusion detection as well as detection of embedded particles. The drawback of the passive acoustic monitoring technique at drive level is that a noisy HDD environment makes the detectability of a useful signal challenging. The proposed HDD level in situ passive HDI acoustic monitoring technique consists of external AE sensors driven by adequate electronics. These are enhanced by advanced signal processing routines that include adaptive discrete wavelet transforms. The system is tuned to monitor acoustic HDI signatures during drive spin-up/spin-down cycles as well as during passive and active height across the data zones. In addition, integrated acoustic energy metrics can be used in AB design evaluation stages. In this work examples of HDD level passive acoustic monitoring applications for HDI development, and manufacturing defect tracking, are presented and supported by failure analysis results.     

Quantifying Prior Determination Knowledge Using the PAC Learning Model

Prior knowledge, or bias, regarding a concept can reduce the number of examples needed to learn it. Probably Approximately Correct (PAC) learning is a mathematical model of concept learning that can be used to quantify the reduction in the number of examples due to different forms of bias. Thus far, PAC learning has mostly been used to analyzesyntactic bias, such as limiting concepts to conjunctions of boolean prepositions. This paper demonstrates that PAC learning can also be used to analyzesemantic bias, such as a domain theory about the concept being learned. The key idea is to view the hypothesis space in PAC learning as that consistent withall prior knowledge, syntactic and semantic. In particular, the paper presents an analysis ofdeterminations, a type of relevance knowledge. The results of the analysis reveal crisp distinctions and relations among different determinations, and illustrate the usefulness of an analysis based on the PAC learning model.     

Diffraction characteristics of 10 fs laser pulses passing through an aperture

Results are presented on experimental and theoretical work performed to compare diffraction phenomena for ultrashort 10 fs pulses and continuous-wave propagation modes illuminating different-sized pinholes and slits. Results demonstrate that 10 fs pulses do not produce high-frequency diffraction like that produced with continuous-wave illumination. The diffraction through a 1 mm pinhole of temporally stretched pulses obtained by using fused silica plates whose frequency spectrum remains the same is compared with those of 10 fs pulses. The overall diffraction intensity profiles are, however, nearly identical in this case. The simulations of diffraction patterns for 100 fs, 10 fs, and 1 fs incident pulse were compared theoretically for different aperture sizes and frequencies. Calculations indicate that the lack of high-frequency diffraction for the mode-locked case is due to the broadband nature of the ultrashort laser pulses; i.e., the distribution of the frequency contained in the pulse ends up washing out when objects are illuminated with pulses of broad frequency content. The results of this work have important application in biomedical imaging and remote imaging applications, to name only a few.