On the Convergence of Temporal-Difference Learning with Linear Function Approximation

The asymptotic properties of temporal-difference learning algorithms with linear function approximation are analyzed in this paper. The analysis is carried out in the context of the approximation of a discounted cost-to-go function associated with an uncontrolled Markov chain with an uncountable finite-dimensional state-space. Under mild conditions, the almost sure convergence of temporal-difference learning algorithms with linear function approximation is established and an upper bound for their asymptotic approximation error is determined. The obtained results are a generalization and extension of the existing results related to the asymptotic behavior of temporal-difference learning. Moreover, they cover cases to which the existing results cannot be applied, while the adopted assumptions seem to be the weakest possible under which the almost sure convergence of temporal-difference learning algorithms is still possible to be demonstrated.     

<Emphasis Type="Italic">hp</Emphasis>-Version <Emphasis Type="Italic">a priori</Emphasis> Error Analysis of Interior Penalty Discontinuous Galerkin Finite Element Approximations to the Biharmonic Equation

We consider the symmetric formulation of the interior penalty discontinuous Galerkin finite element method for the numerical solution of the biharmonic equation with Dirichlet boundary conditions in a bounded polyhedral domain in . For a shape-regular family of meshes consisting of parallelepipeds, we derive hp-version a priori bounds on the global error measured in the L² norm and in broken Sobolev norms. Using these, we obtain hp-version bounds on the error in linear functionals of the solution. The bounds are optimal with respect to the mesh size h and suboptimal with respect to the degree of the piecewise polynomial approximation p. The theoretical results are confirmed by numerical experiments, and some practical applications in Poisson–Kirchhoff thin plate theory are presented.     

Robust regression for high throughput drug screening

Effective analysis of high throughput screening (HTS) data requires automation of dose-response curve fitting for large numbers of datasets. Datasets with outliers are not handled well by standard non-linear least squares methods, and manual outlier removal after visual inspection is tedious and potentially biased. We propose robust non-linear regression via M-estimation as a statistical technique for automated implementation. The approach of finding M-estimates by Iteratively Reweighted Least Squares (IRLS) and the resulting optimization problem are described. Initial parameter estimates for iterative methods are important, so self-starting methods for our model are presented. We outline the software implementation, done in Matlab and deployed as an Excel application via the Matlab Excel Builder Toolkit. Results of M-estimation are compared with least squares estimates before and after manual editing.     

Kernel least-squares models using updates of the pseudoinverse

Sparse nonlinear classification and regression models in reproducing kernel Hilbert spaces (RKHSs) are considered. The use of Mercer kernels and the square loss function gives rise to an overdetermined linear least-squares problem in the corresponding RKHS. When we apply a greedy forward selection scheme, the least-squares problem may be solved by an order-recursive update of the pseudoinverse in each iteration step. The computational time is linear with respect to the number of the selected training samples.     

Interactive visual analysis of families of function graphs

The analysis and exploration of multidimensional and multivariate data is still one of the most challenging areas in the field of visualization. In this paper, we describe an approach to visual analysis of an especially challenging set of problems that exhibit a complex internal data structure. We describe the interactive visual exploration and analysis of data that includes several (usually large) families of function graphs f_i(x, t). We describe analysis procedures and practical aspects of the interactive visual analysis specific to this type of data (with emphasis on the function graph characteristic of the data). We adopted the well-proven approach of multiple, linked views with advanced interactive brushing to assess the data. Standard views such as histograms, scatterplots, and parallel coordinates are used to jointly visualize data. We support iterative visual analysis by providing means to create complex, composite brushes that span multiple views and that are constructed using different combination schemes. We demonstrate that engineering applications represent a challenging but very applicable area for visual analytics. As a case study, we describe the optimization of a fuel injection system in diesel engines of passenger cars     

Estimation and setting starting values in ARMA algorithms

For the given observations set of the ARMA (autoregressive moving average) process, the likelihood function depends, not only on model parameters, but on the starting values of the input and output. Therefore, it is called theconditional likelihood function. Theunconditional likelihood function can be obtained in two ways. The first is to set the starting values to zero, as is often done, and the second is to set them to the properly estimated values. The difference between these two types of likelihood functions is significant when the given data sequence is short, and any of the zeros of the moving average part is close to the boundary of the unit circle.In this paper the direct method of starting value estimation and its application to two off-line ARMA estimation algorithms, the maximum likelihood (ML) algorithm and the iterative inverse filtering (ITIF) algorithm, is proposed. Experimental results prove both increased efficiency and stability of these algorithms.The importance of setting the starting values properly is also significant when the recursive algorithm, with previously estimated parameters, has to be restarted. The advantage of the proposed reinitialization method is shown on the recursive lattice algorithm working in the block mode.     

Robustness of representations in multilayer feedforward neural networks

Recent research has shown that multilayer feedforward networks with sigmoidal activation functions are universal approximators, and that this holds for more general activations as well. The mathematical underpinning for these results has been various: Kolmogorov's resolution of Hilbert's thirteenth problem; the Stone-Weierstrass theorem; approximation of Fourier and Radon integral representations; and convergence of probability measures. This paper

Transition layer thickness in microlaminar deposits

A theoretical analysis was carried out on the change of composition of a deposit obtained by the dual pulse method of forming laminar metal foils, with transition from a low current to a high current pulse, both in the galvanostatic and the potentiostatic mode of deposition. It was shown that the existence of a transition layer of varying composition between a layer of pure metal 1 and a layer consisting predominantly of the metal 2 is an inherent consequence of the electrochemical process, primarily because of an induction period in the concentration polarization with respect to ions of metal 1. The importance of this transition layer increases as the thickness of the layers of the two metals decreases. Eventually this limits the possibility of obtaining a sharp boundary between the layers, when the nanometre region of layer thickness is reached. Equations are given for calculating the deposition current density and rate of stirring of the electrolyte which provide a deposit of a required level of metal 1 in the layer of metal 2, as well as a required sharpness of the boundary between two layers. Experimental proof of the correctness of the analysis was sought. It was found that significant changes in the properties of the deposit occur in the same range of layer thickness in which the transition of the composition takes places.Nomenclature _c,1, a_c,2 transfer coefficient of the cathodic processes - C interfacial capacitance - C₁, C₂ concentration of the ions of metals 1 and 2 at the interface r - C ₀ ¹ , C ₀ ² concentration of ions of the metals 1 and 2 in solution - D ₁ diffusion coefficient for the diffusion of ions of the metal 1 - E _r,1,E _r,2 reversible potentials of metals 1 and 2, respectively - F the Faraday constant - J ₀ ¹ ,J ₀ ² exchange current density of the metals 1 and 2, respectively - M ₁,M ₂ atomic weights of the metals 1 and 2, respectively - kinematic viscosity of solution - ₁, ₂ densities of the metals 1 and 2 respectively - rotation speed (r.p.s.) - z number of electrons exchanged in the deposition process This paper is dedicated to Professor Brian E. Conway on the occasion of his 65th birthday, and in recognition of his outstanding contribution to electrochemistry.     

Synthesis of ZnO-based varistor precursor powders by means of the reaction spray process

High-purity, fine-grained oxide ceramic powders that are commonly used for ZnO-based varistors were synthesized by means of the reaction spray process. Processing steps included formation of the solutions of zinc nitrate and/or certain metal additives and spraying of the solution into a heated reaction column using a two-phase nozzle. After water evaporation from the droplets, the precursor salts were decomposed at elevated temperatures (up to 1473 K) in order to obtain homogeneous oxide powders with complex compositions corresponding to the final multicomponent varistor system. The decomposition behaviour of the precursors, as well as the characteristics of the resulting powders: crystallinity, phase composition, particle shape, morphology and particle-size distribution, were examined. It was shown that the characteristics of the reaction spray-derived powders can be controlled by adjusting the system and the solutions parameters.