Maximum likelihood estimation using square root information filters

The maximum likelihood parameter estimation algorithm is known to provide optimal estimates for linear time-invariant dynamic systems. However, the algorithm is computationally expensive and requires evaluations of the gradient of a log likelihood function and the Fisher information matrix. By using the square-root information filter, a numerically reliable algorithm to compute the required gradient and the Fisher information matrix is developed. The algorithm is a significant improvement over the methods based on the conventional Kalman filter. The square-root information filter relies on the use of orthogonal transformations that are well known for numerical reliability. This algorithm can be extended to real-time system identification and adaptive control     

Extended stochastic gradient identification algorithms for Hammerstein–Wiener ARMAX systems

An extended stochastic gradient algorithm is developed to estimate the parameters of Hammerstein–Wiener ARMAX models. The basic idea is to replace the unmeasurable noise terms in the information vector of the pseudo-linear regression identification model with the corresponding noise estimates which are computed by the obtained parameter estimates. The obtained parameter estimates of the identification model include the product terms of the parameters of the original systems. Two methods of separating the parameter estimates of the original parameters from the product terms are discussed: the average method and the singular value decomposition method. To improve the identification accuracy, an extended stochastic gradient algorithm with a forgetting factor is presented. The simulation results indicate that the parameter estimation errors become small by introducing the forgetting factor.     

Iterative weighted least squares algorithms for neural networks classifiers

This paper discusses learning algorithms of layered neural networks from the standpoint of maximum likelihood estimation. At first we discuss learning algorithms for the most simple network with only one neuron. It is shown that Fisher information of the network, namely minus expected values of Hessian matrix, is given by a weighted covariance matrix of input vectors. A learning algorithm is presented on the basis of Fisher's scoring method which makes use of Fisher information instead of Hessian matrix in Newton's method. The algorithm can be interpreted as iterations of weighted least squares method. Then these results are extended to the layered network with one hidden layer. Fisher information for the layered network is given by a weighted covariance matrix of inputs of the network and outputs of hidden units. Since Newton's method for maximization problems has the difficulty when minus Hessian matrix is not positive definite, we propose a learning algorithm which makes use of Fisher information matrix, which is non-negative, instead of Hessian matrix. Moreover, to reduce the computation of full Fisher information matrix, we propose another algorithm which uses only block diagonal elements of Fisher information. The algorithm is reduced to an iterative weighted least squares algorithm in which each unit estimates its own weights by a weighted least squares method. It is experimentally shown that the proposed algorithms converge with fewer iterations than error back-propagation (BP) algorithm.     

Estimating spiking irregularities under changing environments

We considered a gamma distribution of interspike intervals as a statistical model for neuronal spike generation. A gamma distribution is a natural extension of the Poisson process taking the effect of a refractory period into account. The model is specified by two parameters: a time-dependent firing rate and a shape parameter that characterizes spiking irregularities of individual neurons. Because the environment changes over time, observed data are generated from a model with a time-dependent firing rate, which is an unknown function. A statistical model with an unknown function is called a semiparametric model and is generally very difficult to solve. We used a novel method of estimating functions in information geometry to estimate the shape parameter without estimating the unknown function. We obtained an optimal estimating function analytically for the shape parameter independent of the functional form of the firing rate. This estimation is efficient without Fisher information loss and better than maximum likelihood estimation. We suggest a measure of spiking irregularity based on the estimating function, which may be useful for characterizing individual neurons in changing environments.     

An implementation of the Expectation-Maximisation (EM) algorithm for population pharmacokinetic–pharmacodynamic modelling in ACSLXTREME

An implementation of the Expectation-Maximisation (EM) algorithm in ACSLXTREME (AEGIS Technologies) for the analyses of population pharmacokinetic–pharmacodynamic (PKPD) data is demonstrated. The parameter estimation results are compared with those from NONMEM (Globomax) using the first order conditional estimate method. The estimates are comparable and it is concluded that the EM algorithm is a useful technique in population pharmacokinetic–pharmacodynamic modelling. The implementation also demonstrates the ease with which parameter estimation algorithms for population data can be implemented in simulation software packages.     

On Birnbaum–Saunders inference

The Birnbaum–Saunders distribution, also known as the fatigue-life distribution, is frequently used in reliability studies. We obtain adjustments to the Birnbaum–Saunders profile likelihood function. The modified versions of the likelihood function were obtained for both the shape and scale parameters, i.e., we take the shape parameter to be of interest and the scale parameter to be of nuisance, and then consider the situation in which the interest lies in performing inference on the scale parameter with the shape parameter entering the modeling in nuisance fashion. Modified profile maximum likelihood estimators are obtained by maximizing the corresponding adjusted likelihood functions. We present numerical evidence on the finite sample behavior of the different estimators and associated likelihood ratio tests. The results favor the adjusted estimators and tests we propose. A novel aspect of the profile likelihood adjustments obtained in this paper is that they yield improved point estimators and tests. The two profile likelihood adjustments work well when inference is made on the shape parameter, and one of them displays superior behavior when it comes to performing hypothesis testing inference on the scale parameter. Two empirical applications are briefly presented.     

Singularities affect dynamics of learning in neuromanifolds

The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities, ARMA time-series model, and many other cases. The standard statistical paradigm of the Cramér-Rao theorem does not hold, and the singularity gives rise to strange behaviors in parameter estimation, hypothesis testing, Bayesian inference, model selection, and in particular, the dynamics of learning from examples. Prevailing theories so far have not paid much attention to the problem caused by singularity, relying only on ordinary statistical theories developed for regular (nonsingular) models. Only recently have researchers remarked on the effects of singularity, and theories are now being developed.This article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures. We demonstrate our recent results on these problems. Simple toy models are also used to show explicit solutions. We explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically, because the Fisher information matrix degenerates, that the model selection criteria such as AIC, BIC, and MDL fail to hold in these models, that a smooth Bayesian prior becomes singular in such models, and that the trajectories of dynamics of learning are strongly affected by the singularity, causing plateaus or slow manifolds in the parameter space. The natural gradient method is shown to perform well because it takes the singular geometrical structure into account. The generalization error and the training error are studied in some examples.     

Maximum likelihood estimation via the extended covariance and combined square-root filters

The method of maximum likelihood is a general method for parameter estimation and is often used in system identification. To implement it, it is necessary to maximize the likelihood function, which is usually done using the gradient approach. It involves the computation of the likelihood gradient with respect to unknown system parameters. For linear stochastic system models this leads to the implementation of the Kalman filter, which is known to be numerically unstable. The aim of this work is to present new efficient algorithms for likelihood gradient evaluation. They are more reliable in practice and improve robustness of computations against roundoff errors. All algorithms are derived in measurement and time updates form. The comparison with the conventional Kalman filter approach and results of numerical experiments are given.     

On minimum Fisher information distributions with restricted support and fixed variance

Fisher information is of key importance in estimation theory. It also serves in inference problems as well as in the interpretation of many physical processes. The mean-squared estimation error for the location parameter of a distribution is bounded by the inverse of the Fisher information associated with this distribution. In this paper we look for minimum Fisher information distributions with a restricted support. More precisely, we study the problem of minimizing the Fisher information in the set of distributions with fixed variance defined on a bounded subset S of R or on the positive real line. We show that the solutions of the underlying differential equation can be expressed in terms of Whittaker functions. Then, in the two considered cases, we derive the explicit expressions of the solutions and investigate their behavior. We also characterize the behavior of the minimum Fisher information as a function of the imposed variance.     

Toward a model-based Bayesian theory for estimating and recognizingparameterized 3-D objects using two or more images taken from differentpositions

A parametric modeling and statistical estimation approach is proposed and simulation data are shown for estimating 3-D object surfaces from images taken by calibrated cameras in two positions. The parameter estimation suggested is gradient descent, though other search strategies are also possible. Processing image data in blocks (windows) is central to the approach. After objects are modeled as patches of spheres, cylinders, planes and general quadrics-primitive objects, the estimation proceeds by searching in parameter space to simultaneously determine and use the appropriate pair of image regions, one from each image, and to use these for estimating a 3-D surface patch. The expression for the joint likelihood of the two images is derived and it is shown that the algorithm is a maximum-likelihood parameter estimator. A concept arising in the maximum likelihood estimation of 3-D surfaces is modeled and estimated. Cramer-Rao lower bounds are derived for the covariance matrices for the errors in estimating the a priori unknown object surface shape parameters     

Self-scaled conjugate gradient training algorithms

This article presents some efficient training algorithms, based on conjugate gradient optimization methods. In addition to the existing conjugate gradient training algorithms, we introduce Perry's conjugate gradient method as a training algorithm [A. Perry, A modified conjugate gradient algorithm, Operations Research 26 (1978) 26–43]. Perry's method has been proven to be a very efficient method in the context of unconstrained optimization, but it has never been used in MLP training. Furthermore, a new class of conjugate gradient (CG) methods is proposed, called self-scaled CG methods, which are derived from the principles of Hestenes–Stiefel, Fletcher–Reeves, Polak–Ribière and Perry's method. This class is based on the spectral scaling parameter introduced in [J. Barzilai, J.M. Borwein, Two point step size gradient methods, IMA Journal of Numerical Analysis 8 (1988) 141–148]. The spectral scaling parameter contains second order information without estimating the Hessian matrix. Furthermore, we incorporate to the CG training algorithms an efficient line search technique based on the Wolfe conditions and on safeguarded cubic interpolation [D.F. Shanno, K.H. Phua, Minimization of unconstrained multivariate functions, ACM Transactions on Mathematical Software 2 (1976) 87–94]. In addition, the initial learning rate parameter, fed to the line search technique, was automatically adapted at each iteration by a closed formula proposed in [D.F. Shanno, K.H. Phua, Minimization of unconstrained multivariate functions, ACM Transactions on Mathematical Software 2 (1976) 87–94; D.G. Sotiropoulos, A.E. Kostopoulos, T.N. Grapsa, A spectral version of Perry's conjugate gradient method for neural network training, in: D.T. Tsahalis (Ed.), Fourth GRACM Congress on Computational Mechanics, vol. 1, 2002, pp. 172–179]. Finally, an efficient restarting procedure was employed in order to further improve the effectiveness of the CG training algorithms. Experimental results show that, in general, the new class of methods can perform better with a much lower computational cost and better success performance.     

Parameter Estimation for Biomechanical Models Based on a Special Form of Natural Coordinates

We present parameter estimation results for a full three-dimensional model of the human body with nearly 100 degrees of freedom. This task could only be achieved by the use of sophisticated numerical techniques not only for optimization but also for the model setup. We have developed an object-oriented biomechanical modeling library based on a special form of natural coordinates that does not only serve to establish the full set of equations of motion of highly complex biome-chanical systems, but also to efficiently compute all the derivative information that is required in the parameter estimation context. Our parameter estimation algorithm is based on a multiple shooting state discretization and uses a generalized Gauss–Newton method. Eight experiments are combined in a multiple experiment setting. Inconsistent initial values are treated by a special form of non-stiff Baumgarte relaxation.     

Quantum Fisher information matrix for unitary processes: closed relation for <Emphasis Type="Italic">SU</Emphasis>(2)

Quantum Fisher information plays a central role in the field of quantum metrology. In this paper, we study the problem of quantum Fisher information of unitary processes. Associated with each parameter \(\theta _i\) of unitary process \(U(\varvec{\theta })\), there exists a unique Hermitian matrix \(M_{\theta _i}=i(U^\dagger \partial _{\theta _i} U)\). Except for some simple cases, such as when the parameter under estimation is an overall multiplicative factor in the Hamiltonian, calculation of these matrices is not an easy task to treat even for estimating a single parameter of qubit systems. Using the Bloch vector \(\varvec{m}_{\theta _i}\), corresponding to each matrix \(M_{\theta _i}\), we find a closed relation for the quantum Fisher information matrix of the SU(2) processes for an arbitrary number of estimation parameters and an arbitrary initial state. We extend our results and present an explicit relation for each vector \(\varvec{m}_{\theta _i}\) for a general Hamiltonian with arbitrary parametrization. We illustrate our results by obtaining the quantum Fisher information matrix of the so-called angle-axis parameters of a general SU(2) process. Using a linear transformation between two different parameter spaces of a unitary process, we provide a way to move from quantum Fisher information of a unitary process in a given parametrization to the one of the other parametrizations. Knowing this linear transformation enables one to calculate the quantum Fisher information of a composite unitary process, i.e., a unitary process resulted from successive action of some simple unitary processes. We apply this method for a spin-half system and obtain the quantum Fisher matrix of the coset parameters in terms of the one of the angle-axis parameters.     

Computation of the exact Cramer–Rao lower bound for the parameters of a nonsymmetric half-plane 2-D ARMA model

The Cramer–Rao lower bound (CRLB) that gives the minimal achievable variance/standard deviation for any unbiased estimator offers a useful tool for an assessment of the consistency of parameter estimation techniques. In this paper, a closed-form expression for the computation of the exact CRLB on unbiased estimates of the parameters of a two-dimensional (2-D) autoregressive moving average (ARMA) model with a nonsymmetric half-plane (NSHP) region of support is developed. The proposed formulation is mainly based on a matrix representation of 2-D real-valued discrete and homogeneous random field characterized by the NSHP ARMA model. Assuming that the random field is Gaussian, the covariance matrix of the NSHP ARMA random field is first expressed in terms of the model parameters. Then, using this matrix structure, a closed-form expression of the exact Fisher information matrix required for the CRLB computation of the NSHP ARMA model parameters is developed. Finally, the main formulas derived for the NSHP ARMA model are rearranged for its autoregressive and moving average counterparts, separately. Numerical simulations are included to demonstrate the behavior of the derived CRLB formulas.     

Maximum likelihood estimation of an extended latent Markov model for clustered binary panel data

Computational aspects concerning a model for clustered binary panel data are analyzed. The model is based on the representation of the behavior of a subject (individual panel member) in a given cluster by means of a latent process. This latent process is decomposed into a cluster-specific component and an individual-specific component. The first component follows a first-order Markov chain, whereas the second is time-invariant and is represented by a discrete random variable. An algorithm for computing the joint distribution of the response variables is introduced. The algorithm may be used even in the presence of a large number of subjects in the same cluster. An Expectation-Maximization (EM) scheme for the maximum likelihood estimation of the model is also described together with the estimation of the Fisher information matrix on the basis of the numerical derivative of the score vector. The estimate of this matrix is used to obtain standard errors for the parameter estimates and to check the identifiability of the model and the convergence of the EM algorithm. The approach is illustrated by means of an application to a data set concerning Italian employees’ illness benefits.     

Parameter estimation and system identification for continuously-observed quantum systems

This paper gives an overview of parameter estimation and system identification for quantum input–output systems by continuous observation of the output field. We present recent results on the quantum Fisher information of the output with respect to unknown dynamical parameters. We discuss the structure of continuous-time measurements as solutions of the quantum Zakai equation, and their relationship to parameter estimation methods. Proceeding beyond parameter estimation, the paper also gives an overview of the emerging topic of quantum system identification for black-box modelling of quantum systems by continuous observation of a travelling wave probe, for the case of ergodic quantum input–output systems and linear quantum systems. Empirical methods for such black-box modelling are also discussed.     

Computing and estimating information matrices of weak ARMA models

Numerous time series admit weak autoregressive-moving average (ARMA) representations, in which the errors are uncorrelated but not necessarily independent nor martingale differences. The statistical inference of this general class of models requires the estimation of generalized Fisher information matrices. Analytic expressions are given for these information matrices, and consistent estimators, at any point of the parameter space, are proposed. The theoretical results are illustrated by means of Monte Carlo experiments and by analyzing the dynamics of daily returns and squared daily returns of financial series.     

Identification of Hammerstein Nonlinear Stochastic Systems

Identification of Hammerstein stochastic dynamic systems is investigated. For this problem, the nonlinearities of the system must be taken into account. The solution is derived by recurrent gradient identification algorithms based on the Newton–Raphson and least-squares methods. Convergence is demonstrated, convergence rate is estimated, and parameter estimation accuracy is derived. The procedure is shown to be effective in practice.     

基于高斯分布监督学习样本的Fisher核构造方法

结合实际应用背景, 针对各类样本服从高斯分布的监督学习情形, 提出了构造Fisher核的新方法. 由于利用了样本中的类别信息, 该方法用极大似然估计代替EM算法估计GMM参数, 有效降低了Fisher核构造的时间复杂度. 结合核Fisher分类法, 上述方法在标准人脸库上的仿真实验结果显示, 用所提方法所构造的Fisher核不仅时间复杂度低, 且识别率也优于传统的高斯核与多项式核. 本文的研究有利于将Fisher 核的应用从语音识别领域拓展到图像识别等领域.