Comments on `Identification of non-linear parametrically varying models using separable least squares’ by F. Previdi and M. Lovera: black-box or open box?

This note compares and contrasts the non-linear parameter varying (NLPV) and state-dependent parameter (SDP) model classes. It shows that, while they have similarities, the two-stage SDP modelling procedure, involving non-parametric identification, followed by parametric estimation, is quite different from the single stage NLPV procedure. In particular, the SDP procedure allows for the identification of the model structure and the nature of the non-linearities, prior to the estimation of the parameters that characterize this identified model structure. In contrast to NLPV modelling, therefore, SDP estimation opens up the ‘black box’ and reveals the inner nature of the non-linear system.     

Robust glottal source estimation based on joint source-filter model optimization

This paper describes a robust glottal source estimation method based on a joint source-filter separation technique. In this method, the Liljencrants-Fant (LF) model, which models the glottal flow derivative, is integrated into a time-varying ARX speech production model. These two models are estimated in a joint optimization procedure, in which a Kalman filtering process is embedded for adaptively identifying the vocal tract parameters. Since the formulated joint estimation problem is a multiparameter nonlinear optimization procedure, we separate the optimization procedure into two passes. The first pass initializes the glottal source and vocal tract models by solving a quasi-convex approximate optimization problem. Having robust initial values, the joint estimation procedure determines the accuracy of model estimation implemented with a trust-region descent optimization algorithm. Experiments with synthetic and real voice signals show that the proposed method is a robust glottal source parameter estimation method with a high degree of accuracy.     

Estimates for the generalized cross-validation function via an extrapolation and statistical approach

Generalized cross-validation (GCV) is a popular tool for specifying the tuning parameter in linear regression model or equivalently the regularization parameter in Tikhonov regularization. In this work, we are concerned with the estimation and minimization of the GCV function by using a combination of an extrapolation procedure and a statistical approach. In particular, we derive families of estimates for the GCV function. By minimizing the estimated GCV function over a grid of values, a GCV estimate of the regularization parameter is achieved. We present several numerical examples to illustrate the effectiveness of the derived families of estimates for approximating the regularization parameter for several linear discrete ill-posed problems.     

Identification of Neurofuzzy Models Using GTLS Parameter Estimation

In this paper, nonlinear system identification utilizing generalized total least squares (GTLS) methodologies in neurofuzzy systems is addressed. The problem involved with the estimation of the local model parameters of neurofuzzy networks is the presence of noise in measured data. When some or all input channels are subject to noise, the GTLS algorithm yields consistent parameter estimates. In addition to the estimation of the parameters, the main challenge in the design of these local model networks is the determination of the region of validity for the local models. The method presented in this paper is based on an expectation–maximization algorithm that uses a residual from the GTLS parameter estimation for proper partitioning. The performance of the resulting nonlinear model with local parameters estimated by weighted GTLS is a product both of the parameter estimation itself and the associated residual used for the partitioning process. The applicability and benefits of the proposed algorithm are demonstrated by means of illustrative examples and an automotive application.      

CAD package for industrial process identification

A CAD system for off-line industrial process identification is presented. The system has two modules. In the first data acquisition is performed. The second performs the process identification procedure. The functions and facilities of these methods are described, which include: the configuration of the data acquisition procedure to be performed, preliminary data analysis, model structure and parameter estimation method selection, time varying parameter estimation and model validation. An industrial application in petrochemical process identification is presented.     

A revisit to MacKay algorithm and its application to deep network compression

An iterative procedure introduced in MacKay’s evidence framework is often used for estimating the hyperparameter in empirical Bayes. Together with the use of a particular form of prior, the estimation of the hyperparameter reduces to an automatic relevance determination model, which provides a soft way of pruning model parameters. Despite the effectiveness of this estimation procedure, it has stayed primarily as a heuristic to date and its application to deep neural network has not yet been explored. This paper formally investigates the mathematical nature of this procedure and justifies it as a well-principled algorithm framework, which we call the MacKay algorithm. As an application, we demonstrate its use in deep neural networks, which have typically complicated structure with millions of parameters and can be pruned to reduce the memory requirement and boost computational efficiency. In experiments, we adopt MacKay algorithm to prune the parameters of both simple networks such as LeNet, deep convolution VGG-like networks, and residual netowrks for large image classification task. Experimental results show that the algorithm can compress neural networks to a high level of sparsity with little loss of prediction accuracy, which is comparable with the state-of-the-art.     

Optimal l∞ to l∞ estimation forperiodic systems

In this paper we consider the problem of finding a filter that minimizes the worst-case magnitude (l_∞) of the estimation error in the case of linear periodically time-varying systems subjected to unknown but magnitude-bounded (l_∞) inputs. These inputs consist of process and observation noises, and the optimization problem is considered over an infinite-time horizon. Lifting techniques are utilized to transform the problem to a time invariant l₁-model matching problem subject to additional constraints. Taking advantage of the particular structure of the estimation problem, it is shown how standard methods of l₁ optimization, in particular the delay augmentation technique, can be suitably modified to solve this nonstandard problem     

Dynamic parameter estimation for a street canyon air quality model

The Operational Street Pollution Model (OSPM^®) is a widely used air quality model for urban street canyons. It is a parametric model, simulating the contribution from traffic emissions on a single street at receptor points at the buildings' facades. The OSPM contains a number of empirical parameters, accounting for processes such as emission factors or dispersion of pollutants. The values of these parameters are based on empirical assumptions, and might not be optimal for a specific street. In this work, we allow these parameters to vary within a certain meaningful range.We implemented two different parameter estimation schemes: a dynamic estimation procedure (using an ensemble Kalman filter) that allowed parameter values to vary, and a static estimation procedure scheme (using a least-squares algorithm) that kept parameter values fixed during the course of the simulation. We ran year-long simulations for five different streets in Danish cities, and evaluated performance by comparing forecast concentrations of NO_x, NO₂, O₃ and CO with observations.Overall, the parameter estimation substantially improved the performance of the model in forecasting, especially for NO₂ and CO. However it led to slightly more bias in the modelled daily maximum concentrations, suggesting that the parameter estimation fits to the bulk of the data rather than the extremes. Estimated parameter values varied substantially in time and between sites, making it difficult to generalise parameter estimates to other locations. Modelled concentrations from the OSPM were, on average, notably more accurate in simulations using measured urban background concentrations and meteorological parameters compared to using modelled data for these inputs. However this is only applicable when observations from nearby meteorological and urban background monitoring sites are available.We conclude that although dynamic parameter estimation has limited applicability to real-time air quality forecasting, it can potentially give useful feedback about the quality of model parameterisations or model inputs. Static parameter estimation is a simpler method, which is often as effective as dynamic parameter estimation.     

Identification of discrete-time output error model for industrial processes with time delay subject to load disturbance

In this paper, a bias-eliminated output error model identification method is proposed for industrial processes with time delay subject to unknown load disturbance with deterministic dynamics. By viewing the output response arising from such load disturbance as a dynamic parameter for estimation, a recursive least-squares identification algorithm is developed in the discrete-time domain to estimate the linear model parameters together with the load disturbance response, while the integer delay parameter is derived by using a one-dimensional searching approach to minimize the output fitting error. An auxiliary model is constructed to realize consistent estimation of the model parameters against stochastic noise. Moreover, dual adaptive forgetting factors are introduced with tuning guidelines to improve the convergence rates of estimating the model parameters and the load disturbance response, respectively. The convergence of model parameter estimation is analyzed with a rigorous proof. Illustrative examples for open- and closed-loop identification are shown to demonstrate the effectiveness and merit of the proposed identification method.     

Optimal Adaptive Control—Contradiction in Terms or a Matter of Choosing the Right Cost Functional?

Approaching the problem of optimal adaptive control as ldquooptimal control made adaptive,rdquo namely, as a certainty equivalence combination of linear quadratic optimal control and standard parameter estimation, fails on two counts: numerical (as it requires a solution to a Riccati equation at each time step) and conceptual (as the combination actually does not possess any optimality property). In this note, we present a particular form of optimality achievable in Lyapunov-based adaptive control. State and control are subject to positive definite penalties, whereas the parameter estimation error is penalized through an exponential of its square, which means that no attempt is made to enforce the parameter convergence, but the estimation transients are penalized simultaneously with the state and control transients. The form of optimality we reveal here is different from our work in [Z. H. Li and M. Krstic, ldquoOptimal design of adaptive tracking controllers for nonlinear systems,rdquo Automatica, vol. 33, pp. 1459-1473, 1997] where only the terminal value of the parameter error was penalized. We present our optimality concept on a partial differential equation (PDE) example-boundary control of a particular parabolic PDE with an unknown reaction coefficient. Two technical ideas are central to the developments in the note: a nonquadratic Lyapunov function and a normalization in the Lyapunov-based update law. The optimal adaptive control problem is fundamentally nonlinear and we explore this aspect through several examples that highlight the interplay between the non-quadratic cost and value functions.     

Practical aspects of process identification

After the presentation of various identification and parameter estimation methods in the previous papers, some selected practical aspects of process identification are discussed. This includes, for a given identification method, the steps from the design of experiments to the verification of the final model. Therefore a general procedure of process identification, the selection of input signals, the selection of the sampling time, off-line and on-line identification, comparison of parameter estimation methods, model order testing and model verification is presented. A short discussion on program packages for process identification follows.     

Parameter estimation of radiocardiogram using a minicomputer

Radiocardiography has been widely used as a method for the quantification of cardiac output by applying the principle of the dye dilution method. This paper deals with an automatic analyzing system of radiocardiograms and a parameter estimation procedure using a linear system made up of four compartments with two time delays as a model of transport process in the blood circulatory system. The parameter estimation procedure named the window method in frequency domain is very effective for shortening the computing time and can be easily performed using a minicomputer. Parameter sensitivity analysis is also applied to study behavior of parameters on the model. Some analyzed results of radiocardiograms are shown and it is verified that the procedure is sufficiently useful and efficient for routine clinical use.     

An algorithm for steady-state system optimization and parameter estimation

In this paper an algorithm is described which uses a steady-state mode! to determine the optimum operating point of a process. The model, which is not required to be an accurate representation of the real process, contains parameters to be estimated and the algorithm involves an iterative procedure between the two problems of system optimization and parameter estimation. Lagrangian analysis is employed to account for the interaction between the two problems, resulting in a procedure which may be regarded as a modified two-step approach in which the optimization objective index includes an extra term. The extra term contains a comparison between model and real process output derivatives and ensures that the optimal steady-state operating condition is achieved in spite of model inaccuracies.

The algorithm is shown to perform satisfactorily in a digital simulation study concerned with determining food flow rate and temperature controller set points to maximize the net rate of return from an exothermic chemical reactor using a simplified non-linear model for system optimization and parameter estimation. The simulation is employed to investigate the convergence properties of the algorithm and to study the effects of measurement errors.     

Reflectance Parameter Estimation and its Application to Surface Inspection

In this paper, we present a reflectance parameter estimation technique by using range and brightness and its relation, i.e. reflectance function. Because the reflectance function is quite complex and nonlinear, the parameter estimation is not straightforward. Therefore, we choose a coarse-to-fine approach to estimate the reflectance parameters. In the coarse step, the surface toughness is coarsely estimated by applying the partial linear method to the simplified Torrance-Sparrow reflectance model. Then the genetic algorithm is applied to the Wolff's reflectance model for more accurate estimation. In order to extend the dynamic range of CCD of laser finder, in this paper, we introduce the pseudo-brightness. The proposed reflectance parameter estimation algorithm is tested on the synthesized and real data. The results show that the estimated parameter using the synthesized data is very accurate. We also apply the proposed algorithm to inspect the flaws on shiny surfaces, which would be a promising method to discriminate between a wide range of surfaces.     

Multi-objective parameter estimation via minimal correlation criterion

The paper deals with the problem of parameter estimation using two different sources of information, namely a time series with dynamic data and steady-state data. The new estimator is based on a two-step procedure: first a multi-objective optimization is performed, leading to a set of Pareto-optimal vectors of parameter estimates and, second, a single model is chosen based on the free-run simulation error which is required to be minimally correlated with the model output. The procedure is general in nature and can be applied to any model representation, but for the sake of simplicity, the new procedure is illustrated using NARX polynomial models for which closed formulae for generating the Pareto-set are readily available. Monte Carlo simulation studies suggest that the new estimator, which does not assume any particular noise model, is fairly unbiased even when the conventional least-squares estimator is biased.     

Kalman filter-based identification for systems with randomly missing measurements in a network environment

We consider the problem of parameter estimation and output estimation for systems in a transmission control protocol (TCP) based network environment. As a result of networked-induced time delays and packet loss, the input and output data are inevitably subject to randomly missing data. Based on the available incomplete data, we first model the input and output missing data as two separate Bernoulli processes characterised by probabilities of missing data, then a missing output estimator is designed, and finally we develop a recursive algorithm for parameter estimation by modifying the Kalman filter-based algorithm. Under the stochastic framework, convergence properties of both the parameter estimation and output estimation are established. Simulation results illustrate the effectiveness of the proposed algorithms.     

Estimation of Alonso's Theory of Movements by Means of Instrumental Variables

Alonso's Theory of Movements is a widely applicable Spatial Interaction Model, describing an equilibrium of inflows and outflows. Based on a survey of estimation attempts in the past, we conclude that econometric estimation of the so-called systemic parameters is somewhat problematic. In this paper we describe a new estimation method, using instrumental variables which are derived from the model. The distance deterrence parameter and the unobserved balancing factors can be estimated first, with known methods. The remaining parameters can be estimated in an iterative regression procedure, using instruments for the balancing factors. These instruments are derived as the predicted values of the balancing factors, based on the last obtained parameter estimates.     

Robust l1 estimation using the Popov-Tsypkin multiplier with application to robust fault detection

This paper considers the design of robust l₁ estimators based on multiplier theory (which is intimately related to mixed structured singular value theory) and the application of robust l₁ estimators to robust fault detection. The key to estimator-based, robust fault detection is to generate residuals which are robust against plant uncertainties and external disturbance inputs, which in turn requires the design of robust estimators. Specifically, the Popov-Tsypkin multiplier is used to develop an upper bound on an l₁ cost function over an uncertainty set. The robust l₁ estimation problem is formulated as a parameter optimization problem in which the upper bound is minimized subject to a Riccati equation constraint. A continuation algorithm that uses quasi-Newton BFGS (the algorithm of Broyden, Fletcher, Goldfab and Shanno) corrections is developed to solve the minimization problem. The estimation algorithm has two stages. The first stage solves a mixed-norm H₂/l₁ estimation problem. In particular, it is initialized with a steady-state Kalman filter and, by varying a design parameter from 0 to 1, the Kalman filter is deformed to an l₁ estimator. In the second stage the l₁ estimator is made robust. The robust l₁ estimation framework is then applied to the robust fault detection of dynamic systems. The results are applied to a simplified longitudinal flight control system. It is shown that the robust fault detection procedure based on the robust l₁ estimation methodology proposed in this paper can reduce false alarm rates.     

Cluster number selection for a small set of samples using theBayesian Ying-Yang model

One major problem in cluster analysis is the determination of the number of clusters. In this paper, we describe both theoretical and experimental results in determining the cluster number for a small set of samples using the Bayesian-Kullback Ying-Yang (BYY) model selection criterion. Under the second-order approximation, we derive a new equation for estimating the smoothing parameter in the cost function. Finally, we propose a gradient descent smoothing parameter estimation approach that avoids complicated integration procedure and gives the same optimal result.     

Adaptive control and identification of the dissolved oxygen process

This paper suggests how nonlinear adaptive control might lead to improved control of the dissolved oxygen (DO) concentration in the aerator of a wastewater treatment plant. The DO dynamics can be represented by a bilinear model for which we are interested in both parameter identification and control. The estimation of key parameters of the process model is important because the values of these parameters cannot be obtained from direct measurement. Hence a least-squares procedure for obtaining unique parameter estimates is developed and then combined with a minimum variance control algorithm to obtain an adaptive controller which is used both to generate useful parameter estimates and to control the process. Extensions to the case where the parameters vary at the same rate as the DO are also discussed.