统计机器学习中参数可辨识性研究及其关键问题

参数可辨识性研究在统计机器学习中具有重要的理论意义和应用价值.参数可辨识性是关于模型参数能否被惟一确定的性质.在包含物理参数的学习模型中,可辨识性不仅是物理参数获得正确估计的前提条件,更重要的是,它反映了学习机器中由参数决定的物理特征.为扩展到未来类人智能机器研究的考察视角,我们将学习模型纳入"知识与数据共同驱动模型"的框架中讨论.在此框架下,我们提出两个关键问题.第一是参数可辨识性准则问题.该问题考察与可辨识性密切相关的各种判断准则,其中知识驱动子模型与数据驱动子模型的耦合方式为参数可辨识性问题提供了新的研究空间.第二是参数可辨识性与机器学习理论和应用相关联的研究.该研究包括可辨识性对参数估计、模型选择、学习算法、学习动态过程、奇异学习理论、贝叶斯推断等内容的深刻影响.     

Assessing local structural identifiability for environmental models

The local structural identifiability problem is investigated for the general case and demonstrated for a well-known microbial degradation model that includes 13 unknown parameters and 3 additional states. We address the identifiability question using a novel algorithm that can be used for large models with many parameters to be identified. A key ingredient in the analysis is the application of a singular value decomposition of the normalized parametric output sensitivity matrix that is obtained through a simple model integration. The SVD results are further analysed and verified in a complementary symbolic computation. It is especially the swiftness and accuracy of the suggested method that we consider to be a substantial advantage in comparison to existing methods for a structural identifiability analysis. The method also opens, in a natural way, the analysis of (parametric) uncertainty in general, and this is demonstrated in more detail in the results section.     

Local identifiability and sensitivity analysis of neuromuscular blockade and depth of hypnosis models

This paper addresses the local identifiability and sensitivity properties of two classes of Wiener models for the neuromuscular blockade and depth of hypnosis, when drug dose profiles like the ones commonly administered in the clinical practice are used as model inputs. The local parameter identifiability was assessed based on the singular value decomposition of the normalized sensitivity matrix. For the given input signal excitation, the results show an over-parameterization of the standard pharmacokinetic/pharmacodynamic models. The same identifiability assessment was performed on recently proposed minimally parameterized parsimonious models for both the neuromuscular blockade and the depth of hypnosis. The results show that the majority of the model parameters are identifiable from the available input–output data. This indicates that any identification strategy based on the minimally parameterized parsimonious Wiener models for the neuromuscular blockade and for the depth of hypnosis is likely to be more successful than if standard models are used.     

A parameter estimation and identifiability analysis methodology applied to a street canyon air pollution model

Mathematical models are increasingly used in environmental science thus increasing the importance of uncertainty and sensitivity analyses. In the present study, an iterative parameter estimation and identifiability analysis methodology is applied to an atmospheric model – the Operational Street Pollution Model (OSPM^®). To assess the predictive validity of the model, the data is split into an estimation and a prediction data set using two data splitting approaches and data preparation techniques (clustering and outlier detection) are analysed. The sensitivity analysis, being part of the identifiability analysis, showed that some model parameters were significantly more sensitive than others. The application of the determined optimal parameter values was shown to successfully equilibrate the model biases among the individual streets and species. It was as well shown that the frequentist approach applied for the uncertainty calculations underestimated the parameter uncertainties. The model parameter uncertainty was qualitatively assessed to be significant, and reduction strategies were identified.     

Identification and estimation of discrete state-vector models with stochastic inputs

The study deals with the identification and estimation of the unknown parameters of an ‘extended’ state-vector model, in which stochastic input variables are treated as ‘state’-variables and the observed input-values as ‘output’-values of the model.A parameter identifiability criterion, based on Fisher's information matrix, is applied to the model and a general ML-estimation procedure is given. If a certain restriction on the covariance-matrix of the state-vector is placed, the ML-procedure simplifies and coincides with an operational method, called the Lisrel procedure. This procedure provides also a test for parameter identifiability.     

DAISY: a new software tool to test global identifiability of biological and physiological systems

A priori global identifiability is a structural property of biological and physiological models. It is considered a prerequisite for well-posed estimation, since it concerns the possibility of recovering uniquely the unknown model parameters from measured input-output data, under ideal conditions (noise-free observations and error-free model structure). Of course, determining if the parameters can be uniquely recovered from observed data is essential before investing resources, time and effort in performing actual biomedical experiments. Many interesting biological models are nonlinear but identifiability analysis for nonlinear system turns out to be a difficult mathematical problem. Different methods have been proposed in the literature to test identifiability of nonlinear models but, to the best of our knowledge, so far no software tools have been proposed for automatically checking identifiability of nonlinear models. In this paper, we describe a software tool implementing a differential algebra algorithm to perform parameter identifiability analysis for (linear and) nonlinear dynamic models described by polynomial or rational equations. Our goal is to provide the biological investigator a completely automatized software, requiring minimum prior knowledge of mathematical modelling and no in-depth understanding of the mathematical tools. The DAISY (Differential Algebra for Identifiability of SYstems) software will potentially be useful in biological modelling studies, especially in physiology and clinical medicine, where research experiments are particularly expensive and/or difficult to perform. Practical examples of use of the software tool DAISY are presented. DAISY is available at the web site http://www.dei.unipd.it/~pia/.     

Statistical identifiability and convergence evaluation for nonlinear pharmacokinetic models with particle swarm optimization

The statistical identifiability of nonlinear pharmacokinetic (PK) models with the Michaelis–Menten (MM) kinetic equation is considered using a global optimization approach, which is particle swarm optimization (PSO). If a model is statistically non-identifiable, the conventional derivative-based estimation approach is often terminated earlier without converging, due to the singularity. To circumvent this difficulty, we develop a derivative-free global optimization algorithm by combining PSO with a derivative-free local optimization algorithm to improve the rate of convergence of PSO. We further propose an efficient approach to not only checking the convergence of estimation but also detecting the identifiability of nonlinear PK models. PK simulation studies demonstrate that the convergence and identifiability of the PK model can be detected efficiently through the proposed approach. The proposed approach is then applied to clinical PK data along with a two-compartmental model.     

A method for auto-tuning of PID control parameters

A method for automatic tuning of the PID process control parameters, usually called ‘auto-tuning’, is developed. The procedure of applying the method consists of (1) sampling a process response to a test input signal, (2) processing the sampled data for estimating characteristic values of the process, and (3) calculating the optimal values of the PID control parameters. For the optimization, a new type of performance index, i.e. a weighted integral of squared error is introduced. The procedure is implemented on a digital controller using microprocessors and applied to some real processes, yielding satisfactory results.     

挖掘回归类的混合模型的可识别性

复杂海量数据往往表现为多种结构特征的混合体，回归类混合模型就是对这种混合体的一个描述．该文基于统计学的有限混合分布理论和可识别性的相关结果，针对回归变量的三种情形：(1)解释变量固定，(2)解释变量随机，(3)解释变量固定且类别参数指定，分别讨论挖掘一般回归类的混合模型的可识别性问题，并给出同族回归类混合模型可识别的相应充分条件．这些条件的一个共同特点是它们都与一类特别的解释变量集合有关，而该类集合是由同族的回归函数与回归参数唯一确定的，其元素使不同的回归参数对应回归函数的相同值．特别地，当回归函数线性时，这类集合就是解释变量空间中的超平面．     

物探钻机性能参数测试系统设计

根据物探钻机的结构特点和测试需求,设计了一套适用于现有物探钻机的性能参数测试装置,实现了对钻压、空气压力、流量、转速、扭矩、钻井速度6项性能参数的测试,通过数据采集处理器对测试数据自动采集和处理,并开发了一套计算机软件,实现了对测试系统的自动控制和对性能参数的实时监测.现场试验表明,该装置运行稳定、测试精度高、软件运行平稳、存储数据效率高和处理分析准确,对于研究钻机和钻具以及钻井工艺参数优化具有重要作用.     

Gradient‐based back‐propagation dynamical iterative learning scheme for the neuro‐fuzzy inference system

In this paper, a gradient‐based back propagation dynamical iterative learning algorithm is proposed for structure optimization and parameter tuning of the neuro‐fuzzy system. Premise and consequent parameters of the neuro‐fuzzy model are initialized randomly and then tuned by the proposed iterative algorithm. The learning algorithm is based on the first order partial derivative of the output with respect to the structure parameters. The first order derivative of the model output with respect to the structure parameters determines the sensitivity of the model to structure parameters. The sensitivity values are then used to set the tuning factors and parameters updating step sizes. Therefore, an adaptive dynamical iterative scheme is achieved which adapts the learning procedure to the current state of the performance during the optimization process. Larger tuning step sizes make the convergence speed higher and vice versa. In this regard, this parameter is treated according to the calculated sensitivity of the model to the parameter. The proposed learning algorithm is compared with the least square back propagation method, genetic algorithm and chaotic genetic algorithm in the neuro‐fuzzy model structure optimization. Smaller mean square error and shorter learning time are sought in this paper, and the performance of the proposed learning algorithm is versified regarding these criteria.     

On-line model identification for the machining process based on multirate process data

Accurate identification of the model parameters of the machining process based on on-line process data is a crucial prerequisite for its model-based control and diagnostics. A typical machining process generates multi-output and multirate data streams. Whereas various sensors provide in-process information about the process, many important process outcomes including product qualities can be only measured in postprocess manner. This paper proposes to improve the identification by using both in-process and postprocess data and by analyzing the identifiability of model parameters. The identification of the model parameters based on multirate output is formulated using the maximum-likelihood estimation and the Fisher information matrix for a multirate-sampled system is derived to study the identifiability of model parameters. A strategy is developed to improve accuracy and robustness of the model identification considering the identifiability. The proposed method is tested on two batches of multirate process data from the cylindrical grinding process. The test results demonstrate using both in-process and postprocess data improves the identifiability and the proposed identification strategy results in improved prediction performance.     

Identification and experimental validation of an HIV model for HAART treated patients

The objective of this paper is to identify the parameters of a human immunodeficiency virus (HIV) evolution model from a clinical data set of patients treated with two different highly active antiretroviral therapy (HAART) protocols. After introducing a model with six state variables, a preliminary step considers the reduction of the number of parameters to be identified by means of sensitivity analysis, and then identifiability items are discussed. A nonlinear optimization-based procedure for identification is developed, which divides the unknown parameters into two families: the group dependent and the patient dependent parameters. Numerical results show that the identified model can be individually adapted to each patient and this result is promising for predicting the effects (e.g., failures or successes) of therapeutic actions.     

Structural developments of fuzzy systems with the aid of information granulation

We introduce a design procedure for fuzzy systems using the concept of information granulation and genetic optimization. Information granulation and resulting information granules themselves become an important design aspect of fuzzy models. By accommodating the formalism of fuzzy sets, the model is geared towards capturing relationship between information granules (fuzzy sets) rather than concentrating on plain numeric data. Information granulation realized with the use of the standard C-Means clustering helps determine the initial values of the parameters of the fuzzy models. This in particular concerns such essential components of the rules as the initial apexes of the membership functions standing in the premise part of the fuzzy rules and the initial values of the polynomial functions standing in the consequence part. The initial parameters are afterwards tuned with the aid of the genetic algorithms (GAs) and the least square method (LSM). The overall design methodology arises as a hybrid development process involving structural and parametric optimization. Especially, genetic algorithms and C-Means are used to generate the structurally as well as parametrically optimized fuzzy model. To identify the structure and estimate parameters of the fuzzy model we exploit the methodologies such as joint and successive method realized by means of genetic algorithms. The proposed model is evaluated using experimental data and its performance is contrasted with the behavior of the fuzzy models available in the literature.     

A priori identifiability: An overview on definitions and approaches

For a system, a priori identifiability is a theoretical property depending only on the model and guarantees that its parameters can be uniquely determined from observations. This paper provides a survey of the various and numerous definitions of a priori identifiability given in the literature, for both deterministic continuous and discrete-time models. A classification is done by distinguishing analytical and algebraic definitions as well as local and global ones. Moreover, this paper provides an overview on the distinct methods to test the parameter identifiability. They are classified into the so-called output equality approaches, local state isomorphism approaches and differential algebra approaches. A few examples are detailed to illustrate the methods and complete this survey.     

Identifiability study of an aquatic ecosystem model

An example of ecosystem modelization is presented and used to underline the problems in this area of system analysis. The model construction is analysed. The parameters identification requires the test of a priori identifiability of a complex non-linear model (structural identifiability) ; it then calls for the choice of a good identification method ns the input signals of an ecosystem cannot be manipulated, to guarantee a posteriori identifiability. This method is applied to data collected on an alpine lake     

Surrogate modeling approximation using a mixture of experts based on EM joint estimation

An automatic method to combine several local surrogate models is presented. This method is intended to build accurate and smooth approximation of discontinuous functions that are to be used in structural optimization problems. It strongly relies on the Expectation−Maximization (EM) algorithm for Gaussian mixture models (GMM). To the end of regression, the inputs are clustered together with their output values by means of parameter estimation of the joint distribution. A local expert is then built (linear, quadratic, artificial neural network, moving least squares) on each cluster. Lastly, the local experts are combined using the Gaussian mixture model parameters found by the EM algorithm to obtain a global model. This method is tested over both mathematical test cases and an engineering optimization problem from aeronautics and is found to improve the accuracy of the approximation.     

Selection of outputs for distributed parameter systems by identifiability analysis in the time-scale domain

A method of sensor location selection is introduced for distributed parameter systems. In this method, the sensitivities of spatial outputs to model parameters are computed by a model and transformed via continuous wavelet transforms into the time-scale domain to characterise the shape attributes of output sensitivities and accentuate their differences. Regions are then sought in the time-scale plane wherein the wavelet coefficient of an output sensitivity surpasses all the others’ as indication of the output sensitivity’s distinctness. This yields a comprehensive account of identifiability each output provides to the model parameters as the basis of output selection. The proposed output selection strategy is demonstrated for a numerical case of pollutant dispersion by advection and diffusion in a two-dimensional area.     

On the use of the norm-coerciveness theorem in the identifiability problem of linear compartmental models

The identifiability of parameters in a model with known structure and no noise is the problem of accurate determination of the number of parameter space points which are solutions to the identification problem. Using the norm-coerciveness theorem, we demonstrate a result on global identifiability. We introduce the idea of a strong separator of the parameter space, which divides this space into various connected domains; in each of these domains there is one and only one solution to the problem of identification of parameters. To simplify the presentation, notations and examples of linear compartmental models are used here, but the main result (Theorem 3) is valid for all linear systems. Unfortunately, it is not always possible to use this result because the assumptions of this theorem are strong.     

Visual Parameter Space Exploration in Time and Space

Computational models, such as simulations, are central to a wide range of fields in science and industry. Those models take input parameters and produce some output. To fully exploit their utility, relations between parameters and outputs must be understood. These include, for example, which parameter setting produces the best result (optimization) or which ranges of parameter settings produce a wide variety of results (sensitivity). Such tasks are often difficult to achieve for various reasons, for example, the size of the parameter space, and supported with visual analytics. In this paper, we survey visual parameter space exploration (VPSE) systems involving spatial and temporal data. We focus on interactive visualizations and user interfaces. Through thematic analysis of the surveyed papers, we identify common workflow steps and approaches to support them. We also identify topics for future work that will help enable VPSE on a greater variety of computational models.