Clustering based on gaussian processes

In this letter, we develop a gaussian process model for clustering. The variances of predictive values in gaussian processes learned from a training data are shown to comprise an estimate of the support of a probability density function. The constructed variance function is then applied to construct a set of contours that enclose the data points, which correspond to cluster boundaries. To perform clustering tasks of the data points, an associated dynamical system is built, and its topological invariant property is investigated. The experimental results show that the proposed method works successfully for clustering problems with arbitrary shapes.     

Predictive approaches for choosing hyperparameters in gaussian processes

Gaussian processes are powerful regression models specified by parameterized mean and covariance functions. Standard approaches to choose these parameters (known by the name hyperparameters) are maximum likelihood and maximum a posteriori. In this article, we propose and investigate predictive approaches based on Geisser's predictive sample reuse (PSR) methodology and the related Stone's cross-validation (CV) methodology. More specifically, we derive results for Geisser's surrogate predictive probability (GPP), Geisser's predictive mean square error (GPE), and the standard CV error and make a comparative study. Within an approximation we arrive at the generalized cross-validation (GCV) and establish its relationship with the GPP and GPE approaches. These approaches are tested on a number of problems. Experimental results show that these approaches are strongly competitive with the existing approaches.     

Predictive on-line monitoring of continuous processes

For safety and product quality, it is important to monitor process performance in real time. Since traditional analytical instruments are usually expensive to install, a process model can be used instead to monitor process behavior. In this paper, a monitoring approach using a multivariate statistical modeling technique, namely multi-way principal component analysis (MPCA), is studied. The method overcomes the assumption that the system is at steady state and it provides a real time monitoring approach for continuous processes. The monitoring approach using MPCA models can detect faults in advance of other monitoring approaches. Several issues which are important for the proposed approach, such as the model input structure, data pretreatment, and the length of the predictive horizon are discussed. A multi-block extension of the basic methodology is also treated and this extension is shown to facilitate fault isolation. The Tennessee Eastman process is used for demonstrating the power of the new monitoring approach.     

Properties of separable covariance matrices and their associated gaussian random processes

A number of properties of separable covariance matrices are summarized. Expressions for the divergence of the corresponding two-dimensional Gaussian random processes are given in terms of row and column covariance matrices, and in terms of linear prediction parameters and maximum likelihood spectral estimates. Such time and frequency domain expressions are not widely known, even for one-dimensional random processes.     

Adaptive on-line steady-state optimization of slow dynamic processes

A software package OLIOPT was developed for the on-line optimization of the steady-state behaviour of slow dynamic processes in a relatively short time period. In the starting phase, the independently variable inputs are changed according to a special test signal. A nonlinear dynamic process model is identified on-line. Based on the static part of the model and the known inputs, the gradients of the performance index are calculated. An optimization algorithm changes the inputs towards their optimal values. On-line identification of the nonlinear model continues and the prediction of the optimum improves. In the last phase, the inputs take their optimal values and the process follows, feedforward controlled, to its optimal steady-state. The method is suited for industrial processes with one or more variable inputs, where a small gain in efficiency turns out to give a relatively large financial return. Results are shown for the on-line optimization of a thermal pilot process.     

Fault prognosis of filamentous sludge bulking using an enhanced multi-output gaussian processes regression

The activated sludge process (ASP) is widely adopted to remove pollutants in wastewater treatment plants (WWTPs). However, the occurrence of filamentous sludge bulking often compromises the stable operation of the ASP. For timely diagnosis of filamentous sludge bulking for an activated sludge process in advance, this study proposed a Multi-Output Gaussian Processes Regression (MGPR) model for multi-step prediction and presented the Vector auto-regression (VAR) to learn the MGPR modelling deviation. The resulting models and associated uncertainty levels are used to monitor the filamentous sludge bulking related parameter, sludge volume index (SVI), such that the evolution of SVI can be predicted for both one-step and multi-step ahead. This methodology was validated with SVI data collected from one full-scale WWTP. Online diagnosis and prognosis of filamentous bulking sludge with real-time SVI prediction were tested through a simulation study. The results demonstrated that the proposed methodology was capable of predicting future SVI with good accuracy, thereby providing sufficient time for filamentous sludge bulking.     

An on-line framework for monitoring nonlinear processes with multiple operating modes

A multivariate statistical process monitoring scheme should be able to describe multimodal data. Multimodality typically arises in process data due to varying production regimes. Moreover, multimodality may influence how easy it is for process operators to interpret the monitoring results. To address these challenges, this paper proposes an on-line monitoring framework for anomaly detection where an anomaly may either indicate a fault occurring and developing in the process or the process moving to a new operating mode. The framework incorporates the Dirichlet process, which is an unsupervised clustering method, and kernel principal component analysis with a new kernel specialized for multimode data. A monitoring model is trained using the data obtained from several healthy operating modes. When on-line, if a new healthy operating mode is confirmed by an operator, the monitoring model is updated using data collected in the new mode. Implementation issues of this framework, including the parameter tuning for the kernel and the selection of anomaly indicators, are also discussed. A bivariate numerical simulation is used to demonstrate the performance of anomaly detection of the monitoring model. The ability of this framework in model updating and anomaly detection in new operating modes is shown on data from an industrial-scale process using the PRONTO benchmark dataset. The examples will also demonstrate the industrial applicability of the proposed framework.     

Bayesian framework for least-squares support vector machine classifiers, gaussian processes, and kernel Fisher discriminant analysis

The Bayesian evidence framework has been successfully applied to the design of multilayer perceptrons (MLPs) in the work of MacKay. Nevertheless, the training of MLPs suffers from drawbacks like the nonconvex optimization problem and the choice of the number of hidden units. In support vector machines (SVMs) for classification, as introduced by Vapnik, a nonlinear decision boundary is obtained by mapping the input vector first in a nonlinear way to a high-dimensional kernel-induced feature space in which a linear large margin classifier is constructed. Practical expressions are formulated in the dual space in terms of the related kernel function, and the solution follows from a (convex) quadratic programming (QP) problem. In least-squares SVMs (LS-SVMs), the SVM problem formulation is modified by introducing a least-squares cost function and equality instead of inequality constraints, and the solution follows from a linear system in the dual space. Implicitly, the least-squares formulation corresponds to a regression formulation and is also related to kernel Fisher discriminant analysis. The least-squares regression formulation has advantages for deriving analytic expressions in a Bayesian evidence framework, in contrast to the classification formulations used, for example, in gaussian processes (GPs). The LS-SVM formulation has clear primal-dual interpretations, and without the bias term, one explicitly constructs a model that yields the same expressions as have been obtained with GPs for regression. In this article, the Bayesian evidence framework is combined with the LS-SVM classifier formulation. Starting from the feature space formulation, analytic expressions are obtained in the dual space on the different levels of Bayesian inference, while posterior class probabilities are obtained by marginalizing over the model parameters. Empirical results obtained on 10 public domain data sets show that the LS-SVM classifier designed within the Bayesian evidence framework consistently yields good generalization performances.     

Stage-based soft-transition multiple PCA modeling and on-line monitoring strategy for batch processes

For the hard-partition and misclassification problems of stage-based sub-PCA modeling method, a new STMPCA (soft-transition multiple PCA) modeling method is introduced in this article to overcome these disadvantages. The method is based on the idea that process transition can be detected by analyzing changes in the loading matrices, which reveal evolvement of the underlying process behaviours. By setting a series of multiple PCA models with time-varying covariance structures, it reflects objectively the diversity of transitional characteristics and can preferably solve the stage-transition monitoring problem in multistage batch processes. The superiority of the proposed method is illustrated by applying it to both the real three-tank system and the simulation benchmark of fed-batch penicillin fermentation process with more reliable monitoring charts. Both results of real experiment and simulation clearly demonstrate the effectiveness and feasibility of the proposed method, which detects various faults more promptly with desirable reliability.     

New coordination strategy for on-line hierarchical optimizing control of large scale industrial processes

A new coordination strategy for hierarchical optimizing control is presented, Unlike the price coordination method this approach is not a primal and dual method and its convergence behaviour does not depend on the feature of the saddle point of the problem lagrangian. Compared with the price coordination method, this approach has two advantages. Firstly, its applicability conditions are easier to satisfy and secondly, its convergence behaviour is more desirable in terms of convergence rate and insensitivity of the hessian structure of the problem. A variable augmentation technique is employed to increase the flexibility of the iterative gain selection and to improve further the convergence behaviour of the method. Optimality and convergence analysis are provided for each different version of the algorithm presented. A comparative study between different versions of the algorithm presented and a single iterative integrated system optimization and parameter estimation (ISOPE) method with global feedback is also provided using computer simulation.     

Psychopedagogical components and processes in e-learning. Lessons from an unsuccessful on-line course

In this study we are reporting the results of an external evaluation carried out on an experimental on-line course developed as part of the European project Multidimensional Approach for Multiplication of Training Environments (MAMUT) (E/99/1/61440/PI/III.3.a/CONT). The aim was to identify psychopedagogical processes that might influence the dynamics of the on-line course and to detect unexpected results. Qualitative analysis of the content of the three hundred and fifty six (356) messages written by the participants in the virtual environment was undertaken. The aspects analysed were: Adaptation to Virtual Environment, Content, Resources, Timing, Tasks, Students’ characteristics, Students’ interaction, and Students-Facilitator interaction.     

Measurement-based optimization of batch processes: Meeting terminal constraints on-line via trajectory following

The context of this paper is the use of process measurements to optimize batch processes in the presence of uncertainty. The optimal solution consists of (i) keeping certain path and terminal constraints active and (ii) driving the sensitivities to zero. In particular, the problem of meeting the active terminal constraints in each run is considered here, which is important when these constraints have a larger bearing on the cost than the sensitivities. A two-time-scale methodology is proposed, whereby the task of meeting the active terminal constraints is addressed on-line using trajectory following, while pushing the sensitivities to zero is implemented on a run-to-run basis. The proposed methodology is illustrated via the simulation of a batch distillation system.     

Edgeworth-expanded gaussian mixture density modeling

Instead of increasing the order of the Edgeworth expansion of a single gaussian kernel, we suggest using mixtures of Edgeworth-expanded gaussian kernels of moderate order. We introduce a simple closed-form solution for estimating the kernel parameters based on weighted moment matching. Furthermore, we formulate the extension to the multivariate case, which is not always feasible with algebraic density approximation procedures.     

The variational gaussian approximation revisited

The variational approximation of posterior distributions by multivariate gaussians has been much less popular in the machine learning community compared to the corresponding approximation by factorizing distributions. This is for a good reason: the gaussian approximation is in general plagued by an Omicron(N)(2) number of variational parameters to be optimized, N being the number of random variables. In this letter, we discuss the relationship between the Laplace and the variational approximation, and we show that for models with gaussian priors and factorizing likelihoods, the number of variational parameters is actually Omicron(N). The approach is applied to gaussian process regression with nongaussian likelihoods.     

Contrastive divergence in gaussian diffusions

This letter presents an analysis of the contrastive divergence (CD) learning algorithm when applied to continuous-time linear stochastic neural networks. For this case, powerful techniques exist that allow a detailed analysis of the behavior of CD. The analysis shows that CD converges to maximum likelihood solutions only when the network structure is such that it can match the first moments of the desired distribution. Otherwise, CD can converge to solutions arbitrarily different from the log-likelihood solutions, or they can even diverge. This result suggests the need to improve our theoretical understanding of the conditions under which CD is expected to be well behaved and the conditions under which it may fail. In, addition the results point to practical ideas on how to improve the performance of CD.     

An integrated approach to active model adaptation and on-line dynamic optimisation of batch processes

In the application of on-line, dynamic process optimisation, adaptive estimation of the system states and parameters is usually needed to minimise the unavoidable model-process mismatch. This work presents an integrated approach to optimal model adaptation and dynamic optimisation, with specific focus on batch processes. An active approach is proposed whereby the input variables are designed so as to maximise the information content of the data for optimal model adaptation. Then, this active adaptation method is combined with the objective of process performance to form a multi-objective optimisation problem. This integrative approach is in contrast to the traditional adaptation method, where only the process performance is considered and adaptation is passively carried out by using the data as is. Two strategies for solving the multi-objective problem are investigated: weighted average and constrained optimisation, and the latter is recommended for the ease in determining the balance between these two objectives. The proposed methodology is demonstrated on a simulated semi-batch fermentation process.     

An on-line batch span minimization and quality control strategy for batch and semi-batch processes

A novel strategy is proposed to minimize the operation time of batch and semi-batch processes. The proposed on-line strategy is based on linear regression models and employs a cascade control structure in which the primary controller calculates an optimal operation profile for the secondary controller to follow. A special feature of the proposed on-line strategy is that it conducts run-wise information feedback and achieves the attainable minimum operation time as the batch run is repeated despite model uncertainty. The performance of the proposed strategy is illustrated through simulation studies involving an exothermic batch reactor and a semi-batch reactor producing 2-acetoacetyl pyrrole.     

Recursive bayesian estimation using gaussian sums

The Bayesian recursion relations which describe the behavior of the a posteriori probability density function of the state of a time-discrete stochastic system conditioned on available measurement data cannot generally be solved in closed-form when the system is either non-linear or nongaussian. In this paper a density approximation involving convex combinations of gaussian density functions is introduced and proposed as a meaningful way of circumventing the difficulties encountered in evaluating these relations and in using the resulting densities to determine specific estimation policies. It is seen that as the number of terms in the gaussian sum increases without bound, the approximation converges uniformly to any density function in a large class. Further, any finite sum is itself a valid density function unlike many other approximations that have been investigated.     

Gradient-based adaptation of general gaussian kernels

Gradient-based optimizing of gaussian kernel functions is considered. The gradient for the adaptation of scaling and rotation of the input space is computed to achieve invariance against linear transformations. This is done by using the exponential map as a parameterization of the kernel parameter manifold. By restricting the optimization to a constant trace subspace, the kernel size can be controlled. This is, for example, useful to prevent overfitting when minimizing radius-margin generalization performance measures. The concepts are demonstrated by training hard margin support vector machines on toy data.