Improving the approximation and convergence capabilities of projection pursuit learning

One nonparametric regression technique that has been successfully applied to high-dimensional data is projection pursuit regression (PPR). In this method, the regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors. Projection pursuit learning (PPL) proposed by Hwanget al. formulates PPR using a two-layer feedforward neural network. One of the main differences between PPR and PPL is that the smoothers in PPR are nonparametric, whereas those in PPL are based on Hermite functions of some predefined highest orderR. While the convergence property of PPR is already known, that for PPL has not been thoroughly studied. In this paper, we demonstrate that PPL networks do not have the universal approximation and strong convergence properties for any finiteR. But, by including a bias term in each linear combination of the predictor variables, PPL networks can regain these capabilities, independent of the exact choice ofR. It is also shown experimentally that this modification improves the generalization performance in regression problems, and creates smoother decision surfaces for classification problems.     

Implementing projection pursuit learning

This paper examines the implementation of projection pursuit regression (PPR) in the context of machine learning and neural networks. We propose a parametric PPR with direct training which achieves improved training speed and accuracy when compared with nonparametric PPR. Analysis and simulations are done for heuristics to choose good initial projection directions. A comparison of a projection pursuit learning network with a single hidden-layer sigmoidal neural network shows why grouping hidden units in a projection pursuit learning network is useful. Learning robot arm inverse dynamics is used as an example problem.     

The cascade-correlation learning: a projection pursuit learningperspective

Cascade-correlation (Cascor) is a popular supervised learning architecture that dynamically grows layers of hidden neurons of fixed nonlinear activations (e.g., sigmoids), so that the network topology (size, depth) can be efficiently determined. Similar to a cascade-correlation learning network (CCLN), a projection pursuit learning network (PPLN) also dynamically grows the hidden neurons. Unlike a CCLN where cascaded connections from the existing hidden units to the new candidate hidden unit are required to establish high-order nonlinearity in approximating the residual error, a PPLN approximates the high-order nonlinearity by using trainable parametric or semi-parametric nonlinear smooth activations based on minimum mean squared error criterion. An analysis is provided to show that the maximum correlation training criterion used in a CCLN tends to produce hidden units that saturate and thus makes it more suitable for classification tasks instead of regression tasks as evidenced in the simulation results. It is also observed that this critical weakness in CCLN can also potentially carry over to classification tasks, such as the two-spiral benchmark used in the original CCLN paper.     

Application of projection pursuit learning to boundary detection and deblurring in images

Projection pursuit learning networks (PPLNs) have been used in many fields of research but have not been widely used in image processing. In this paper we demonstrate how this highly promising technique may be used to connect edges and produce continuous boundaries. We also propose the application of PPLN to deblurring a degraded image when little or no a priori information about the blur is available. The PPLN was successful at developing an inverse blur filter to enhance blurry images. Theory and background information on projection pursuit regression (PPR) and PPLN are also presented.     

Hierarchical discretized pursuit nonlinear learning automata withrapid convergence and high accuracy

A new absorbing multiaction learning automaton that is epsilon-optimal is introduced. It is a hierarchical discretized pursuit nonlinear learning automaton that uses a new algorithm for positioning the actions on the leaves of the hierarchical tree. The proposed automaton achieves the highest performance (speed of convergence, central processing unit (CPU) time, and accuracy) among all the absorbing learning automata reported in the literature up to now. Extensive simulation results indicate the superiority of the proposed scheme. Furthermore, it is proved that the proposed automaton is epsilon-optimal in every stationary stochastic environment     

Combining topographic map formation with projection pursuit learning for nonparametric regression analysis

The original Self-Organizing Map (SOM) algorithm is known to perform poorly on regression problems due to the occurrence of nonfunctional mappings. Recently, we have introduced an unsupervised learning rule, called the Maximum Entropy learning Rule (MER), which performs topographic map formation without using a neighborhood function. In the present paper, MER is extended with a neighborhood function and applied to nonparametric projection pursuit regression. The extended rule, called eMER, alleviates the occurrence of nonfunctional mappings. The performance of our regression procedure is quantified and compared to other neural network-based parametric and nonparametric regression procedures.     

Time and frequency domain convergence properties in iterative learning control

The convergence properties of iterative learning control (ILC) algorithms are considered. The analysis is carried out in a framework using linear iterative systems, which enables several results from the theory of linear systems to be applied. This makes it possible to analyse both first-order and high-order ILC algorithms in both the time and frequency domains. The time and frequency domain results can also be tied together in a clear way. Results are also given for the iterationvariant case, i.e. when the dynamics of the system to be controlled or the ILC algorithm itself changes from iteration to iteration.     

投影寻踪在瓦斯涌出量预测中的应用

针对瓦斯涌出量预测时常用的瓦斯监测数据降维方法会不同程度地造成数据信息丢失、导致预测精度降低的问题,利用投影寻踪原理并结合差分进化算法将高维样本数据转化成1维投影数据,运用Matlab曲线拟合工具箱建立了一种正弦和函数瓦斯涌出量预测模型。实验结果表明,该模型具有较高的预测精度和可操作性。     

Establishing improved convergence properties for the adaptive learning control

The adaptive learning approach proposed in Del Vecchio, Marino, and Tomei (2003) and Liuzzo, Marino, and Tomei (2007a), which guarantees output tracking of sufficiently smooth periodic reference signals (of known period) for classes of time-invariant nonlinear systems with sufficiently smooth unstructured uncertainties, is considered. By using a slightly different analysis (and in particular less conservative bounds), we show how improved convergence properties can be established while maintaining the same learning control structures. The delayed first arithmetic mean of the Fourier series of the uncertain periodic time function f (of known period) can be used instead of the partial sums of the Fourier series of f to further improve the result: a trigonometric polynomial, whose approximation to f is almost as good as the best approximation of f, is obtained.     

Pair-perturbation influence functions of nongaussianity by projection pursuit

The most nongaussian direction to explore the clustering structure of the data is considered to be the interesting linear projection direction by applying projection pursuit. Nongaussianity is often measured by kurtosis, however, kurtosis is well known to be sensitive to influential points/outliers and the projection direction is essentially affected by unusual points. Hence in this paper we focus on developing the influence functions of projection directions to investigate the influence of abnormal observations especially on the pair-perturbation influence functions to uncover the masked unusual observations. A technique is proposed for defining and calculating influence functions for statistical functional of the multivariate distribution. A simulation study and a real data example are provided to illustrate the applications of these approaches.     

Robust optimal design and convergence properties analysis of iterative learning control approaches

In this paper, we address four major issues in the field of iterative learning control (ILC) theory and design. The first issue is concerned with ILC design in the presence of system interval uncertainties. Targeting at time-optimal (fastest convergence) and robustness properties concurrently, we formulate the ILC design into a min-max optimization problem and provide a systematic solution for linear-type ILC consisting of the first-order and higher-order ILC schemes. Inherently relating to the first issue, the second issue is concerned with the performance evaluation of various ILC schemes. Convergence speed is one of the most important factors in ILC. A learning performance index—Q-factor—is introduced, which provides a rigorous and quantified evaluation criterion for comparing the convergence speed of various ILC schemes. We further explore a key issue: how does the system dynamics affect the learning performance. By associating the time weighted norm with the supreme norm, we disclose the dynamics impact in ILC, which can be assessed by global uniform bound and monotonicity in iteration domain. Finally we address a rather controversial issue in ILC: can the higher-order ILC outperform the lower-order ILC in terms of convergence speed and robustness? By applying the min-max design, which is robust and optimal, and conducting rigorous analysis, we reach the conclusion that the Q-factor of ILC sequences of lower-order ILC is lower than that of higher-order ILC in terms of the time-weighted norm.     

Authority and convergence in collaborative learning

Teachers and students have established social roles, norms and conventions when they encounter Computer-Supported Collaborative Learning (CSCL) systems in the classroom. Authority, a major force in the classroom, gives certain people, objects, representations or ideas the power to affect thought and behavior and influences communication and interaction. Effective computer-supported collaborative learning requires students and teachers to change how they understand and assign authority. This paper describes two studies in which students' ideas about authority led them to converge on what they viewed as authoritative representations and styles of representation too early, and the early convergence then hindered their learning. It also describes a third study that illustrates how changes to the CSCL system CAROUSEL (Collaborative Algorithm Representations Of Undergraduates for Self-Enhanced Learning) improved this situation, encouraging students to create representations that were unique, had different styles and emphasized different aspects of algorithms. Based on this research, methods to help students avoid premature convergence during collaborative learning are suggested.     

On convergence rate of projection neural networks

This note presents an analysis of the convergence rate for a projection neural network with application to constrained optimization and related problems. It is shown that the state trajectory of the projection neural network is exponentially convergent to its equilibrium point if the Jacobian matrix of the nonlinear mapping is positive definite, while the convergence rate is proportional to a design parameter if the Jacobian matrix is only positive semidefinite. Moreover, the convergence time is guaranteed to be finite if the design parameter is chosen to be sufficiently large. Furthermore, if a diagonal block of the Jacobian matrix is positive definite, then the corresponding partial state trajectory of the projection neural network is also exponentially convergent. Three optimization examples are used to show the convergence performance of the projection neural network.     

Genetic algorithms and particle swarm optimization for exploratory projection pursuit

Exploratory Projection Pursuit (EPP) methods have been developed thirty years ago in the context of exploratory analysis of large data sets. These methods consist in looking for low-dimensional projections that reveal some interesting structure existing in the data set but not visible in high dimension. Each projection is associated with a real valued index which optima correspond to valuable projections. Several EPP indices have been proposed in the statistics literature but the main problem lies in their optimization. In the present paper, we propose to apply Genetic Algorithms (GA) and recent Particle Swarm Optimization (PSO) algorithm to the optimization of several projection pursuit indices. We explain how the EPP methods can be implemented in order to become an efficient and powerful tool for the statistician. We illustrate our proposal on several simulated and real data sets.     

From projection pursuit and CART to adaptive discriminant analysis?

While many efforts have been put into the development of nonlinear approximation theory and its applications to signal and image compression, encoding and denoising, there seems to be very few theoretical developments of adaptive discriminant representations in the area of feature extraction, selection and signal classification. In this paper, we try to advocate the idea that such developments and efforts are worthwhile, based on the theoretical study of a data-driven discriminant analysis method on a simple-yet instructive-example. We consider the problem of classifying a signal drawn from a mixture of two classes, using its projections onto low-dimensional subspaces. Unlike the linear discriminant analysis (LDA) strategy, which selects subspaces that do not depend on the observed signal, we consider an adaptive sequential selection of projections, in the spirit of nonlinear approximation and classification and regression trees (CART): at each step, the subspace is enlarged in a direction that maximizes the mutual information with the unknown class. We derive explicit characterizations of this adaptive discriminant analysis (ADA) strategy in two situations. When the two classes are Gaussian with the same covariance matrix but different means, the adaptive subspaces are actually nonadaptive and can be computed with an algorithm similar to orthonormal matching pursuit. When the classes are centered Gaussians with different covariances, the adaptive subspaces are spanned by eigen-vectors of an operator given by the covariance matrices (just as could be predicted by regular LDA), however we prove that the order of observation of the components along these eigen-vectors actually depends on the observed signal. Numerical experiments on synthetic data illustrate how data-dependent features can be used to outperform LDA on a classification task, and we discuss how our results could be applied in practice.     

A projection pursuit algorithm for anomaly detection in hyperspectral imagery

The main goal of this paper is to propose an innovative technique for anomaly detection in hyperspectral imageries. This technique allows anomalies to be identified whose signatures are spectrally distinct from their surroundings, without any a priori knowledge of the target spectral signature. It is based on an one-dimensional projection pursuit with the Legendre index as the measure of interest. The index optimization is performed with a simulated annealing over a simplex in order to bypass local optima which could be sub-optimal in certain cases. It is argued that the proposed technique could be considered as seeking a projection to depart from the normal distribution, and unfolding the outliers as a consequence. The algorithm is tested with AHS and HYDICE hyperspectral imageries, where the results show the benefits of the approach in detecting a great variety of objects whose spectral signatures have sufficient deviation from the background. The technique proves to be automatic in the sense that there is no need for parameter tuning, giving meaningful results in all cases. Even objects of sub-pixel size, which cannot be made out by the human naked eye in the original image, can be detected as anomalies. Furthermore, a comparison between the proposed approach and the popular RX technique is given. The former outperforms the latter demonstrating its ability to reduce the proportion of false alarms.     

On projection methods,convergence and robust formulations in topology optimization

Mesh convergence and manufacturability of topology optimized designs have previously mainly been assured using density or sensitivity based filtering techniques. The drawback of these techniques has been gray transition regions between solid and void parts, but this problem has recently been alleviated using various projection methods. In this paper we show that simple projection methods do not ensure local mesh-convergence and propose a modified robust topology optimization formulation based on erosion, intermediate and dilation projections that ensures both global and local mesh-convergence.     

Generalized pursuit learning schemes: new families of continuous and discretized learning automata

The fastest learning automata (LA) algorithms currently available fall in the family of estimator algorithms introduced by Thathachar and Sastry (1986). The pioneering work of these authors was the pursuit algorithm, which pursues only the current estimated optimal action. If this action is not the one with the minimum penalty probability, this algorithm pursues a wrong action. In this paper, we argue that a pursuit scheme that generalizes the traditional pursuit algorithm by pursuing all the actions with higher reward estimates than the chosen action, minimizes the probability of pursuing a wrong action, and is a faster converging scheme. To attest this, we present two new generalized pursuit algorithms (GPAs) and also present a quantitative comparison of their performance against the existing pursuit algorithms. Empirically, the algorithms proposed here are among the fastest reported LA to date.     

On convergence conditions of an extended projection neural network

The output trajectory convergence of an extended projection neural network was developed under the positive definiteness condition of the Jacobian matrix of nonlinear mapping. This note offers several new convergence results. The state trajectory convergence and the output trajectory convergence of the extended projection neural network are obtained under the positive semidefiniteness condition of the Jacobian matrix. Comparison and illustrative examples demonstrate applied significance of these new results.     

Information fusion between short term learning and long term learning in content based image retrieval systems

Content based image retrieval (CBIR) systems could provide more precise results by taking the user’s feedbacks into account. Two types of the relevance feedback learning paradigms are short term learning (STL) and long term learning (LTL). By using both STL and LTL, a collaborative CBIR system is proposed in this paper. The proposed system introduced three fusion methods: including fusion in retrieved images, fusion in ranks, and fusion in similarities to make cooperation between STL and LTL. The proposed fusion methods are examined in a CBIR system equipped with a proposed statistical semantic clustering (SSC) method of LTL. The SSC method works based on the concept of semantic categories of the images by clustering techniques and constructing a relevancy matrix between images and semantic categories. The results of the SSC method with the suggested fusion methods are compared with two state-of-the-art LTL methods, namely virtual feature based method and dynamic semantic clustering. Comparative results confirm the efficiency of the proposed method. Furthermore, experimental results demonstrate that for a unique LTL method, various fusion methods lead to different results.