Improving the approximation and convergence capabilities of projection pursuit learning

One nonparametric regression technique that has been successfully applied to high-dimensional data is projection pursuit regression (PPR). In this method, the regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors. Projection pursuit learning (PPL) proposed by Hwanget al. formulates PPR using a two-layer feedforward neural network. One of the main differences between PPR and PPL is that the smoothers in PPR are nonparametric, whereas those in PPL are based on Hermite functions of some predefined highest orderR. While the convergence property of PPR is already known, that for PPL has not been thoroughly studied. In this paper, we demonstrate that PPL networks do not have the universal approximation and strong convergence properties for any finiteR. But, by including a bias term in each linear combination of the predictor variables, PPL networks can regain these capabilities, independent of the exact choice ofR. It is also shown experimentally that this modification improves the generalization performance in regression problems, and creates smoother decision surfaces for classification problems.     

Implementing projection pursuit learning

This paper examines the implementation of projection pursuit regression (PPR) in the context of machine learning and neural networks. We propose a parametric PPR with direct training which achieves improved training speed and accuracy when compared with nonparametric PPR. Analysis and simulations are done for heuristics to choose good initial projection directions. A comparison of a projection pursuit learning network with a single hidden-layer sigmoidal neural network shows why grouping hidden units in a projection pursuit learning network is useful. Learning robot arm inverse dynamics is used as an example problem.     

The cascade-correlation learning: a projection pursuit learningperspective

Cascade-correlation (Cascor) is a popular supervised learning architecture that dynamically grows layers of hidden neurons of fixed nonlinear activations (e.g., sigmoids), so that the network topology (size, depth) can be efficiently determined. Similar to a cascade-correlation learning network (CCLN), a projection pursuit learning network (PPLN) also dynamically grows the hidden neurons. Unlike a CCLN where cascaded connections from the existing hidden units to the new candidate hidden unit are required to establish high-order nonlinearity in approximating the residual error, a PPLN approximates the high-order nonlinearity by using trainable parametric or semi-parametric nonlinear smooth activations based on minimum mean squared error criterion. An analysis is provided to show that the maximum correlation training criterion used in a CCLN tends to produce hidden units that saturate and thus makes it more suitable for classification tasks instead of regression tasks as evidenced in the simulation results. It is also observed that this critical weakness in CCLN can also potentially carry over to classification tasks, such as the two-spiral benchmark used in the original CCLN paper.     

Application of projection pursuit learning to boundary detection and deblurring in images

Projection pursuit learning networks (PPLNs) have been used in many fields of research but have not been widely used in image processing. In this paper we demonstrate how this highly promising technique may be used to connect edges and produce continuous boundaries. We also propose the application of PPLN to deblurring a degraded image when little or no a priori information about the blur is available. The PPLN was successful at developing an inverse blur filter to enhance blurry images. Theory and background information on projection pursuit regression (PPR) and PPLN are also presented.     

Hierarchical discretized pursuit nonlinear learning automata withrapid convergence and high accuracy

A new absorbing multiaction learning automaton that is epsilon-optimal is introduced. It is a hierarchical discretized pursuit nonlinear learning automaton that uses a new algorithm for positioning the actions on the leaves of the hierarchical tree. The proposed automaton achieves the highest performance (speed of convergence, central processing unit (CPU) time, and accuracy) among all the absorbing learning automata reported in the literature up to now. Extensive simulation results indicate the superiority of the proposed scheme. Furthermore, it is proved that the proposed automaton is epsilon-optimal in every stationary stochastic environment     

Combining topographic map formation with projection pursuit learning for nonparametric regression analysis

The original Self-Organizing Map (SOM) algorithm is known to perform poorly on regression problems due to the occurrence of nonfunctional mappings. Recently, we have introduced an unsupervised learning rule, called the Maximum Entropy learning Rule (MER), which performs topographic map formation without using a neighborhood function. In the present paper, MER is extended with a neighborhood function and applied to nonparametric projection pursuit regression. The extended rule, called eMER, alleviates the occurrence of nonfunctional mappings. The performance of our regression procedure is quantified and compared to other neural network-based parametric and nonparametric regression procedures.     

Time and frequency domain convergence properties in iterative learning control

The convergence properties of iterative learning control (ILC) algorithms are considered. The analysis is carried out in a framework using linear iterative systems, which enables several results from the theory of linear systems to be applied. This makes it possible to analyse both first-order and high-order ILC algorithms in both the time and frequency domains. The time and frequency domain results can also be tied together in a clear way. Results are also given for the iterationvariant case, i.e. when the dynamics of the system to be controlled or the ILC algorithm itself changes from iteration to iteration.     

投影寻踪在瓦斯涌出量预测中的应用

针对瓦斯涌出量预测时常用的瓦斯监测数据降维方法会不同程度地造成数据信息丢失、导致预测精度降低的问题,利用投影寻踪原理并结合差分进化算法将高维样本数据转化成1维投影数据,运用Matlab曲线拟合工具箱建立了一种正弦和函数瓦斯涌出量预测模型。实验结果表明,该模型具有较高的预测精度和可操作性。     

投影寻踪在火灾后混凝土性能评价中的应用

在研究高强混凝土高温后性能的基础上,通过对影响火灾后高强混凝土结构性能的主要因素进行分析,选择抗压强度损失系数、耐久性损失系数、爆裂损失系数和裂纹损失系数作为评价指标,以投影寻踪回归理论为基础,提出综合评定高温后损伤混凝土性能的方法,建立了投影寻踪综合评价模型,编制了基于MATLAB的相应程序,采用人口迁移算法寻求最优投影方向,根据投影特征指标值对高强混凝土高温后性能进行综合评价。研究表明,该方法避免了评判专家人为确定权重,评价结果客观准确、方法简单,为混凝土结构火灾损伤的诊断评估与修复加固提供科学依据。     

Establishing improved convergence properties for the adaptive learning control

The adaptive learning approach proposed in Del Vecchio, Marino, and Tomei (2003) and Liuzzo, Marino, and Tomei (2007a), which guarantees output tracking of sufficiently smooth periodic reference signals (of known period) for classes of time-invariant nonlinear systems with sufficiently smooth unstructured uncertainties, is considered. By using a slightly different analysis (and in particular less conservative bounds), we show how improved convergence properties can be established while maintaining the same learning control structures. The delayed first arithmetic mean of the Fourier series of the uncertain periodic time function f (of known period) can be used instead of the partial sums of the Fourier series of f to further improve the result: a trigonometric polynomial, whose approximation to f is almost as good as the best approximation of f, is obtained.     

Pair-perturbation influence functions of nongaussianity by projection pursuit

The most nongaussian direction to explore the clustering structure of the data is considered to be the interesting linear projection direction by applying projection pursuit. Nongaussianity is often measured by kurtosis, however, kurtosis is well known to be sensitive to influential points/outliers and the projection direction is essentially affected by unusual points. Hence in this paper we focus on developing the influence functions of projection directions to investigate the influence of abnormal observations especially on the pair-perturbation influence functions to uncover the masked unusual observations. A technique is proposed for defining and calculating influence functions for statistical functional of the multivariate distribution. A simulation study and a real data example are provided to illustrate the applications of these approaches.     

量子粒子群在投影寻踪聚类中的应用

提出基于量子粒子群的投影寻踪聚类算法,该算法将量子粒子群的全局搜索能力与投影寻踪对高维数据的降维能力相结合,有效解决了高维数据聚类计算量大效率低的问题。并将该算法应用于三种不同的测试数据,仿真实验结果表明该算法具有更好的效率,且提高了聚类效果,是解决高维聚类问题的一种有效方法。     

Robust optimal design and convergence properties analysis of iterative learning control approaches

In this paper, we address four major issues in the field of iterative learning control (ILC) theory and design. The first issue is concerned with ILC design in the presence of system interval uncertainties. Targeting at time-optimal (fastest convergence) and robustness properties concurrently, we formulate the ILC design into a min-max optimization problem and provide a systematic solution for linear-type ILC consisting of the first-order and higher-order ILC schemes. Inherently relating to the first issue, the second issue is concerned with the performance evaluation of various ILC schemes. Convergence speed is one of the most important factors in ILC. A learning performance index—Q-factor—is introduced, which provides a rigorous and quantified evaluation criterion for comparing the convergence speed of various ILC schemes. We further explore a key issue: how does the system dynamics affect the learning performance. By associating the time weighted norm with the supreme norm, we disclose the dynamics impact in ILC, which can be assessed by global uniform bound and monotonicity in iteration domain. Finally we address a rather controversial issue in ILC: can the higher-order ILC outperform the lower-order ILC in terms of convergence speed and robustness? By applying the min-max design, which is robust and optimal, and conducting rigorous analysis, we reach the conclusion that the Q-factor of ILC sequences of lower-order ILC is lower than that of higher-order ILC in terms of the time-weighted norm.     

Authority and convergence in collaborative learning

Teachers and students have established social roles, norms and conventions when they encounter Computer-Supported Collaborative Learning (CSCL) systems in the classroom. Authority, a major force in the classroom, gives certain people, objects, representations or ideas the power to affect thought and behavior and influences communication and interaction. Effective computer-supported collaborative learning requires students and teachers to change how they understand and assign authority. This paper describes two studies in which students' ideas about authority led them to converge on what they viewed as authoritative representations and styles of representation too early, and the early convergence then hindered their learning. It also describes a third study that illustrates how changes to the CSCL system CAROUSEL (Collaborative Algorithm Representations Of Undergraduates for Self-Enhanced Learning) improved this situation, encouraging students to create representations that were unique, had different styles and emphasized different aspects of algorithms. Based on this research, methods to help students avoid premature convergence during collaborative learning are suggested.     

The learning convergence of CMAC in cyclic learning

In this paper we discuss the learning convergence of the cerebellar model articulation controller (CMAC) in cyclic learning.We prove the following results.First,if the training samples are noiseless,the training algorithm converges if and only if the learning rate is chosen from (0,2).Second,when the training samples have noises,the learning algorithm will converge with a probability of one if the learning rate is dynamically decreased.Third,in the case with noises,with a small but fixed learning rate ε the mean square error of the weight sequences generated by the CMAC learning algorithm will be bounded by O(ε).Some simulation experiments are carried out to test these results.     

On convergence rate of projection neural networks

This note presents an analysis of the convergence rate for a projection neural network with application to constrained optimization and related problems. It is shown that the state trajectory of the projection neural network is exponentially convergent to its equilibrium point if the Jacobian matrix of the nonlinear mapping is positive definite, while the convergence rate is proportional to a design parameter if the Jacobian matrix is only positive semidefinite. Moreover, the convergence time is guaranteed to be finite if the design parameter is chosen to be sufficiently large. Furthermore, if a diagonal block of the Jacobian matrix is positive definite, then the corresponding partial state trajectory of the projection neural network is also exponentially convergent. Three optimization examples are used to show the convergence performance of the projection neural network.     

A projection pursuit algorithm for anomaly detection in hyperspectral imagery

The main goal of this paper is to propose an innovative technique for anomaly detection in hyperspectral imageries. This technique allows anomalies to be identified whose signatures are spectrally distinct from their surroundings, without any a priori knowledge of the target spectral signature. It is based on an one-dimensional projection pursuit with the Legendre index as the measure of interest. The index optimization is performed with a simulated annealing over a simplex in order to bypass local optima which could be sub-optimal in certain cases. It is argued that the proposed technique could be considered as seeking a projection to depart from the normal distribution, and unfolding the outliers as a consequence. The algorithm is tested with AHS and HYDICE hyperspectral imageries, where the results show the benefits of the approach in detecting a great variety of objects whose spectral signatures have sufficient deviation from the background. The technique proves to be automatic in the sense that there is no need for parameter tuning, giving meaningful results in all cases. Even objects of sub-pixel size, which cannot be made out by the human naked eye in the original image, can be detected as anomalies. Furthermore, a comparison between the proposed approach and the popular RX technique is given. The former outperforms the latter demonstrating its ability to reduce the proportion of false alarms.     

基于一次投影子空间追踪的压缩感知信号重构

为了降低信号重构算法的复杂度,实现对稀疏度未知信号的重构,提出了一种基于一次投影子空间追踪(OPSP)的信号重构方法。首先根据约束等距性质确定信号稀疏度的上下界,并将最接近上下界中值的整数作为稀疏度的估计值;然后在子空间追踪(SP)算法的框架下,去掉了迭代中观测向量在支撑集上的投影过程,降低了算法的复杂度。为了更准确地衡量算法的重构性能,提出用完整信号的重构概率作为衡量算法重构性能的指标。与传统的SP算法相比,所提算法可以重构稀疏度未知的信号,且重构时间短,重构概率高。仿真结果验证了该算法的有效性。     

From projection pursuit and CART to adaptive discriminant analysis?

While many efforts have been put into the development of nonlinear approximation theory and its applications to signal and image compression, encoding and denoising, there seems to be very few theoretical developments of adaptive discriminant representations in the area of feature extraction, selection and signal classification. In this paper, we try to advocate the idea that such developments and efforts are worthwhile, based on the theoretical study of a data-driven discriminant analysis method on a simple-yet instructive-example. We consider the problem of classifying a signal drawn from a mixture of two classes, using its projections onto low-dimensional subspaces. Unlike the linear discriminant analysis (LDA) strategy, which selects subspaces that do not depend on the observed signal, we consider an adaptive sequential selection of projections, in the spirit of nonlinear approximation and classification and regression trees (CART): at each step, the subspace is enlarged in a direction that maximizes the mutual information with the unknown class. We derive explicit characterizations of this adaptive discriminant analysis (ADA) strategy in two situations. When the two classes are Gaussian with the same covariance matrix but different means, the adaptive subspaces are actually nonadaptive and can be computed with an algorithm similar to orthonormal matching pursuit. When the classes are centered Gaussians with different covariances, the adaptive subspaces are spanned by eigen-vectors of an operator given by the covariance matrices (just as could be predicted by regular LDA), however we prove that the order of observation of the components along these eigen-vectors actually depends on the observed signal. Numerical experiments on synthetic data illustrate how data-dependent features can be used to outperform LDA on a classification task, and we discuss how our results could be applied in practice.     

Genetic algorithms and particle swarm optimization for exploratory projection pursuit

Exploratory Projection Pursuit (EPP) methods have been developed thirty years ago in the context of exploratory analysis of large data sets. These methods consist in looking for low-dimensional projections that reveal some interesting structure existing in the data set but not visible in high dimension. Each projection is associated with a real valued index which optima correspond to valuable projections. Several EPP indices have been proposed in the statistics literature but the main problem lies in their optimization. In the present paper, we propose to apply Genetic Algorithms (GA) and recent Particle Swarm Optimization (PSO) algorithm to the optimization of several projection pursuit indices. We explain how the EPP methods can be implemented in order to become an efficient and powerful tool for the statistician. We illustrate our proposal on several simulated and real data sets.