Use of bias term in projection pursuit learning improvesapproximation and convergence properties

In a regression problem, one is given a multidimensional random vector X, the components of which are called predictor variables, and a random variable, Y, called response. A regression surface describes a general relationship between X and Y. A nonparametric regression technique that has been successfully applied to high-dimensional data is projection pursuit regression (PPR). The regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors. Projection pursuit learning (PPL) formulates PPR using a 2-layer feedforward neural network. The smoothers in PPR are nonparametric, whereas those in PPL are based on Hermite functions of some predefined highest order R. We demonstrate that PPL networks in the original form do not have the universal approximation property for any finite R, and thus cannot converge to the desired function even with an arbitrarily large number of hidden units. But, by including a bias term in each linear projection of the predictor variables, PPL networks can regain these capabilities, independent of the exact choice of R. Experimentally, it is shown in this paper that this modification increases the rate of convergence with respect to the number of hidden units, improves the generalization performance, and makes it less sensitive to the setting of R. Finally, we apply PPL to chaotic time series prediction, and obtain superior results compared with the cascade-correlation architecture.     

Implementing projection pursuit learning

This paper examines the implementation of projection pursuit regression (PPR) in the context of machine learning and neural networks. We propose a parametric PPR with direct training which achieves improved training speed and accuracy when compared with nonparametric PPR. Analysis and simulations are done for heuristics to choose good initial projection directions. A comparison of a projection pursuit learning network with a single hidden-layer sigmoidal neural network shows why grouping hidden units in a projection pursuit learning network is useful. Learning robot arm inverse dynamics is used as an example problem.     

遥感影像投影寻踪回归分类模型

将投影寻踪回归分析技术引入遥感影像分类中,详尽叙述遥感影像投影寻踪回归分类模型的建立和实现过程。将广州地区的TM影像用于分类实验,并用混合蛙跳算法来优化投影寻踪回归分类模型中的参数矩阵,取得了较为理想的分类效果。此外,还进一步分析了投影中心的设定、调整以及优化算法和岭函数个数对投影寻踪回归模型分类精度的影响。实验结果表明,该模型易于优化实现,稳定性强,模型中岭函数的个数对投影寻踪回归模型的分类精度没有显著影响。     

车辆跟驰投影寻踪回归模型

车辆跟驰模型是微观交通仿真的一个基本模型,基于非参数回归算法的跟车模型较好地解决了以往模型存在的典型问题,但随着样本维数增加,容易出现“维数祸根”现象。提出一种基于投影寻踪回归（PPR）技术的车辆跟驰模型,解决了“维数祸根”和高维数据间的非正态、非线性问题。PPR建模不需要对数据结构作任何假定,而只通过直接审视和分析数据进行建模,因此,该方法能充分地发掘数据中存在的信息,建立的模型符合客观实际,精度较高。经过实测数据验证,该算法用于车辆跟驰模型的研究是可行的。     

Simulation-based two-stage estimation for multiple nonparametric regression

To reduce the curse of dimensionality arising from nonparametric estimation procedures for multiple nonparametric regression, in this paper we suggest a simulation-based two-stage estimation. We first introduce a simulation-based method to decompose the multiple nonparametric regression into two parts. The first part can be estimated with the parametric convergence rate and the second part is small enough so that it can be approximated by orthogonal basis functions with a small trade-off parameter. Then the linear combination of the first and second step estimators results in a two-stage estimator for the multiple regression function. Our method does not need any specified structural assumption on the regression function and it is proved that the newly proposed estimation is always consistent even if the trade-off parameter is designed to be small. Thus when the common nonparametric estimator such as local linear smoothing collapses because of the curse of dimensionality, our estimator still works well.     

Neural-network design for small training sets of high dimension

We introduce a statistically based methodology for the design of neural networks when the dimension d of the network input is comparable to the size n of the training set. If one proceeds straightforwardly, then one is committed to a network of complexity exceeding n. The result will be good performance on the training set but poor generalization performance when the network is presented with new data. To avoid this we need to select carefully the network architecture, including control over the input variables. Our approach to selecting a network architecture first selects a subset of input variables (features) using the nonparametric statistical process of difference-based variance estimation and then selects a simple network architecture using projection pursuit regression (PPR) ideas combined with the statistical idea of slicing inverse regression (SIR). The resulting network, which is then retrained without regard to the PPR/SIR determined parameters, is one of moderate complexity (number of parameters significantly less than n) whose performance on the training set can be expected to generalize well. The application of this methodology is illustrated in detail in the context of short-term forecasting of the demand for electric power from an electric utility.     

Quantile regression neural networks: Implementation in R and application to precipitation downscaling

The qrnn package for R implements the quantile regression neural network, which is an artificial neural network extension of linear quantile regression. The model formulation follows from previous work on the estimation of censored regression quantiles. The result is a nonparametric, nonlinear model suitable for making probabilistic predictions of mixed discrete-continuous variables like precipitation amounts, wind speeds, or pollutant concentrations, as well as continuous variables. A differentiable approximation to the quantile regression error function is adopted so that gradient-based optimization algorithms can be used to estimate model parameters. Weight penalty and bootstrap aggregation methods are used to avoid overfitting. For convenience, functions for quantile-based probability density, cumulative distribution, and inverse cumulative distribution functions are also provided. Package functions are demonstrated on a simple precipitation downscaling task.     

Regularization Methods for Blind Deconvolution and Blind Source Separation Problems

This paper is devoted to blind deconvolution and blind separation problems. Blind deconvolution is the identification of a point spread function and an input signal from an observation of their convolution. Blind source separation is the recovery of a vector of input signals from a vector of observed signals, which are mixed by a linear (unknown) operator. We show that both problems are paradigms of nonlinear ill-posed problems. Consequently, regularization techniques have to be used for stable numerical reconstructions. In this paper we develop a rigorous convergence analysis for regularization techniques for the solution of blind deconvolution and blind separation problems. Convergence of regularized point spread functions and signals to a solution is established and a convergence rate result in dependence of the noise level is presented. Moreover, we prove convergence of the alternating minimization algorithm for the numerical solution of regularized blind deconvolution problems and present some numerical examples. Moreover, we show that many neural network approaches for blind inversion can be considered in the framework of regularization theory. Date received: August 17, 1999. Date revised: September 1, 2000.     

Application of projection pursuit learning to boundary detection and deblurring in images

Projection pursuit learning networks (PPLNs) have been used in many fields of research but have not been widely used in image processing. In this paper we demonstrate how this highly promising technique may be used to connect edges and produce continuous boundaries. We also propose the application of PPLN to deblurring a degraded image when little or no a priori information about the blur is available. The PPLN was successful at developing an inverse blur filter to enhance blurry images. Theory and background information on projection pursuit regression (PPR) and PPLN are also presented.     

Continuous-time Wiener system identification

A continuous-time Wiener system is identified. The system consists of a linear dynamic subsystem and a memoryless nonlinear one connected in a cascade. The input signal is a stationary white Gaussian random process. The system is disturbed by stationary white random Gaussian noise. Both subsystems are identified from input-output observations taken at the input and output of the whole system. The a priori information is very small and, therefore, resulting identification problems are nonparametric. The impulse impulse of the linear part is recovered by a correlation method, while the nonlinear characteristic is estimated with the help of the nonparametric kernel regression method. The authors prove convergence of the proposed identification algorithms and examine their convergence rates     

Regression modeling in back-propagation and projection pursuitlearning

We study and compare two types of connectionist learning methods for model-free regression problems: 1) the backpropagation learning (BPL); and 2) the projection pursuit learning (PPL) emerged in recent years in the statistical estimation literature. Both the BPL and the PPL are based on projections of the data in directions determined from interconnection weights. However, unlike the use of fixed nonlinear activations (usually sigmoidal) for the hidden neurons in BPL, the PPL systematically approximates the unknown nonlinear activations. Moreover, the BPL estimates all the weights simultaneously at each iteration, while the PPL estimates the weights cyclically (neuron-by-neuron and layer-by-layer) at each iteration. Although the BPL and the PPL have comparable training speed when based on a Gauss-Newton optimization algorithm, the PPL proves more parsimonious in that the PPL requires a fewer hidden neurons to approximate the true function. To further improve the statistical performance of the PPL, an orthogonal polynomial approximation is used in place of the supersmoother method originally proposed for nonlinear activation approximation in the PPL.     

A Regression Model for Fuzzy Initial Data

A fuzzy combined linear regression model for fuzzy initial data, which are tolerant (L-R)-numbers with constraints on the functions L and R, is designed. The model is called combined since it is a combination of two regression models—a fuzzy model and a classical model. Its coefficients are determined as unimodal (L-R)-numbers. The solution method consists in determining weighted intervals for the tolerant (L-R)-numbers and then applying of the least-squares method.     

Nonparametric predictive distributions based on conformal prediction

This paper applies conformal prediction to derive predictive distributions that are valid under a nonparametric assumption. Namely, we introduce and explore predictive distribution functions that always satisfy a natural property of validity in terms of guaranteed coverage for IID observations. The focus is on a prediction algorithm that we call the Least Squares Prediction Machine (LSPM). The LSPM generalizes the classical Dempster–Hill predictive distributions to nonparametric regression problems. If the standard parametric assumptions for Least Squares linear regression hold, the LSPM is as efficient as the Dempster–Hill procedure, in a natural sense. And if those parametric assumptions fail, the LSPM is still valid, provided the observations are IID.

     

A New Rotated Nonconforming Quadrilateral Element

In this paper, a new nonparametric nonconforming quadrilateral finite element is introduced. This element takes the four edge mean values as the degrees of the freedom and the finite element space is a subspace of \(P_{2}\). Different from the other nonparametric elements, the basis functions of this new element can be expressed explicitly without solving linear systems locally, which can be achieved by introducing a new reference quadrilateral. To evaluate the integration, a class of new quadrature formulae with only three equally weighted points on quadrilateral are constructed. Hence the stiffness matrix can be calculated by the same way with the parametric elements. Numerical results are shown to confirm the optimality of the convergence order for the second order elliptic problems and the Stokes problem.     

Piecewise linear solution paths with application to direct weight optimization

Recently, pathfollowing algorithms for parametric optimization problems with piecewise linear solution paths have been developed within the field of regularized regression. This paper presents a generalization of these algorithms to a wider class of problems. It is shown that the approach can be applied to the nonparametric system identification method, Direct Weight Optimization (DWO), and be used to enhance the computational efficiency of this method. The most important design parameter in the DWO method is a parameter (λ) controlling the bias-variance trade-off, and the use of parametric optimization with piecewise linear solution paths means that the DWO estimates can be efficiently computed for all values of λ simultaneously. This allows for designing computationally attractive adaptive bandwidth selection algorithms. One such algorithm for DWO is proposed and demonstrated in two examples.     

A hybrid inversion method for mapping leaf area index from MODIS data: experiments and application to broadleaf and needleleaf canopies

Leaf area index (LAI) is an important variable needed by various land surface process models. It has been produced operationally from the Moderate Resolution Imaging Spectroradiometer (MODIS) data using a look-up table (LUT) method, but the inversion accuracy still needs significant improvements. We propose an alternative method in this study that integrates both the radiative transfer (RT) simulation and nonparametric regression methods. Two nonparametric regression methods (i.e., the neural network [NN] and the projection pursuit regression [PPR]) were examined. An integrated database was constructed from radiative transfer simulations tuned for two broad biome categories (broadleaf and needleleaf vegetations). A new soil reflectance index (SRI) and analytically simulated leaf optical properties were used in the parameterization process. This algorithm was tested in two sites, one at Maryland, USA, a middle latitude temperate agricultural area, and the other at Canada, a boreal forest site, and LAI was accurately estimated. The derived LAI maps were also compared with those from MODIS science team and ETM+ data. The MODIS standard LAI products were found consistent with our results for broadleaf crops, needleleaf forest, and other cover types, but overestimated broadleaf forest by 2.0-3.0 due to the complex biome types.     

GA-SVR: a novel hybrid data-driven model to simulate vertical load capacity of driven piles

Piles are widely applied to substructures of various infrastructural buildings. Soil has a complex nature; thus, a variety of empirical models have been proposed for the prediction of the bearing capacity of piles. The aim of this study is to propose a novel artificial intelligent approach to predict vertical load capacity of driven piles in cohesionless soils using support vector regression (SVR) optimized by genetic algorithm (GA). To the best of our knowledge, no research has been developed the GA-SVR model to predict vertical load capacity of driven piles in different timescales as of yet, and the novelty of this study is to develop a new hybrid intelligent approach in this field. To investigate the efficacy of GA-SVR model, two other models, i.e., SVR and linear regression models, are also used for a comparative study. According to the obtained results, GA-SVR model clearly outperformed the SVR and linear regression models by achieving less root mean square error (RMSE) and higher coefficient of determination (R²). In other words, GA-SVR with RMSE of 0.017 and R² of 0.980 has higher performance than SVR with RMSE of 0.035 and R² of 0.912, and linear regression model with RMSE of 0.079 and R² of 0.625.

     

A nonmonotone scaled conjugate gradient algorithm for large-scale unconstrained optimization

This paper proposes a nonmonotone scaled conjugate gradient algorithm for solving large-scale unconstrained optimization problems, which combines the idea of scaled memoryless Broyden–Fletcher–Goldfarb–Shanno preconditioned conjugate gradient method with the nonmonotone technique. An attractive property of the proposed method is that the search direction always provides sufficient descent step at each iteration. This property is independent of the line search used. Under appropriate assumptions, the method is proven to possess global convergence for nonconvex smooth functions, and R-linear convergence for strongly convex functions. Preliminary numerical results and related comparisons show the efficiency of the proposed method in practical computation.     

A nonparametric circular–linear multivariate regression model with a rule-of-thumb bandwidth selector

The purpose of this paper is to propose a nonparametric circular–linear multivariate regression model using a kernel-weighted local linear method. The case of several linear regressors and one circular regressor is considered. We extend results on the asymptotic bias and variance of the linear multivariate variable to the case of circular–linear multivariate variable. The rule-of-thumb selector is used to establish the optimal bandwidths for the nonparametric model. The suitability of the model is judged from the coefficient of determination. One simulation experiment and one real problem concerning wind energy are used to study the power performance of the nonparametric model.     

Validation of artificial neural network techniques in the estimation of nitrogen concentration in rape using canopy hyperspectral reflectance data

A systematic comparison of two types of method for estimating the nitrogen concentration of rape is presented: the traditional statistical method based on linear regression and the emerging computationally powerful technique based on artificial neural networks (ANN). Five optimum bands were selected using stepwise regression. Comparison between the two methods was based primarily on analysis of the statistic parameters. The rms. error for the back-propagation network (BPN) was significantly lower than that for the stepwise regression method, and the T-value was higher for BPN. In particular, for the first-difference of inverse-log spectra (log 1/R)′, T-values performed with a 127.71% success rate using BPN. The results show that the neural network is more robust to training and estimating rape nitrogen concentrations using canopy hyperspectral reflectance data.