Sparse regression mixture modeling with the multi-kernel relevance vector machine

A regression mixture model is proposed where each mixture component is a multi-kernel version of the Relevance Vector Machine (RVM). This mixture model exploits the enhanced modeling capability of RVMs, due to their embedded sparsity enforcing properties. In order to deal with the selection problem of kernel parameters, a weighted multi-kernel scheme is employed, where the weights are estimated during training. The mixture model is trained using the maximum a posteriori approach, where the Expectation Maximization (EM) algorithm is applied offering closed form update equations for the model parameters. Moreover, an incremental learning methodology is also presented that tackles the parameter initialization problem of the EM algorithm along with a BIC-based model selection methodology to estimate the proper number of mixture components. We provide comparative experimental results using various artificial and real benchmark datasets that empirically illustrate the efficiency of the proposed mixture model.     

Sparse modeling using orthogonal forward regression with PRESS statistic and regularization

The paper introduces an efficient construction algorithm for obtaining sparse linear-in-the-weights regression models based on an approach of directly optimizing model generalization capability. This is achieved by utilizing the delete-1 cross validation concept and the associated leave-one-out test error also known as the predicted residual sums of squares (PRESS) statistic, without resorting to any other validation data set for model evaluation in the model construction process. Computational efficiency is ensured using an orthogonal forward regression, but the algorithm incrementally minimizes the PRESS statistic instead of the usual sum of the squared training errors. A local regularization method can naturally be incorporated into the model selection procedure to further enforce model sparsity. The proposed algorithm is fully automatic, and the user is not required to specify any criterion to terminate the model construction procedure. Comparisons with some of the existing state-of-art modeling methods are given, and several examples are included to demonstrate the ability of the proposed algorithm to effectively construct sparse models that generalize well.     

Sparse kernel regression technique for self-cleansing channel design

The application of a robust learning technique is inevitable in the development of a self-cleansing sediment transport model. This study addresses this problem and advocates the use of sparse kernel regression (SKR) technique to design a self-cleaning model. The SKR approach is a regression technique operating in the kernel space which also benefits from the desirable properties of a sparse solution. In order to develop a model applicable to a wide range of channel characteristics, five different experimental data sets from 14 different channels are utilized in this study. In this context, the efficacy of the SKR model is compared against the support vector regression (SVR) approach along with several other methods from the literature. According to the statistical analysis results, the SKR method is found to outperform the SVR and other regression equations. In particular, while empirical regression models fail to generate accurate results for other channel cross-section shapes and sizes, the SKR model provides promising results due to the inclusion of a channel parameter at the core of its structure and also by operating on an extensive range of experimental data. The superior efficacy of the SKR approach is also linked to its formulation in the kernel space while also benefiting from a sparse representation method to select the most useful training samples for model construction. As such, it also circumvents the requirement to evaluate irrelevant or noisy observations during the test phase of the model, and thus improving on the test phase running time.     

Integrated thermal assembly using hierarchical kernel regression method

ABSTRACT

To design thermal fit parameters such as heating temperature and insertion force, it is generally necessary to conduct structural and thermal analyses, respectively. This study aims to build an integrated thermal assembly model to design the heating temperature and insertion force parameters simultaneously. A hierarchical multiple-kernel regression is proposed to map the two parameters and the thickness of the workpiece to the contact stress. Using the regression method, we can design the thermal fit parameters efficiently. The proposed model is validated experimentally based on two shaft-bearing assemblies. The results show that the model is capable of accurately predicting the contact stress in assembly.     

Nonparametric regression models for right-censored data using Bernstein polynomials

In some applications of survival analysis with covariates, the commonly used semiparametric assumptions (e.g., proportional hazards) may turn out to be stringent and unrealistic, particularly when there is scientific background to believe that survival curves under different covariate combinations will cross during the study period. We present a new nonparametric regression model for the conditional hazard rate using a suitable sieve of Bernstein polynomials. The proposed nonparametric methodology has three key features: (i) the smooth estimator of the conditional hazard rate is shown to be a unique solution of a strictly convex optimization problem for a wide range of applications; making it computationally attractive, (ii) the model is shown to encompass a proportional hazards structure, and (iii) large sample properties including consistency and convergence rates are established under a set of mild regularity conditions. Empirical results based on several simulated data scenarios indicate that the proposed model has reasonably robust performance compared to other semiparametric models particularly when such semiparametric modeling assumptions are violated. The proposed method is further illustrated on the gastric cancer data and the Veterans Administration lung cancer data.     

Sparse high‐degree polynomials for wide‐angle lenses

Rendering with accurate camera models greatly increases realism and improves the match of synthetic imagery to real‐life footage. Photographic lenses can be simulated by ray tracing, but the performance depends on the complexity of the lens system, and some operations required for modern algorithms, such as deterministic connections, can be difficult to achieve. We generalise the approach of polynomial optics, i.e. expressing the light field transformation from the sensor to the outer pupil using a polynomial, to work with extreme wide angle (fisheye) lenses and aspherical elements. We also show how sparse polynomials can be constructed from the large space of high‐degree terms (we tested up to degree 15). We achieve this using a variant of orthogonal matching pursuit instead of a Taylor series when computing the polynomials. We show two applications: photorealistic rendering using Monte Carlo methods, where we introduce a new aperture sampling technique that is suitable for light tracing, and an interactive preview method suitable for rendering with deep images.     

Sparse multinomial logistic regression: fast algorithms and generalization bounds

Recently developed methods for learning sparse classifiers are among the state-of-the-art in supervised learning. These methods learn classifiers that incorporate weighted sums of basis functions with sparsity-promoting priors encouraging the weight estimates to be either significantly large or exactly zero. From a learning-theoretic perspective, these methods control the capacity of the learned classifier by minimizing the number of basis functions used, resulting in better generalization. This paper presents three contributions related to learning sparse classifiers. First, we introduce a true multiclass formulation based on multinomial logistic regression. Second, by combining a bound optimization approach with a component-wise update procedure, we derive fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces. To the best of our knowledge, these are the first algorithms to perform exact multinomial logistic regression with a sparsity-promoting prior. Third, we show how nontrivial generalization bounds can be derived for our classifier in the binary case. Experimental results on standard benchmark data sets attest to the accuracy, sparsity, and efficiency of the proposed methods.     

Sparse \varepsilon -tube support vector regression by active learning

A method for the sparse solution of $\varepsilon $ -tube support vector regression machines is presented. The proposed method achieves a high accuracy versus complexity ratio and allows the user to adjust the complexity of the resulting models. The sparse representation is guaranteed by limiting the number of training data points for the support vector regression method. Each training data point is selected based on its influence on the accuracy of the model using the active learning principle. The training time can be adjusted by the user by selecting how often the hyper-parameters of the algorithm are optimised. The advantages of the proposed method are illustrated on several examples. The algorithm performance is compared with the performance of several state-of-the-art algorithms on the well-known benchmark data sets. The application of the proposed algorithm for the black-box modelling of the electronic circuits is also demonstrated. The experiments clearly show that it is possible to reduce the number of support vectors and significantly improve the accuracy versus complexity ratio of $\varepsilon $ -tube support vector regression.     

Sparse kernel density construction using orthogonal forward regression with leave-one-out test score and local regularization

This paper presents an efficient construction algorithm for obtaining sparse kernel density estimates based on a regression approach that directly optimizes model generalization capability. Computational efficiency of the density construction is ensured using an orthogonal forward regression, and the algorithm incrementally minimizes the leave-one-out test score. A local regularization method is incorporated naturally into the density construction process to further enforce sparsity. An additional advantage of the proposed algorithm is that it is fully automatic and the user is not required to specify any criterion to terminate the density construction procedure. This is in contrast to an existing state-of-art kernel density estimation method using the support vector machine (SVM), where the user is required to specify some critical algorithm parameter. Several examples are included to demonstrate the ability of the proposed algorithm to effectively construct a very sparse kernel density estimate with comparable accuracy to that of the full sample optimized Parzen window density estimate. Our experimental results also demonstrate that the proposed algorithm compares favorably with the SVM method, in terms of both test accuracy and sparsity, for constructing kernel density estimates.     

基于内点法的稀疏逻辑回归财务预警模型

逻辑回归已广泛应用于财务危机建模,但是一定程度存在过拟合问题.为了避免建模出现上述问题,提出了基于L1正则化逻辑回归的财务预警模型.该模型是一种稀疏模型,能同时实现变量选择和参数估计,具有较强的鲁棒性.同时,针对L1正则化逻辑回归问题的求解,提出了一种高效的基于内点法的求解算法.结合沪深股市A股制造业上市公司进行实证分析,分析结果表明,L1正则化逻辑回归模型在预报精度、经济解释性等方面明显优于其他逻辑回归模型,并且提出的内点法与其它求解算法相比具有一定的优越性.     

Sparse seemingly unrelated regression modelling: Applications in finance and econometrics

A sparse seemingly unrelated regression (SSUR) model is proposed to generate substantively relevant structures in the high-dimensional distributions of seemingly unrelated regression (SUR) model parameters. The SSUR framework includes prior specifications, posterior computations using Markov chain Monte Carlo methods, evaluations of model uncertainty, and model structure searches. Extensions of the SSUR model to dynamic models embed general structure constraints and model uncertainty in dynamic models. The models represent specific varieties of models recently developed in the growing high-dimensional sparse modelling literature. Two simulated examples illustrate the model and highlight questions regarding model uncertainty, searching, and comparison. The model is then applied to two real-world examples in macroeconomics and finance, according to which its identified structures have practical significance.     

Sparse kernel logistic regression based on L 1/2 regularization

The sparsity driven classification technologies have attracted much attention in recent years, due to their capability of providing more compressive representations and clear interpretation. Two most popular classification approaches are support vector machines (SVMs) and kernel logistic regression (KLR), each having its own advantages. The sparsification of SVM has been well studied, and many sparse versions of 2-norm SVM, such as 1-norm SVM (1-SVM), have been developed. But, the sparsification of KLR has been less studied. The existing sparsification of KLR is mainly based on L ₁ norm and L ₂ norm penalties, which leads to the sparse versions that yield solutions not so sparse as it should be. A very recent study on L _1/2 regularization theory in compressive sensing shows that L _1/2 sparse modeling can yield solutions more sparse than those of 1 norm and 2 norm, and, furthermore, the model can be efficiently solved by a simple iterative thresholding procedure. The objective function dealt with in L _1/2 regularization theory is, however, of square form, the gradient of which is linear in its variables (such an objective function is the so-called linear gradient function). In this paper, through extending the linear gradient function of L _1/2 regularization framework to the logistic function, we propose a novel sparse version of KLR, the 1/2 quasi-norm kernel logistic regression (1/2-KLR). The version integrates advantages of KLR and L _1/2 regularization, and defines an efficient implementation scheme of sparse KLR. We suggest a fast iterative thresholding algorithm for 1/2-KLR and prove its convergence. We provide a series of simulations to demonstrate that 1/2-KLR can often obtain more sparse solutions than the existing sparsity driven versions of KLR, at the same or better accuracy level. The conclusion is also true even in comparison with sparse SVMs (1-SVM and 2-SVM). We show an exclusive advantage of 1/2-KLR that the regularization parameter in the algorithm can be adaptively set whenever the sparsity (correspondingly, the number of support vectors) is given, which suggests a methodology of comparing sparsity promotion capability of different sparsity driven classifiers. As an illustration of benefits of 1/2-KLR, we give two applications of 1/2-KLR in semi-supervised learning, showing that 1/2-KLR can be successfully applied to the classification tasks in which only a few data are labeled.     

Stochastic variational hierarchical mixture of sparse Gaussian processes for regression

In this article, we propose a scalable Gaussian process (GP) regression method that combines the advantages of both global and local GP approximations through a two-layer hierarchical model using a variational inference framework. The upper layer consists of a global sparse GP to coarsely model the entire data set, whereas the lower layer comprises a mixture of sparse GP experts which exploit local information to learn a fine-grained model. A two-step variational inference algorithm is developed to learn the global GP, the GP experts and the gating network simultaneously. Stochastic optimization can be employed to allow the application of the model to large-scale problems. Experiments on a wide range of benchmark data sets demonstrate the flexibility, scalability and predictive power of the proposed method.     

Semi-supervised feature selection via hierarchical regression for web image classification

Feature selection is an important step for large-scale image data analysis, which has been proved to be difficult due to large size in both dimensions and samples. Feature selection firstly eliminates redundant and irrelevant features and then chooses a subset of features that performs as efficient as the complete set. Generally, supervised feature selection yields better performance than unsupervised feature selection because of the utilization of labeled information. However, labeled data samples are always expensive to obtain, which constraints the performance of supervised feature selection, especially for the large web image datasets. In this paper, we propose a semi-supervised feature selection algorithm that is based on a hierarchical regression model. Our contribution can be highlighted as: (1) Our algorithm utilizes a statistical approach to exploit both labeled and unlabeled data, which preserves the manifold structure of each feature type. (2) The predicted label matrix of the training data and the feature selection matrix are learned simultaneously, making the two aspects mutually benefited. Extensive experiments are performed on three large-scale image datasets. Experimental results demonstrate the better performance of our algorithm, compared with the state-of-the-art algorithms.     

A smoothed residual based goodness-of-fit statistic for logistic hierarchical regression models

We develop a goodness-of-fit measure with desirable properties for use in the hierarchical logistic regression setting. The statistic is an unweighted sum of squares (USS) of the kernel smoothed model residuals. We develop expressions for the moments of this statistic and create a standardized statistic with hypothesized asymptotic standard normal distribution under the null hypothesis that the model is correctly specified. Extensive simulation studies demonstrate satisfactory adherence to Type I error rates of the Kernel smoothed USS statistic in a variety of likely data settings. Finally, we discuss issues of bandwidth selection for using our proposed statistic in practice and illustrate its use in an example.     

Implicit polynomials, orthogonal distance regression, and the closest point on a curve

Implicit polynomials (i.e., multinomials) have a number of properties that make them attractive for modeling curves and surfaces in computer vision. The paper considers the problem of finding the best fitting implicit polynomial (or algebraic curve) to a collection of points in the plane using an orthogonal distance metric. Approximate methods for orthogonal distance regression have been shown by others to be prone to the problem of cusps in the solution and this is confirmed here. Consequently, this work focuses on exact methods for orthogonal distance regression. The most difficult and costly part of exact methods is computing the closest point on the algebraic curve to an arbitrary point in the plane. The paper considers three methods for achieving this in detail. The first is the standard Newton's method, the second is based on resultants which are making a resurgence in computer graphics, and the third is a novel technique based on successive circular approximations to the curve. It is shown that Newton's method is the quickest, but that it can fail sometimes even with a good initial guess. The successive circular approximation algorithm is not as fast, but is robust. The resultant method is the slowest of the three, but does not require an initial guess. The driving application of this work was the fitting of implicit quartics in two variables to thinned oblique ionogram traces.     

Sparse kernel regression modeling using combined locally regularized orthogonal least squares and D-optimality experimental design

The note proposes an efficient nonlinear identification algorithm by combining a locally regularized orthogonal least squares (LROLS) model selection with a D-optimality experimental design. The proposed algorithm aims to achieve maximized model robustness and sparsity via two effective and complementary approaches. The LROLS method alone is capable of producing a very parsimonious model with excellent generalization performance. The D-optimality design criterion further enhances the model efficiency and robustness. An added advantage is that the user only needs to specify a weighting for the D-optimality cost in the combined model selecting criterion and the entire model construction procedure becomes automatic. The value of this weighting does not influence the model selection procedure critically and it can be chosen with ease from a wide range of values.     

Data Mining with Sparse Grids

O (h _n ⁻¹ n ^{d −1}) instead of O(h _n ^−d ) grid points and unknowns are involved. Here d denotes the dimension of the feature space and h _n = 2 ⁻ⁿ gives the mesh size. To be precise, we suggest to use the sparse grid combination technique [42] where the classification problem is discretized and solved on a certain sequence of conventional grids with uniform mesh sizes in each coordinate direction. The sparse grid solution is then obtained from the solutions on these different grids by linear combination. In contrast to other sparse grid techniques, the combination method is simpler to use and can be parallelized in a natural and straightforward way. We describe the sparse grid combination technique for the classification problem in terms of the regularization network approach. We then give implementational details and discuss the complexity of the algorithm. It turns out that the method scales only linearly with the number of instances, i.e. the amount of data to be classified. Finally we report on the quality of the classifier built by our new method. Here we consider standard test problems from the UCI repository and problems with huge synthetical data sets in up to 9 dimensions. It turns out that our new method achieves correctness rates which are competitive to that of the best existing methods. Received April 25, 2001