Free knot splines for logistic models and threshold selection

The logistic regression model has been in use in statistical analysis for many years. The paper introduces a spline model to remove the linear restriction on logit function. By considering knot locations as free variables, spline approximation of data is improved. The number of knots and the degree of the spline functions can still be determined by using a model selection procedure. Moreover, a knot, seen as a free parameter for a piecewise linear spline, represents a break point in the logit function which may be interpreted as a threshold value. This method is applied to a clinical trial for an in vitro fertilization program.     

Discussion of different logistic models with functional data. Application to Systemic Lupus Erythematosus

The relationship between time evolution of stress and flares in Systemic Lupus Erythematosus patients has recently been studied. Daily stress data can be considered as observations of a single variable for a subject, carried out repeatedly at different time points (functional data). In this study, we propose a functional logistic regression model with the aim of predicting the probability of lupus flare (binary response variable) from a functional predictor variable (stress level). This method differs from the classical approach, in which longitudinal data are considered as observations of different correlated variables. The estimation of this functional model may be inaccurate due to multicollinearity, and so a principal component based solution is proposed. In addition, a new interpretation is made of the parameter function of the model, which enables the relationship between the response and the predictor variables to be evaluated. Finally, the results provided by different logit approaches (functional and longitudinal) are compared, using a sample of Lupus patients.     

Fuzzy functions based ARX model and new fuzzy basis function models for nonlinear system identification

In this study, auto regressive with exogenous input (ARX) modeling is improved with fuzzy functions concept (FF-ARX). Fuzzy function with least squares estimation (FF-LSE) method has been recently developed and widely used with a small improvement with respect to least squares estimation method (LSE). FF-LSE is structured with only inputs and their membership values. This proposed model aims to increase the capability of the FF-LSE by widening the regression matrix with lagged input–output values. In addition, by using same idea, we proposed also two new fuzzy basis function models. In the first, basis of the fuzzy system and lagged input–output values are structured together in the regression matrix and named as “L-FBF”. Secondly, instead of using basis function, the membership values of the lagged input–output values are used in the regression matrix by using Gaussian membership functions, called “M-FBF”. Therefore, the power of the fuzzy basis function is also enhanced. For the corresponding models, antecedent part parameters for the input vectors are determined with fuzzy c-means (FCM) clustering algorithm. The consequent parameters of the all models are estimated with the LSE. The proposed models are utilized and compared for the identification of nonlinear benchmark problems.     

Discussion of different logistic models with functional data. Application to Systemic Lupus Erythematosus

The relationship between time evolution of stress and flares in Systemic Lupus Erythematosus patients has recently been studied. Daily stress data can be considered as observations of a single variable for a subject, carried out repeatedly at different time points (functional data). In this study, we propose a functional logistic regression model with the aim of predicting the probability of lupus flare (binary response variable) from a functional predictor variable (stress level). This method differs from the classical approach, in which longitudinal data are considered as observations of different correlated variables. The estimation of this functional model may be inaccurate due to multicollinearity, and so a principal component based solution is proposed. In addition, a new interpretation is made of the parameter function of the model, which enables the relationship between the response and the predictor variables to be evaluated. Finally, the results provided by different logit approaches (functional and longitudinal) are compared, using a sample of Lupus patients.     

Hessian optimal design for image retrieval

Recently there has been a considerable interest in active learning from the perspective of optimal experimental design (OED). OED selects the most informative samples to minimize the covariance matrix of the parameters, so that the expected prediction error of the parameters, as well as the model output, can be minimized. Most of the existing OED methods are based on either linear regression or Laplacian regularized least squares (LapRLS) models. Although LapRLS has shown a better performance than linear regression, it suffers from the fact that the solution is biased towards a constant and the lack of extrapolating power. In this paper, we propose a novel active learning algorithm called Hessian optimal design (HOD). HOD is based on the second-order Hessian energy for semi-supervised regression which overcomes the drawbacks of Laplacian based methods. Specifically, HOD selects those samples which minimize the parameter covariance matrix of the Hessian regularized regression model. The experimental results on content-based image retrieval have demonstrated the effectiveness of our proposed approach.     

Logistic regression in sealed-bid auctions with multiple rounds: Application in Korean court auction

This paper proposes a forecasting method for court auction information system using logistic regression model with heterogeneity across the multiple round. The goal is to predict whether an individual auction item in a certain round will be sold or not. A simple linear regression and the least angle regression (LARS) containing random effect terms were used to select meaningful variables for our logit model. The link function of the proposed logit model is represented by two bundles of parameters. The former part consists of the parameters whose values do not change over rounds. The latter part has parameters whose values interact with rounds. The observed data corresponding to an appraiser price as well as an intercept term reflecting local characteristics are used without any change. Data that corresponds to all the other parameters is not directly used, but transformed based on similarities between the original item and the surrounding auction items being recommended by the court auction experts. We tested the Bayesian logistic regression by establishing different priors: Dunson’s prior, Gelman’s prior and Ansari’s prior. Dunson’s prior was found to perform the best. Little significant difference was found between the results of the other two priors. These findings indicate that logistic regression taking the heterogeneity of multi-round into account performs better than a one-layered neural network over all time periods.     

基于迁移子空间学习的偏最小二乘回归软测量方法

针对流程工业中工况改变易导致当前样本与历史样本分布失配,传统软测量模型失准的问题,考虑工业数据时序性、动态性以及存在过程漂移等特性对建模的影响,提出一种基于迁移子空间学习的偏最小二乘回归软测量方法.首先,回归框架采用非线性迭代偏最小二乘方法,对其求解映射向量的目标函数施加基于子空间重构的域适应正则项,映射过程中保证当前工况中每个样本能够被历史工况样本线性重构.在此基础上对重构矩阵施加低秩稀疏约束,保持数据结构的同时使重构矩阵具备块状结构以应对过程漂移特性.将所提出方法在1个数值案例和3个不同的多工况数据集中进行实验,并与现有域适应回归方法进行对比分析.实验表明,所提出方法能够有效提高模型在跨工况条件下的预测精度,减少工况间数据分布差异对模型性能的影响.     

一种基于容错的感知数据回归模型研究

传感器主要用于对外部环境进行监测,然而当传感器发生故障时监测结果会出现误差。为了提高传感器发生故障时系统的容错能力,提出了一种容错的感知数据回归模型。首先,对最小二乘和岭回归两种线性回归模型进行分析,并分析了线性回归模型的相关统计量;然后,分析了部分传感器发生故障时系统的相关统计量,并以此为基础分析了协变量矩阵的上下界;最后,依据协变量矩阵定义了故障指标,并将优化模型转化为同时最小化故障指标和均方误差的问题。实验表明,提出的容错回归模型与传统的最小二乘法和岭回归方法相比具有更小的预测误差,因而当传感器发生故障时所提模型具有更好的健壮性。     

Shrinkage estimation in general linear models

We propose a James-Stein-type shrinkage estimator for the parameter vector in a general linear model when it is suspected that some of the parameters may be restricted to a subspace. The James-Stein estimator is shown to demonstrate asymptotically superior risk performance relative to the conventional least squares estimator under quadratic loss. An extensive simulation study based on a multiple linear regression model and a logistic regression model further demonstrates the improved performance of this James-Stein estimator in finite samples. The application of this new estimator is illustrated using Ontario newborn infants data spanning four fiscal years.     

Model selection for primal SVM

This paper introduces two types of nonsmooth optimization methods for selecting model hyperparameters in primal SVM models based on cross-validation. Unlike common grid search approaches for model selection, these approaches are scalable both in the number of hyperparameters and number of data points. Taking inspiration from linear-time primal SVM algorithms, scalability in model selection is achieved by directly working with the primal variables without introducing any dual variables. The proposed implicit primal gradient descent (ImpGrad) method can utilize existing SVM solvers. Unlike prior methods for gradient descent in hyperparameters space, all work is done in the primal space so no inversion of the kernel matrix is required. The proposed explicit penalized bilevel programming (PBP) approach optimizes both the hyperparameters and parameters simultaneously. It solves the original cross-validation problem by solving a series of least squares regression problems with simple constraints in both the hyperparameter and parameter space. Computational results on least squares support vector regression problems with multiple hyperparameters establish that both the implicit and explicit methods perform quite well in terms of generalization and computational time. These methods are directly applicable to other learning tasks with differentiable loss functions and regularization functions. Both the implicit and explicit algorithms investigated represent powerful new approaches to solving large bilevel programs involving nonsmooth loss functions.     

基于Householder变换的贪婪正交最小二乘辨识算法

针对含有未知时滞的多输入受控自回归系统模型的时滞与参数辨识问题,基于Householder变换探讨一种贪婪正交最小二乘辨识算法.首先,由于各输入通道的时滞未知,通过设置输入数据回归长度对系统模型进行过参数化,得到一个含有稀疏参数向量的高维辨识模型;其次,为了避免最小二乘算法中对高维协方差矩阵的求逆运算,利用Householder变换对信息矩阵进行正交分解,推导基于Householder变换的正交最小二乘算法;然后,为了提高辨识效率,降低辨识成本,推导基于Householder变换的贪婪准则,进而得到基于Householder变换的贪婪正交最小二乘辨识算法,该算法能够在少量采样数据的条件下获得稀疏参数向量的估计值;最后,根据估计的稀疏参数向量的结构得到系统时滞估计.仿真结果表明了所提出算法的有效性.     

Stepwise-then-intelligent algorithm (STIA) for optimizing remotely sensed image rectification

ABSTRACT

To address the problems of parameter selection and accuracy optimization of models in image rectification, this article first proposes a novel stepwise-then-intelligent algorithm (STIA) for image rectification optimization, which includes the following steps. First, stepwise regression is suggested to simultaneously solve the over-parameterization problem and select the optimum parameters of the polynomial model and rational function model according to different terrains. Second, intelligent algorithms, e.g. the genetic algorithm (GA) and particle swarm optimization (PSO), are proposed to search for better results based on an innovative search range determined by the uncertainty propagation and 3-sigma rule. The experimental results show that the proposed STIA can achieve higher accuracy than conventional methods; and in most cases, the PSO algorithm used in STIA is superior to the GA used in STIA in measures of time and accuracy. Moreover, stepwise-then-PSO algorithm exhibits the best performance of all compared methods, including least squares, stepwise regression, total least squares and partial least squares.     

An unbiased LSSVM model for classification and regression

Aiming at the common support vector machine’s biased disadvantage and computational complexity, an unbiased least squares support vector machine (LSSVM) model is proposed in this paper. The model eliminates the bias item of LSSVM by improving the form of structure risk, then the unbiased least squares support vector classifier and the unbiased least squares support vector regression are deduced. Based on this model, we design a new learning algorithm using Cholesky factorization according to the characteristic of kernel function matrix, in this way the calculation of Lagrangian multipliers is greatly simplified. Several experiments on diffenert datasets are carried out, including the common datasets classification, synthetic aperture radar image automatic target recognition and chaotic time series prediction. The experimental results of correct recognition rate and the fitting precision testify that the unbiased LSSVM model has good universal ability and fitting accuracy, better generalization capability and stability, and have a great improvement in learning speed.     

Hierarchical Newton and least squares iterative estimation algorithm for dynamic systems by transfer functions based on the impulse responses

This paper develops a parameter estimation algorithm for linear continuous-time systems based on the hierarchical principle and the parameter decomposition strategy. Although the linear continuous-time system is a linear system, its output response is a highly nonlinear function with respect to the system parameters. In order to propose a direct estimation algorithm, a criterion function is constructed between the response output and the observation output by means of the discrete sampled data. Then a scheme by combining the Newton iteration and the least squares iteration is builded to minimise the criterion function and derive the parameter estimation algorithm. In light of the different features between the system parameters and the output function, two sub-algorithms are derived by using the parameter decomposition. In order to remove the associate terms between the two sub-algorithms, a Newton and least squares iterative algorithm is deduced to identify system parameters. Compared with the Newton iterative estimation algorithm without the parameter decomposition, the complexity of the hierarchical Newton and least squares iterative estimation algorithm is reduced because the dimension of the Hessian matrix is lessened after the parameter decomposition. The experimental results show that the proposed algorithm has good performance.     

基于核偏最小二乘的锌层重量预测模型

为了给带钢热镀锌生产的质量控制提供必要的决策支持和分析手段,针对气刀对锌层重量的控制工艺,提出了基于核偏最小二乘回归的锌层重量预测模型。利用核函数将低维空间的非线性回归转化为高维空间的线性回归,克服了实际生产工艺中非线性因素对预测模型的不利影响。应用鞍山钢铁集团公司带钢热镀锌的生产实际数据进行验证,结果表明,基于核偏最小二乘的锌层重量预测方法与线性偏最小二乘、BP神经网络等方法相比,具有更好的预测精度。     

Fuzzy linear regression model based on fuzzy scalar product

The new concept and method of imposing imprecise (fuzzy) input and output data upon the conventional linear regression model is proposed in this paper. We introduce the fuzzy scalar (inner) product to formulate the fuzzy linear regression model. In order to invoke the conventional approach of linear regression analysis for real-valued data, we transact the α-level linear regression models of the fuzzy linear regression model. We construct the membership functions of fuzzy least squares estimators via the form of “Resolution Identity” which is a well-known formula in fuzzy sets theory. In order to obtain the membership value of any given least squares estimate taken from the fuzzy least squares estimator, we transform the original problem into the optimization problems. We also provide two computational procedures to solve the optimization problems.     

稀疏最小二乘支持向量机及其应用研究

提出一种构造稀疏化最小二乘支持向量机的方法．该方法首先通过斯密特正交化法对核矩阵进 行简约,得到核矩阵的基向量组;再利用核偏最小二乘方法对最小二乘支持向量机进行回归计算,从而使最 小二乘向量机具有一定稀疏性．基于稀疏最小二乘向量机建立了非线性动态预测模型,对铜转炉造渣期吹炼 时间进行滚动预测．仿真结果表明,基于核偏最小二乘辨识的稀疏最小二乘支持向量机具有计算效率高、预 测精度好的特点．     

基于- 范数约束的LSSVR 多核学习算法

针对核函数选择对最小二乘支持向量机回归模型泛化性的影响, 提出一种新的基于????- 范数约束的最小二乘支持向量机多核学习算法. 该算法提供了两种求解方法, 均通过两重循环进行求解, 外循环用于更新核函数的权值, 内循环用于求解最小二乘支持向量机的拉格朗日乘数, 充分利用该多核学习算法, 有效提高了最小二乘支持向量机的泛化能力, 而且对惩罚参数的选择具有较强的鲁棒性. 基于单变量和多变量函数的仿真实验表明了所提出算法的有效性.

     

A data mining approach to dynamic multiple responses in Taguchi experimental design

To simultaneously optimize the parameter robust design of dynamic multiple responses is difficult due to product complexity; however, the design is what determines most of the production time, cost, and quality. Although several methods tackling this problem have been published, they have proven unable to effectively resolve the situation if a system has continuous control factors. This work proposes a data mining approach, consisting of four stages based on artificial neural networks (ANN), desirability functions, and a simulated annealing (SA) algorithm to resolve the problems of dynamic parameter design with multiple responses. An ANN is employed to build a system’s response function model. Desirability functions are used to evaluate the performance measures of multiple responses. A SA algorithm is applied to obtain the best factor settings through the response function model. By using the proposed approach, the obtained best factor settings can be any values within their upper and lower bounds so that the system’s multiple responses have the least sensitivity to noise factors along the magnitude of the signal factor. An example from the literature is illustrated to confirm the feasibility and effectiveness of the proposed approach.     

差分模型参数递推估计的Householder变换法

本文提出了利用Householder变换进行差分模型参数递推估计的新方法.并由该方法导 出了新的递推最小二乘法、递推增广矩阵法、递推广义最小二乘法、递推极大似然法. 文中分单变量、多变量两种情况重点讨论了新递推最小二乘法及其与传统递推最小二乘 法的比较,并给出了计算实例.