Bayesian support vector regression using a unified loss function

In this paper, we use a unified loss function, called the soft insensitive loss function, for Bayesian support vector regression. We follow standard Gaussian processes for regression to set up the Bayesian framework, in which the unified loss function is used in the likelihood evaluation. Under this framework, the maximum a posteriori estimate of the function values corresponds to the solution of an extended support vector regression problem. The overall approach has the merits of support vector regression such as convex quadratic programming and sparsity in solution representation. It also has the advantages of Bayesian methods for model adaptation and error bars of its predictions. Experimental results on simulated and real-world data sets indicate that the approach works well even on large data sets.     

On-Line Resource Management with Application to Routing and Scheduling

We propose a framework to model on-line resource management problems based on an on-line version of positive linear programming. We consider both min cost problems and max benefit problems and propose logarithmic competitive algorithms that are optimal up to a constant factor. The proposed framework provides a general methodology that applies to a wide class of on-line problems: shop scheduling, packet routing, and in general a class of packing and assignment problems. Previously studied problems as on-line multiprocessor scheduling and on-line virtual circuit routing can also be modeled within this framework. Received December 18, 1996; revised March 2, 1997.     

A continuous optimization framework for hybrid system identification

We propose a new framework for hybrid system identification, which relies on continuous optimization. This framework is based on the minimization of a cost function that can be chosen as either the minimum or the product of loss functions. The former is inspired by traditional estimation methods, while the latter is inspired by recent algebraic and support vector regression approaches to hybrid system identification. In both cases, the identification problem is recast as a continuous optimization program involving only the real parameters of the model as variables, thus avoiding the use of discrete optimization. This program can be solved efficiently by using standard optimization methods even for very large data sets. In addition, the proposed framework easily incorporates robustness to different kinds of outliers through the choice of the loss function.     

The generalized LASSO

In the last few years, the support vector machine (SVM) method has motivated new interest in kernel regression techniques. Although the SVM has been shown to exhibit excellent generalization properties in many experiments, it suffers from several drawbacks, both of a theoretical and a technical nature: the absence of probabilistic outputs, the restriction to Mercer kernels, and the steep growth of the number of support vectors with increasing size of the training set. In this paper, we present a different class of kernel regressors that effectively overcome the above problems. We call this approach generalized LASSO regression. It has a clear probabilistic interpretation, can handle learning sets that are corrupted by outliers, produces extremely sparse solutions, and is capable of dealing with large-scale problems. For regression functionals which can be modeled as iteratively reweighted least-squares (IRLS) problems, we present a highly efficient algorithm with guaranteed global convergence. This defies a unique framework for sparse regression models in the very rich class of IRLS models, including various types of robust regression models and logistic regression. Performance studies for many standard benchmark datasets effectively demonstrate the advantages of this model over related approaches.     

Linear programming support vector regression with wavelet kernel: A new approach to nonlinear dynamical systems identification

Wavelet theory has a profound impact on signal processing as it offers a rigorous mathematical framework to the treatment of multiresolution problems. The combination of soft computing and wavelet theory has led to a number of new techniques. On the other hand, as a new generation of learning algorithms, support vector regression (SVR) was developed by Vapnik et al. recently, in which ?-insensitive loss function was defined as a trade-off between the robust loss function of Huber and one that enables sparsity within the SVs. The use of support vector kernel expansion also provides us a potential avenue to represent nonlinear dynamical systems and underpin advanced analysis. However, for the support vector regression with the standard quadratic programming technique, the implementation is computationally expensive and sufficient model sparsity cannot be guaranteed. In this article, from the perspective of model sparsity, the linear programming support vector regression (LP-SVR) with wavelet kernel was proposed, and the connection between LP-SVR with wavelet kernel and wavelet networks was analyzed. In particular, the potential of the LP-SVR for nonlinear dynamical system identification was investigated.     

Bayesian framework for least-squares support vector machine classifiers, gaussian processes, and kernel Fisher discriminant analysis

The Bayesian evidence framework has been successfully applied to the design of multilayer perceptrons (MLPs) in the work of MacKay. Nevertheless, the training of MLPs suffers from drawbacks like the nonconvex optimization problem and the choice of the number of hidden units. In support vector machines (SVMs) for classification, as introduced by Vapnik, a nonlinear decision boundary is obtained by mapping the input vector first in a nonlinear way to a high-dimensional kernel-induced feature space in which a linear large margin classifier is constructed. Practical expressions are formulated in the dual space in terms of the related kernel function, and the solution follows from a (convex) quadratic programming (QP) problem. In least-squares SVMs (LS-SVMs), the SVM problem formulation is modified by introducing a least-squares cost function and equality instead of inequality constraints, and the solution follows from a linear system in the dual space. Implicitly, the least-squares formulation corresponds to a regression formulation and is also related to kernel Fisher discriminant analysis. The least-squares regression formulation has advantages for deriving analytic expressions in a Bayesian evidence framework, in contrast to the classification formulations used, for example, in gaussian processes (GPs). The LS-SVM formulation has clear primal-dual interpretations, and without the bias term, one explicitly constructs a model that yields the same expressions as have been obtained with GPs for regression. In this article, the Bayesian evidence framework is combined with the LS-SVM classifier formulation. Starting from the feature space formulation, analytic expressions are obtained in the dual space on the different levels of Bayesian inference, while posterior class probabilities are obtained by marginalizing over the model parameters. Empirical results obtained on 10 public domain data sets show that the LS-SVM classifier designed within the Bayesian evidence framework consistently yields good generalization performances.     

基于支持向量机的数据库学习算法

文中介绍了一个利用数据库中的大量数据进行决策的方法。对于仅涉及数据库中部分数据的问题,对数据库中与当前问题相关的数据采用具有强泛化能力的支持向量机方法学习分类规则和回归函数,完成对当前问题的分类和估值。支持向量机算法用非线性映射把数据映射到一个高维特征空间,在高维特征空间进行线性分类和线性回归,将原问题转化为一个凸次优化问题。上述算法实现了一个隧道工程支护设计系统,并取得了较好的效果。     

基于深度代价敏感CNN的年龄估计算法

针对年龄估计中样本数量不平衡及不同类间发生误分类时付出代价不同的问题,将代价敏感性嵌入深度学习框架中,提出基于深度代价敏感CNN的年龄估计算法.首先为每个年龄类别分别建立损失函数,解决训练样本的不平衡问题.然后,定义代价向量以反映不同类之间发生误分类而付出的代价差异性,构造逆交叉熵误差函数.最后,融合上述方法,为卷积神经网络(CNN)构造一个损失函数,使CNN在训练阶段学习针对年龄估计的鲁棒人脸表征.在不同种族的年龄估计标准图像集上的实验验证文中算法的有效性.     

Multi-view Laplacian twin support vector machines

Twin support vector machines are a recently proposed learning method for pattern classification. They learn two hyperplanes rather than one as in usual support vector machines and often bring performance improvements. Semi-supervised learning has attracted great attention in machine learning in the last decade. Laplacian support vector machines and Laplacian twin support vector machines have been proposed in the semi-supervised learning framework. In this paper, inspired by the recent success of multi-view learning we propose multi-view Laplacian twin support vector machines, whose dual optimization problems are quadratic programming problems. We further extend them to kernel multi-view Laplacian twin support vector machines. Experimental results demonstrate that our proposed methods are effective.     

基于二次多项式的光滑支持向量机在分类中的若干问题

袁玉波等人采用一个二次多项式作为光滑函数,提出一个基于二次多项式的光滑支持向量机。然而,这种光滑支持向量机作为一种新模型,还存在如下4个尚待解决的问题：能否用其它二次多项式作为光滑函数,构建光滑支持向量机？这种光滑支持向量机有多少个？其收敛性如何？其分类效果如何？本文从二次多项式入手,分析了二次多项式作为光滑函数的问题,还分析了基于二次多项式的光滑支持向量杌问题,解决了上述四个问题。     

Resampling methods for meta-model validation with recommendations for evolutionary computation

Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.     

Distribution-free consistency of empirical risk minimization and support vector regression

In this paper, we focus on the generalization ability of the empirical risk minimization technique in the framework of agnostic learning, and consider the support vector regression method as a special case. We give a set of analytic conditions that characterize the empirical risk minimization methods and their approximations that are distribution-free consistent. Then utilizing the weak topology of the feature space, we show that the support vector regression, possibly with a discontinuous kernel, is distribution-free consistent. Moreover, a tighter generalization error bound is shown to be achieved in certain cases if the value of the regularization parameter grows as the sample size increases. The results carry over to the ν-support vector regression.     

Parametric concavity in stochastic dynamic programs

e study a family of dynamic programs that are characterized by a deterministic vector of cost parameters. We show that if the single period cost function is concave with respect to this vector, then the optimal costs of the family of dynamic programs are also concave in the vector of costs. We also establish that the optimal cost inherits other properties, namely, super-additivity, +∞-star-shaped, 0-star-shaped, concavity-along-rays and monotonicity. When the vector of cost parameters evolves as a stochastic process and the single period cost is concave with respect to this vector, we show that the optimal cost is bounded above by the optimal cost for the dynamic program in which these stochastic cost parameters are replaced by their expectations in each period. We provide examples to illustrate how our results can be used to derive bounds which are either easy to compute or have analytical expressions. We also explain why such bounds are useful.     

A Rate and Distortion Analysis of Multiscale Binary Shape Coding Based on Statistical Learning

In this paper, we propose a statistical learning-based approach to analyze the rate-distortion characteristics of MPEG-4 multiscale binary shape coding. We employ the polynomial kernel function and epsiv-insensitive loss function for our support vector regression. To improve the accuracy of the estimation, rate and distortion related features are incorporated in the statistical learning framework. Our experimental results show that the proposed approach can achieve good performance, e.g., modelling the rate-distortion curves accurately.     

一种向量预选取的分段支持向量机回归算法

针对数据波动剧烈时,一组特定的支持向量机回归参数无法满足随数据分布而改变的要求,导致回归曲线达不到所要求的精度的问题,同时针对如何有效删除在回归过程中某些非必要的数据以加快求解速度的问题,本文提出一种向量预选取的分段支持向量机回归算法．该算法首先根据数据空间分布特点删除一些非必要数据,然后根据不同区域样本的复杂程度对区间进行分段,针对各个区域设置相应的参数．仿真实验证明：p-p-SVR算法在保持回归精度的同时,较传统方法具有更好的泛化性能．     

Leave-one-out cross-validation-based model selection for multi-input multi-output support vector machine

As an effective approach for multi-input multi-output regression estimation problems, a multi-dimensional support vector regression (SVR), named M-SVR, is generally capable of obtaining better predictions than applying a conventional support vector machine (SVM) independently for each output dimension. However, although there are many generalization error bounds for conventional SVMs, all of them cannot be directly applied to M-SVR. In this paper, a new leave-one-out (LOO) error estimate for M-SVR is derived firstly through a virtual LOO cross-validation procedure. This LOO error estimate can be straightway calculated once a training process ended with less computational complexity than traditional LOO method. Based on this LOO estimate, a new model selection methods for M-SVR based on multi-objective optimization strategy is further proposed in this paper. Experiments on toy noisy function regression and practical engineering data set, that is, dynamic load identification on cylinder vibration system, are both conducted, demonstrating comparable results of the proposed method in terms of generalization performance and computational cost.     

非对称ν-无核二次曲面支持向量回归机

针对回归问题提出了非对称[ν]-无核二次曲面支持向量回归机。通过引入Pinball损失函数,使得位于[ε]带上方和下方的样本点具有不同的惩罚,从而得到更优的回归函数。进一步从理论上分析了参数[p]和[ν]控制[ε]带上方和下方错误样本点数目的上界。当[p=0.5]时,该方法就退化成了对称[ν]-无核二次曲面支持向量回归机,此时也证明了参数[ν]可控制支持向量的个数。事实上,该算法不需要使用核函数,从而避免了核参数的选择且不损失决策函数的可解释性。数值实验部分展示了该算法具有更好的拟合性能且耗时较少,也分析了参数[p]不会增加计算成本。     

Efficient SVM Regression Training with SMO

The sequential minimal optimization algorithm (SMO) has been shown to be an effective method for training support vector machines (SVMs) on classification tasks defined on sparse data sets. SMO differs from most SVM algorithms in that it does not require a quadratic programming solver. In this work, we generalize SMO so that it can handle regression problems. However, one problem with SMO is that its rate of convergence slows down dramatically when data is non-sparse and when there are many support vectors in the solution—as is often the case in regression—because kernel function evaluations tend to dominate the runtime in this case. Moreover, caching kernel function outputs can easily degrade SMO's performance even more because SMO tends to access kernel function outputs in an unstructured manner. We address these problems with several modifications that enable caching to be effectively used with SMO. For regression problems, our modifications improve convergence time by over an order of magnitude.     

两种基于等式约束支持向量回归机比较研究

为了减小支持向量回归机(SVR)的计算复杂度、缩短训练时间,将应用于分类问题的近似支持向量机(PSVM)扩展到回归问题中,针对其原始优化问题采用直接法求取最优解,而不是转换为对偶问题求解,给出了近似支持向量回归机(PSVR)线性和非线性回归算法.并与同样基于等式约束的最小二乘支持向量回归机(LSSVR)进行了比较,在一维、二维函数回归以及不同规模通用数据集上的测试结果表明,PSVR算法简单,训练速度快,尤其在大规模数据集处理上更具优势.     

Constructing boosting algorithms from SVMs: an application toone-class classification

We show via an equivalence of mathematical programs that a support vector (SV) algorithm can be translated into an equivalent boosting-like algorithm and vice versa. We exemplify this translation procedure for a new algorithm: one-class leveraging, starting from the one-class support vector machine (1-SVM). This is a first step toward unsupervised learning in a boosting framework. Building on so-called barrier methods known from the theory of constrained optimization, it returns a function, written as a convex combination of base hypotheses, that characterizes whether a given test point is likely to have been generated from the distribution underlying the training data. Simulations on one-class classification problems demonstrate the usefulness of our approach