Primal least squares twin support vector regression

The training algorithm of classical twin support vector regression (TSVR) can be attributed to the solution of a pair of quadratic programming problems (QPPs) with inequality constraints in the dual space.However,this solution is affected by time and memory constraints when dealing with large datasets.In this paper,we present a least squares version for TSVR in the primal space,termed primal least squares TSVR (PLSTSVR).By introducing the least squares method,the inequality constraints of TSVR are transformed into equality constraints.Furthermore,we attempt to directly solve the two QPPs with equality constraints in the primal space instead of the dual space;thus,we need only to solve two systems of linear equations instead of two QPPs.Experimental results on artificial and benchmark datasets show that PLSTSVR has comparable accuracy to TSVR but with considerably less computational time.We further investigate its validity in predicting the opening price of stock.     

Recursive reduced least squares support vector regression

Combining reduced technique with iterative strategy, we propose a recursive reduced least squares support vector regression. The proposed algorithm chooses the data which make more contribution to target function as support vectors, and it considers all the constraints generated by the whole training set. Thus it acquires less support vectors, the number of which can be arbitrarily predefined, to construct the model with the similar generalization performance. In comparison with other methods, our algorithm also gains excellent parsimoniousness. Numerical experiments on benchmark data sets confirm the validity and feasibility of the presented algorithm. In addition, this algorithm can be extended to classification.     

Regularized least squares fuzzy support vector regression for financial time series forecasting

In this paper, we propose a novel approach, termed as regularized least squares fuzzy support vector regression, to handle financial time series forecasting. Two key problems in financial time series forecasting are noise and non-stationarity. Here, we assign a higher membership value to data samples that contain more relevant information, where relevance is related to recency in time. The approach requires only a single matrix inversion. For the linear case, the matrix order depends only on the dimension in which the data samples lie, and is independent of the number of samples. The efficacy of the proposed algorithm is demonstrated on financial datasets available in the public domain.     

Extended least squares support vector machines for ordinal regression

We extend LS-SVM to ordinal regression, which has wide applications in many domains such as social science and information retrieval where human-generated data play an important role. Most current methods based on SVM for ordinal regression suffer from the problem of ignoring the distribution information reflected by the samples clustered around the centers of each class. This problem would degrade the performance of SVM-based methods since the classifiers only depend on the scattered samples on the border which induce large margin. Our method takes the samples clustered around class centers into account and has a competitive computational complexity. Moreover, our method would easily produce the optimal cut-points according to the prior class probabilities and hence may obtain more reasonable results when the prior class probabilities are not the same. Experiments on simulated datasets and benchmark datasets, especially on the real ordinal datasets, demonstrate the effectiveness of our method.     

一种改进的在线最小二乘支持向量机回归算法

针对一般最小二乘支持向量机处理大规模数据集会出现训练速度幔、计算量大、不易在线训练的缺点,将修正后的遗忘因子矩形窗方法与支持向量机相结合,提出一种基于改进的遗忘因子矩形窗算法的在线最小二乘支持向量机回归算法,既突出了当前窗口数据的作用,又考虑了历史数据的影响.所提出的算法可减少计算量,提高在线辨识精度.仿真算例表明了该方法的有效性.     

A wavelet-based image denoising using least squares support vector machine

The least squares support vector machine (LS-SVM) is a modified version of SVM, which uses the equality constraints to replace the original convex quadratic programming problem. Consequently, the global minimizer is much easier to obtain in LS-SVM by solving the set of linear equation. LS-SVM has shown to exhibit excellent classification performance in many applications. In this paper, a wavelet-based image denoising using LS-SVM is proposed. Firstly, the noisy image is decomposed into different subbands of frequency and orientation responses using the wavelet transform. Secondly, the feature vector for a pixel in a noisy image is formed by the spatial regularity in wavelet domain, and the LS-SVM model is obtained by training. Then the wavelet coefficients are divided into two classes (noisy coefficients and noise-free ones) by LS-SVM training model. Finally, all noisy wavelet coefficients are relatively well denoised by soft-thresholding method. Extensive experimental results demonstrate that our method can obtain better performances in terms of both subjective and objective evaluations than those state-of-the-art denoising techniques. Especially, the proposed method can preserve edges very well while removing noise.     

Fuzzy least squares twin support vector clustering

In this paper, we have formulated a fuzzy least squares version of recently proposed clustering method, namely twin support vector clustering (TWSVC). Here, a fuzzy membership value of each data pattern to different cluster is optimized and is further used for assigning each data pattern to one or other cluster. The formulation leads to finding k cluster center planes by solving modified primal problem of TWSVC, instead of the dual problem usually solved. We show that the solution of the proposed algorithm reduces to solving a series of system of linear equations as opposed to solving series of quadratic programming problems along with system of linear equations as in TWSVC. The experimental results on several publicly available datasets show that the proposed fuzzy least squares twin support vector clustering (F-LS-TWSVC) achieves comparable clustering accuracy to that of TWSVC with comparatively lesser computational time. Further, we have given an application of F-LS-TWSVC for segmentation of color images.

     

基于核偏最小二乘的支持向量机回归算法研究

对SVM的特征提取问题进行了研究,提出了KPLS-SVM组合回归建模方法.该方法在输入空间映射得到的高维特征空间中进行PLS特征提取后,再进行SVM回归,不仅保持了SVM良好的模型性能,并且兼具KPLS和SVM的优点.仿真和实验结果表明,该KPLS-SVM建模方法是正确且有效的,采用该方法构建的SVM模型,泛化性能明显优于没有特征提取的SVM.     

Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression

Given n training examples, the training of a least squares support vector machine (LS-SVM) or kernel ridge regression (KRR) corresponds to solving a linear system of dimension n. In cross-validating LS-SVM or KRR, the training examples are split into two distinct subsets for a number of times (l) wherein a subset of m examples are used for validation and the other subset of (n-m) examples are used for training the classifier. In this case l linear systems of dimension (n-m) need to be solved. We propose a novel method for cross-validation (CV) of LS-SVM or KRR in which instead of solving l linear systems of dimension (n-m), we compute the inverse of an n dimensional square matrix and solve l linear systems of dimension m, thereby reducing the complexity when l is large and/or m is small. Typical multi-fold, leave-one-out cross-validation (LOO-CV) and leave-many-out cross-validations are considered. For five-fold CV used in practice with five repetitions over randomly drawn slices, the proposed algorithm is approximately four times as efficient as the naive implementation. For large data sets, we propose to evaluate the CV approximately by applying the well-known incomplete Cholesky decomposition technique and the complexity of these approximate algorithms will scale linearly on the data size if the rank of the associated kernel matrix is much smaller than n. Simulations are provided to demonstrate the performance of LS-SVM and the efficiency of the proposed algorithm with comparisons to the naive and some existent implementations of multi-fold and LOO-CV.     

Entropy based fuzzy least squares twin support vector machine for class imbalance learning

In classification problems, the data samples belonging to different classes have different number of samples. Sometimes, the imbalance in the number of samples of each class is very high and the interest is to classify the samples belonging to the minority class. Support vector machine (SVM) is one of the widely used techniques for classification problems which have been applied for solving this problem by using fuzzy based approach. In this paper, motivated by the work of Fan et al. (Knowledge-Based Systems 115: 87–99 2017), we have proposed two efficient variants of entropy based fuzzy SVM (EFSVM). By considering the fuzzy membership value for each sample, we have proposed an entropy based fuzzy least squares support vector machine (EFLSSVM-CIL) and entropy based fuzzy least squares twin support vector machine (EFLSTWSVM-CIL) for class imbalanced datasets where fuzzy membership values are assigned based on entropy values of samples. It solves a system of linear equations as compared to the quadratic programming problem (QPP) as in EFSVM. The least square versions of the entropy based SVM are faster than EFSVM and give higher generalization performance which shows its applicability and efficiency. Experiments are performed on various real world class imbalanced datasets and compared the results of proposed methods with new fuzzy twin support vector machine for pattern classification (NFTWSVM), entropy based fuzzy support vector machine (EFSVM), fuzzy twin support vector machine (FTWSVM) and twin support vector machine (TWSVM) which clearly illustrate the superiority of the proposed EFLSTWSVM-CIL.     

Subspace identification of Hammerstein systems using least squares support vector machines

This paper presents a method for the identification of multiple-input-multiple-output (MIMO) Hammerstein systems for the goal of prediction. The method extends the numerical algorithms for subspace state space system identification (N4SID), mainly by rewriting the oblique projection in the N4SID algorithm as a set of componentwise least squares support vector machines (LS-SVMs) regression problems. The linear model and static nonlinearities follow from a low-rank approximation of a matrix obtained from this regression problem.     

Identification of MIMO Hammerstein models using least squares support vector machines

This paper studies a method for the identification of Hammerstein models based on least squares support vector machines (LS-SVMs). The technique allows for the determination of the memoryless static nonlinearity as well as the estimation of the model parameters of the dynamic ARX part. This is done by applying the equivalent of Bai's overparameterization method for identification of Hammerstein systems in an LS-SVM context. The SISO as well as the MIMO identification cases are elaborated. The technique can lead to significant improvements with respect to classical overparameterization methods as illustrated in a number of examples. Another important advantage is that no stringent assumptions on the nature of the nonlinearity need to be imposed except for a certain degree of smoothness.     

Robust energy-based least squares twin support vector machines

Twin support vector machine (TSVM), least squares TSVM (LSTSVM) and energy-based LSTSVM (ELS-TSVM) satisfy only empirical risk minimization principle. Moreover, the matrices in their formulations are always positive semi-definite. To overcome these problems, we propose in this paper a robust energy-based least squares twin support vector machine algorithm, called RELS-TSVM for short. Unlike TSVM, LSTSVM and ELS-TSVM, our RELS-TSVM maximizes the margin with a positive definite matrix formulation and implements the structural risk minimization principle which embodies the marrow of statistical learning theory. Furthermore, RELS-TSVM utilizes energy parameters to reduce the effect of noise and outliers. Experimental results on several synthetic and real-world benchmark datasets show that RELS-TSVM not only yields better classification performance but also has a lower training time compared to ELS-TSVM, LSPTSVM, LSTSVM, TBSVM and TSVM.     

Multi-category laplacian least squares twin support vector machine

In this paper, we have formulated a Laplacian Least Squares Twin Support Vector Machine called Lap-LST-KSVC for semi-supervised multi-category k-class classification problem. Similar to Least Squares Twin Support Vector Machine for multi-classification(LST-KSVC), Lap-LST-KSVC, evaluates all the training samples into “1-versus-1-versus-rest” classification paradigm, so as to generate ternary output {?1, 0, +1}. Experimental results prove the efficacy of the proposed method over other inline Laplacian Twin Support Vector Machine(Lap-TWSVM) in terms of classification accuracy and computational time.     

1-Norm least squares twin support vector machines

During the last few years, nonparallel plane classifiers, such as Multisurface Proximal Support Vector Machine via Generalized Eigenvalues (GEPSVM), and Least Squares TWSVM (LSTSVM), have attracted much attention. However, there are not any modifications of them that have been presented to automatically select the input features. This motivates the rush towards new classifiers. In this paper, we develop a new nonparallel plane classifier, which is designed for automatically selecting the relevant features. We first introduce a Tikhonov regularization (TR) term that is usually used for regularizing least squares into the LSTSVM learning framework, and then convert this formulation to a linear programming (LP) problem. By minimizing an exterior penalty (EP) problem of the dual of the LP formulation and using a fast generalized Newton algorithm, our method yields very sparse solutions, such that it generates a classifier that depends on only a smaller number of input features. In other words, this approach is capable of suppressing input features. This makes the classifier easier to store and faster to compute in the classification phase. Lastly, experiments on both toy and real problems disclose the effectiveness of our method.     

Online quality prediction for cobalt oxalate synthesis process using least squares support vector regression approach with dual updating

Online measurement of the average particle size is typically unavailable in industrial cobalt oxalate synthesis process, soft sensor prediction of the important quality variable is therefore required. Cobalt oxalate synthesis process is a complex multivariable and highly nonlinear process. In this paper, an effective soft sensor based on least squares support vector regression (LSSVR) with dual updating is developed for prediction the average particle size. In this soft sensor model, the methods of moving window LSSVR (MWLSSVR) updating and the model output offset updating is activated based on model performance assessment. Feasibility and efficiency of the proposed soft sensor are demonstrated through the application to an industrial cobalt oxalate synthesis process.     

Prediction of flashover voltage of insulators using least squares support vector machines

The importance of the research on insulator pollution has been increased considerably with the rise of the voltage of transmission lines. In order to determine the flashover behavior of polluted high voltage insulators and to identify to physical mechanisms that govern this phenomenon, the researchers have been brought to establish a modeling. In this paper, a dynamic model of AC flashover voltages of the polluted insulators is constructed using the least square support vector machine (LS-SVM) regression method. For this purpose, a training set is generated by using a numerical method based on Finite Element Method (FEM) for several of common insulators with different geometries. To improve the resulting model’s generalization ability, an efficient optimization algorithm known as the grid search are adopted to tune parameters in LS-SVM design.In addition, two different testing set, which are not introduced to the LS-SVM during the training procedures, is used to evaluate the effectiveness and feasibility of the proposed method. Then, optimum LS-SVM model is firstly obtained and the performance of the proposed system with other intelligence method based on ANN is compared. It can be concluded that the performance of LS-SVM model outperforms those of ANN, for the data set available, which indicates that the LS-SVM model has better generalization ability.     

基于IMK-WLSSVR的WiFi室内定位算法

针对传统的基于单核加权最小二乘支持向量回归(WLSSVR)定位算法对接收信号强度指示(RSSI)的拟合度较差,导致定位精度不高的问题,提出了一种基于改进多核WLSSVR(IMK-WLSSVR)的Wi Fi室内定位算法。采用多核支持向量回归机模型构建指纹点RSSI值与位置坐标的非线性关系,采用多项式核函数和高斯径向基核函数构建多核函数,利用正态分布权值函数为每个指纹点赋予不同的权重;然后利用改进的模拟退火算法优化定位模型参数,进一步提升模型的定位性能;最后利用构建的定位模型进行定位。实验结果表明:在3 m网格、3 d Bm标准差噪声条件下,算法定位误差小于2 m的概率可达92. 2%,其定位精度优于现有的加权K最近邻(WKNN)算法以及单核WLSSVR方法。     

The construction of fuzzy least squares estimators in fuzzy linear regression models

A new concept and method of imposing imprecise (fuzzy) input and output data upon the conventional linear regression model is proposed. Under the considerations of fuzzy parameters and fuzzy arithmetic operations (fuzzy addition and multiplication), we propose a fuzzy linear regression model which has the similar form as that of conventional one. We conduct the h-level (conventional) linear regression models of fuzzy linear regression model for the sake of invoking the statistical techniques in (conventional) linear regression analysis for real-valued data. In order to determine the sign (nonnegativity or nonpositivity) of fuzzy parameters, we perform the statistical testing hypotheses and evaluate the confidence intervals. Using the least squares estimators obtained from the h-level linear regression models, we can construct the membership functions of fuzzy least squares estimators via the form of “Resolution Identity” which is well-known in fuzzy sets theory. In order to obtain the membership degree of any given estimate taken from the fuzzy least squares estimator, optimization problems have to be solved. We also provide two computational procedures to deal with those optimization problems.