K-nearest neighbor-based weighted twin support vector regression

Twin support vector regression (TSVR) finds ?-insensitive up- and down-bound functions by resolving a pair of smaller-sized quadratic programming problems (QPPs) rather than a single large one as in a classical SVR, which makes its computational speed greatly improved. However the local information among samples are not exploited in TSVR. To make full use of the knowledge of samples and improve the prediction accuracy, a K-nearest neighbor-based weighted TSVR (KNNWTSVR) is proposed in this paper, where the local information among samples are utilized. Specifically, a major weight is given to the training sample if it has more K-nearest neighbors. Otherwise a minor weight is given to it. Moreover, to further enhance the computational speed, successive overrelaxation approach is employed to resolve the QPPs. Experimental results on eight benchmark datasets and a real dataset demonstrate our weighted TSVR not only yields lower prediction error but also costs lower running time in comparison with other algorithms.     

An efficient augmented Lagrangian method for support vector machine

ABSTRACT

Support vector machine (SVM) has proved to be a successful approach for machine learning. Two typical SVM models are the L1-loss model for support vector classification (SVC) and ε-L1-loss model for support vector regression (SVR). Due to the non-smoothness of the L1-loss function in the two models, most of the traditional approaches focus on solving the dual problem. In this paper, we propose an augmented Lagrangian method for the L1-loss model, which is designed to solve the primal problem. By tackling the non-smooth term in the model with Moreau–Yosida regularization and the proximal operator, the subproblem in augmented Lagrangian method reduces to a non-smooth linear system, which can be solved via the quadratically convergent semismooth Newton's method. Moreover, the high computational cost in semismooth Newton's method can be significantly reduced by exploring the sparse structure in the generalized Jacobian. Numerical results on various datasets in LIBLINEAR show that the proposed method is competitive with the most popular solvers in both speed and accuracy.     

最小二乘孪生参数化不敏感支持向量回归机

孪生参数化不敏感支持向量回归机（Twin Parametric Insensitive Support Vector Regression,TPISVR）是最近提出的一种新型机器学习方法.和其它回归方法相比,TPISVR在处理异方差噪声方面具有独特的优势.标准TPISVR的训练算法可以归结为在对偶空间求解一对具有不等式约束的二次规划问题.然而,这种求解方法的时间消耗比较大.本文引入最小二乘思想,将TPISVR的两个二次规划问题转化为两个线性方程组,并在原始空间上直接求解,提出了最小二乘孪生参数化不敏感支持向量回归机（Least Squares TPISVR,LSTPISVR）.为了解决LSTPISVR的参数选择问题,提出混沌布谷鸟优化算法,并用其对LSTPISVR的参数进行优化选择.在人工数据集和UCI数据集上的实验表明了LSTPISVR在保持精度不下降的情况下比TPISVR具有更高的运行效率.     

Discriminative least squares regression for multiclass classification based on within-class scatter minimization

Least square regression has been widely used in pattern classification, due to the compact form and efficient solution. However, two main issues limit its performance for solving the multiclass classification problems. The first one is that employing the hard discrete labels as the regression targets is inappropriate for multiclass classification. The second one is that it focus only on exactly fitting the instances to the target matrix while ignoring the within-class similarity of the instances, resulting in overfitting. To address this issues, we propose a discriminative least squares regression for multiclass classification based on within-class scatter minimization (WCSDLSR). Specifically, a ε-dragging technique is first introduced to relax the hard discrete labels into the slack soft labels, which enlarges the between-class margin for the soft labels as much as possible. The within-class scatter for the soft labels is then constructed as a regularization term to make the transformed instances of the same class closer to each other. These factors ensure WCSDLSR can learn a more compact and discriminative transformation for classification, thus avoiding the overfitting problems. Furthermore, the proposed WCSDLSR can obtain a closed-form solution in each iteration with the lower computational costs. Experimental results on the benchmark datasets demonstrate that the proposed WCSDLSR achieves the better classification performance with the lower computational costs.

     

Stochastic support vector regression with probabilistic constraints

Support Vector Regression (SVR) solves regression problems based on the concept of Support Vector Machine (SVM). In this paper, we introduce a novel model of SVR in which any training samples containing inputs and outputs are considered the random variables with known or unknown distribution functions. Constraints occurrence have a probability density function which helps to obtain maximum margin and achieve robustness. The optimal hyperplane regression can be obtained by solving a quadratic optimization problem. The proposed method is illustrated by several experiments including artificial data sets and real-world benchmark data sets.     

ε-支持向量回归机算法及其应用

针对现有传统的一些图像去噪方法难以获得清晰图像边缘的问题,提出了利用ε-SVR技术构建图像去噪滤波器的新方法。ε-支持向量回归机通过引入ε不敏感损失函数,可以实现具有较强鲁棒性的回归,而且回归估计是稀疏的,保留了SVM的所有优点。分析了ε-支持向量回归机理论算法及其在图像去噪中的应用,使用ε-支持向量回归机对图像进行滤波并且与最小值滤波、均值滤波和维纳滤波等常用的滤波方法相比较,还比较了SVM各种核函数对不同噪声的滤波效果和分析了不同阶数的Multinomial核的滤波效果。实验结果表明了ε-支持向量回归机能够有效地去除噪声,不但信噪比较高而且比较清晰,同时具有良好的稀疏性。     

An efficient implicit regularized Lagrangian twin support vector regression

Twin support vector regression (TSVR) and Lagrangian TSVR (LTSVR) satisfy only empirical risk minimization principle. Moreover, the matrices in their formulations are always positive semi-definite. To overcome these problems, we propose an efficient implicit Lagrangian formulation for the dual regularized twin support vector regression, called IRLTSVR for short. By introducing a regularization term to each objective function, the optimization problems in our IRLTSVR are positive definite and implement the structural risk minimization principle. Moreover, the 1-norm of the vector of slack variable is replaced with 2-norm to make the objective functions strongly convex. Our IRLTSVR solves two systems of linear equations instead of solving two quadratic programming problems (QPPs) in TSVR and one large QPP in SVR, which makes the learning speed of IRLTSVR faster than TSVR and SVR. Particularly, we compare three implementations of IRLTSVR with existing approaches. Computational results on several synthetic and real-world benchmark datasets clearly indicate the effectiveness and applicability of the IRLTSVR in comparison to SVR, TSVR and LTSVR.     

Smooth twin support vector regression

Twin support vector regression (TSVR) was proposed recently as a novel regressor that tries to find a pair of nonparallel planes, i.e., ε-insensitive up- and down-bounds, by solving two related SVM-type problems. However, it may incur suboptimal solution since its objective function is positive semi-definite and the lack of complexity control. In order to address this shortcoming, we develop a novel SVR algorithm termed as smooth twin SVR (STSVR). The idea is to reformulate TSVR as a strongly convex problem, which results in unique global optimal solution for each subproblem. To solve the proposed optimization problem, we first adopt a smoothing technique to convert the original constrained quadratic programming problems into unconstrained minimization problems, and then use the well-known Newton–Armijo algorithm to solve the smooth TSVR. The effectiveness of the proposed method is demonstrated via experiments on synthetic and real-world benchmark datasets.     

Parameter-insensitive kernel in extreme learning for non-linear support vector regression

Support vector regression (SVR) is a state-of-the-art method for regression which uses the ε‐sensitive loss and produces sparse models. However, non-linear SVRs are difficult to tune because of the additional kernel parameter. In this paper, a new parameter-insensitive kernel inspired from extreme learning is used for non-linear SVR. Hence, the practitioner has only two meta-parameters to optimise. The proposed approach reduces significantly the computational complexity yet experiments show that it yields performances that are very close from the state-of-the-art. Unlike previous works which rely on Monte-Carlo approximation to estimate the kernel, this work also shows that the proposed kernel has an analytic form which is computationally easier to evaluate.     

Fuzzy least squares twin support vector clustering

In this paper, we have formulated a fuzzy least squares version of recently proposed clustering method, namely twin support vector clustering (TWSVC). Here, a fuzzy membership value of each data pattern to different cluster is optimized and is further used for assigning each data pattern to one or other cluster. The formulation leads to finding k cluster center planes by solving modified primal problem of TWSVC, instead of the dual problem usually solved. We show that the solution of the proposed algorithm reduces to solving a series of system of linear equations as opposed to solving series of quadratic programming problems along with system of linear equations as in TWSVC. The experimental results on several publicly available datasets show that the proposed fuzzy least squares twin support vector clustering (F-LS-TWSVC) achieves comparable clustering accuracy to that of TWSVC with comparatively lesser computational time. Further, we have given an application of F-LS-TWSVC for segmentation of color images.

     

A two-stage support vector regression assisted sequential sampling approach for global metamodeling

Support vector regression (SVR), as a promising surrogate model, has been widely used to approximate expensive simulations in engineering design problem. To build a SVR model accurately and efficiently, a two-stage support vector regression (TSSVR) assisted sequential sampling approach is proposed in this paper with the consideration of SVR’s two unique features. In each sampling iteration of TSSVR, two SVR models are constructed successively based on the same training data. As for the first feature that only support vectors (SVs) have impact on the construction of SVR, the first-stage SVR with lower ε precision is built to prescreen some important samples as current SVs. As for the second feature that SVR model does not completely go through the samples, the second-stage SVR with higher ε precision is built to calculate the prediction errors at the SVs without any other computational cost, and the prediction errors are used to approximately measure the accuracy of the local regions around the SVs. Moreover, to describe the local regions around the SVs, the design space is partitioned into a set of Voronoi cells according to the current samples before prescreening SVs from the sample points. Then a new sample can be exploited in the corresponding Voronoi cell with the largest prediction error. In the next sampling iteration, the Voronoi cells and SVs are redefined. As the change of the local cell with the largest prediction error, global exploration is achieved. Finally, the proposed approach is validated by seven numerical examples and an engineering example. An overall comparison between the proposed approach and some other methods demonstrates that the proposed approach is efficient and suitable for engineering design problems involving computational-expensive simulations.     

GA-SVR: a novel hybrid data-driven model to simulate vertical load capacity of driven piles

Piles are widely applied to substructures of various infrastructural buildings. Soil has a complex nature; thus, a variety of empirical models have been proposed for the prediction of the bearing capacity of piles. The aim of this study is to propose a novel artificial intelligent approach to predict vertical load capacity of driven piles in cohesionless soils using support vector regression (SVR) optimized by genetic algorithm (GA). To the best of our knowledge, no research has been developed the GA-SVR model to predict vertical load capacity of driven piles in different timescales as of yet, and the novelty of this study is to develop a new hybrid intelligent approach in this field. To investigate the efficacy of GA-SVR model, two other models, i.e., SVR and linear regression models, are also used for a comparative study. According to the obtained results, GA-SVR model clearly outperformed the SVR and linear regression models by achieving less root mean square error (RMSE) and higher coefficient of determination (R²). In other words, GA-SVR with RMSE of 0.017 and R² of 0.980 has higher performance than SVR with RMSE of 0.035 and R² of 0.912, and linear regression model with RMSE of 0.079 and R² of 0.625.

     

An improved robust and sparse twin support vector regression via linear programming

Twin support vector regression (TSVR) was proposed recently as a novel regressor that tries to find a pair of nonparallel planes, i.e. \(\epsilon \) -insensitive up- and down-bounds, by solving two related SVM-type problems. Though TSVR exhibits good performance compared with conventional methods like SVR, it suffers from the following issues: (1) it lacks model complexity control and thus may incur overfitting and suboptimal solution; (2) it needs to solve a pair of quadratic programming problems which are relatively complex to implement; (3) it is sensitive to outliers; and (4) its solution is not sparse. To address these problems, we propose in this paper a novel regression algorithm termed as robust and sparse twin support vector regression. The central idea is to reformulate TSVR as a convex problem by introducing regularization technique first and then derive a linear programming (LP) formulation which is not only simple but also allows robustness and sparseness. Instead of solving the resulting LP problem in the primal, we present a Newton algorithm with Armijo step-size to resolve the corresponding exact exterior penalty problem. The experimental results on several publicly available benchmark data sets show the feasibility and effectiveness of the proposed method.     

Using Bayesian regression and EM algorithm with missing handling for software effort prediction

ContextAlthough independent imputation techniques are comprehensively studied in software effort prediction, there are few studies on embedded methods in dealing with missing data in software effort prediction.ObjectiveWe propose BREM (Bayesian Regression and Expectation Maximization) algorithm for software effort prediction and two embedded strategies to handle missing data.MethodThe MDT (Missing Data Toleration) strategy ignores the missing data when using BREM for software effort prediction and the MDI (Missing Data Imputation) strategy uses observed data to impute missing data in an iterative manner while elaborating the predictive model.ResultsExperiments on the ISBSG and CSBSG datasets demonstrate that when there are no missing values in historical dataset, BREM outperforms LR (Linear Regression), BR (Bayesian Regression), SVR (Support Vector Regression) and M5′ regression tree in software effort prediction on the condition that the test set is not greater than 30% of the whole historical dataset for ISBSG dataset and 25% of the whole historical dataset for CSBSG dataset. When there are missing values in historical datasets, BREM with the MDT and MDI strategies significantly outperforms those independent imputation techniques, including MI, BMI, CMI, MINI and M5′. Moreover, the MDI strategy provides BREM with more accurate imputation for the missing values than those given by the independent missing imputation techniques on the condition that the level of missing data in training set is not larger than 10% for both ISBSG and CSBSG datasets.ConclusionThe experimental results suggest that BREM is promising in software effort prediction. When there are missing values, the MDI strategy is preferred to be embedded with BREM.     

Combining support vector regression with feature selection for multivariate calibration

Multivariate calibration is a classic problem in the analytical chemistry field and frequently solved by partial least squares (PLS) and artificial neural networks (ANNs) in the previous works. The spaciality of multivariate calibration is high dimensionality with small sample. Here, we apply support vector regression (SVR) as well as ANNs, and PLS to the multivariate calibration problem in the determination of the three aromatic amino acids (phenylalanine, tyrosine and tryptophan) in their mixtures by fluorescence spectroscopy. The results of the leave-one-out method show that SVR performs better than other methods, and appear to be one good method for this task. Furthermore, feature selection is performed for SVR to remove redundant features and a novel algorithm named Prediction RIsk based FEature selection for support vector Regression (PRIFER) is proposed. Results on the above multivariate calibration data set show that PRIFER is a powerful tool for solving the multivariate calibration problems.     

拉格朗日支持向量回归的有限牛顿算法

拉格朗日支持向量回归是一种有效的快速回归算法,求解时需要对维数等于样本数加一的矩阵求逆,求解需要较多的迭代次数才能收敛。采用一种Armijo步长有限牛顿迭代算法求解拉格朗日支持向量回归的优化问题,只需有限次求解一组线性等式而不需要求解二次规划问题,该方法具有全局收敛和有限步终止的性质。在多个标准数据集上的实验验证了所提算法的有效性和快速性。     

A comparison of machine learning methods for cutting parameters prediction in high speed turning process

Support vector machines are arguably one of the most successful methods for data classification, but when using them in regression problems, literature suggests that their performance is no longer state-of-the-art. This paper compares performances of three machine learning methods for the prediction of independent output cutting parameters in a high speed turning process. Observed parameters were the surface roughness (Ra), cutting force \((F_{c})\), and tool lifetime (T). For the modelling, support vector regression (SVR), polynomial (quadratic) regression, and artificial neural network (ANN) were used. In this research, polynomial regression has outperformed SVR and ANN in the case of \(F_{c}\) and Ra prediction, while ANN had the best performance in the case of T, but also the worst performance in the case of \(F_{c}\) and Ra. The study has also shown that in SVR, the polynomial kernel has outperformed linear kernel and RBF kernel. In addition, there was no significant difference in performance between SVR and polynomial regression for prediction of all three output machining parameters.     

Feasibility of imperialist competitive algorithm to predict the surface settlement induced by tunneling

Surface settlement is considered as an adverse effect induced by tunneling in the civil projects. This paper proposes the use of the imperialist competitive algorithm (ICA) for predicting the maximum surface settlement (MMS) resulting from the tunneling. For this work, three forms of equations, i.e., linear, quadratic and power are developed and their weights are then optimized/updated with the ICA. The requirement datasets were collected from the line No. 2 of Karaj urban railway, in Iran. In the ICA models, vertical to horizontal stress ratio, cohesion and Young’s modulus, as the effective parameters on the MSS, are adopted as the inputs. The statistical performance parameters such as root mean square error (RMSE), mean bias error (MBE), and square correlation coefficient (R²) are presented and compared to validate the performance. The findings indicate that the developed ICA-based models with the R² of 0.979, 0.948 and 0.941, obtained from ICA power, ICA quadratic and ICA linear models, respectively, are the acceptable and accurate tools to estimate MSS, and furthermore prove their prediction capability for future research works in this field.

     

A new design of evolutionary hybrid optimization of SVR model in predicting the blast-induced ground vibration

This study aims to identify the suitability of hybridizing the firefly algorithm (FA), genetic algorithm (GA), and particle swarm optimization (PSO) with two well-known data-driven models of support vector regression (SVR) and artificial neural network (ANN) to predict blast-induced ground vibration. Here, these combinations are abbreviated using FA–SVR, PSO–SVR, GA–SVR, FA–ANN, PSO–ANN, and GA–ANN models. In addition, a modified FA (MFA) combined with SVR model is also proposed in this study, namely, MFA–SVR. The feasibility of the proposed models is examined using a case study, located in Johor, Malaysia. Then, to provide an objective assessment of performances of the predictive models, their results were compared based on several well known and popular statistical criteria. According to the results, the MFA–SVR with the coefficient of determination (R²) of 0.984 and root mean square error (RMSE) of 0.614 was more accurate model to predict PPV than the PSO–SVR with R² = 0.977 and RMSE = 0.725, the FA–SVR with R² = 0.964 and RMSE = 0.923, the GA–SVR with R² = 0.957 and RMSE = 1.016, the GA–ANN with R² = 0.936 and RMSE = 1.252, the FA–ANN with R² = 0.925 and RMSE = 1.368, and the PSO–ANN with R² = 0.924 and RMSE = 1.366. Consequently, the MFA–SVR model can be sufficiently employed in estimating the ground vibration, and has the capacity to generalize.

     

求解非线性回归问题的Newton算法

针对大规模非线性回归问题,提出基于静态储备池的Newton算法.利用储备池搭建高维特征空间,将原始问题转化成与储备池维数相关的线性支持向量回归问题,并应用Newton算法求解.鲁棒损失函数的应用可抑制异常点对预测结果的干扰.通过与SVR(Support Vector Regression)及储备池Tikhonov正则化方法比较,验证了所提方法的快速性、较高的预测精度和较好的鲁棒性.