Incorporating prior knowledge in support vector machines for classification: A review

For classification, support vector machines (SVMs) have recently been introduced and quickly became the state of the art. Now, the incorporation of prior knowledge into SVMs is the key element that allows to increase the performance in many applications. This paper gives a review of the current state of research regarding the incorporation of two general types of prior knowledge into SVMs for classification. The particular forms of prior knowledge considered here are presented in two main groups: class-invariance and knowledge on the data. The first one includes invariances to transformations, to permutations and in domains of input space, whereas the second one contains knowledge on unlabeled data, the imbalance of the training set or the quality of the data. The methods are then described and classified into the three categories that have been used in literature: sample methods based on the modification of the training data, kernel methods based on the modification of the kernel and optimization methods based on the modification of the problem formulation. A recent method, developed for support vector regression, considers prior knowledge on arbitrary regions of the input space. It is exposed here when applied to the classification case. A discussion is then conducted to regroup sample and optimization methods under a regularization framework.     

一种支持向量逐步回归机算法研究

支持向量机是解决非线性问题的重要工具,对多元线性回归模型和支持向量机的原始形式进行比较,拟定从样本子集的多元线性回归模型出发,逐步搜索支持向量,提出了一种建立支持向量回归机的快速算法,以降低核矩阵的规模从而降低解凸二次规划的复杂度;最后,分析了该算法的复杂度,并提供了一个算例。     

Reduced twin support vector regression

We propose the reduced twin support vector regressor (RTSVR) that uses the notion of rectangular kernels to obtain significant improvements in execution time over the twin support vector regressor (TSVR), thus facilitating its application to larger sized datasets.     

结构可调的支持向量回归估计

针对定义域各分区间内样本数据的噪声强度不同,以及在局部范围内数据变化急剧等复杂情况,提出了结构可调的支持向量回归估计(AS-SVR)方法,包括采用不同的损失函数,对各样本点自适应地选用不同的参数等。推导了求解公式,给出了调整算法。实例测试表明,AS-SVR方法的楚模效果优于常规方法。     

Adapting prior knowledge activation: Mobilisation, perspective taking, and learners’ prior knowledge

This study investigates the effects of two prior knowledge activation strategies, namely, mobilisation and perspective taking, on learning. It is hypothesised that the effectiveness of these strategies is influenced by learners’ prior domain knowledge. More specifically, mobilisation is expected to be the most effective activation strategy at lower levels of prior knowledge. Mobilisation is a bottom-up oriented strategy that serves a broad stage-setting function. It provides learners with a relevant context in which new information can be integrated, which might be especially beneficial for learners with lower levels of prior knowledge to help them extend their limited knowledge base. As prior knowledge increases, perspective taking is expected to become the most effective strategy for activating learners’ prior knowledge. Perspective taking is a top-down oriented strategy that results in the activation of a corresponding schema. This schema guides the selection and processing of information relevant to the schema, which might especially support learners with higher levels of prior knowledge to refine their already elaborated knowledge base. The effectiveness of the activation strategies (in terms of learning task performance) was indeed influenced by learners’ prior knowledge in the hypothesised direction.     

Support vector regression from simulation data and few experimental samples

This paper considers nonlinear modeling based on a limited amount of experimental data and a simulator built from prior knowledge. The problem of how to best incorporate the data provided by the simulator, possibly biased, into the learning of the model is addressed. This problem, although particular, is very representative of numerous situations met in engine control, and more generally in engineering, where complex models, more or less accurate, exist and where the experimental data which can be used for calibration are difficult or expensive to obtain. The first proposed method constrains the function to fit to the values given by the simulator with a certain accuracy, allowing to take the bias of the simulator into account. The second method constrains the derivatives of the model to fit to the derivatives of a prior model previously estimated on the simulation data. The combination of these two forms of prior knowledge is also possible and considered. These approaches are implemented in the linear programming support vector regression (LP-SVR) framework by the addition, to the optimization problem, of constraints, which are linear with respect to the parameters. Tests are then performed on an engine control application, namely, the estimation of the in-cylinder residual gas fraction in Spark Ignition (SI) engine with Variable Camshaft Timing (VCT). Promising results are obtained on this application. The experiments have also shown the importance of adding potential support vectors in the model when using Gaussian RBF kernels with very few training samples.     

A multiple criteria active learning method for support vector regression

This paper presents a novel active learning method developed in the framework of ε-insensitive support vector regression (SVR) for the solution of regression problems with small size initial training data. The proposed active learning method selects iteratively the most informative as well as representative unlabeled samples to be included in the training set by jointly evaluating three criteria: (i) relevancy, (ii) diversity, and (iii) density of samples. All three criteria are implemented according to the SVR properties and are applied in two clustering-based consecutive steps. In the first step, a novel measure to select the most relevant samples that have high probability to be located either outside or on the boundary of the ε-tube of SVR is defined. To this end, initially a clustering method is applied to all unlabeled samples together with the training samples that are inside the ε-tube (those that are not support vectors, i.e., non-SVs); then the clusters with non-SVs are eliminated. The unlabeled samples in the remaining clusters are considered as the most relevant patterns. In the second step, a novel measure to select diverse samples among the relevant patterns from the high density regions in the feature space is defined to better model the SVR learning function. To this end, initially clusters with the highest density of samples are chosen to identify the highest density regions in the feature space. Then, the sample from each selected cluster that is associated with the portion of feature space having the highest density (i.e., the most representative of the underlying distribution of samples contained in the related cluster) is selected to be included in the training set. In this way diverse samples taken from high density regions are efficiently identified. Experimental results obtained on four different data sets show the robustness of the proposed technique particularly when a small-size initial training set are available.     

Accurate on-line support vector regression incorporated with compensated prior knowledge

Neural Computing and Applications - When the training data required by the data-driven model is insufficient or difficult to cover the sample space completely, incorporating the prior knowledge and...     

回归型支持向量机的调节熵函数法

基于最优化理论中的ＫＫＴ 互补条件建立支持向量回归机的无约束不可微优化模型,并给出了一种有效的光滑近似解法———调节熵函数方法．该方法不需参数取值很大便可逼近问题的最优解,从而避免了一般熵函数法为了逼近精确解,参数取得过大而导致数值的溢出现象,为求解支持向量回归机提供了一条新途径．数值实验结果表明,回归型支持向量机的调节熵函数法改善了支持向量机的回归性能和效率．     

Incorporating prior model into Gaussian processes regression for WEDM process modeling

Sufficient sampling is usually time-consuming and expensive but also is indispensable for supporting high precise data-driven modeling of wire-cut electrical discharge machining (WEDM) process. Considering the natural way to describe the behavior of a WEDM process by IF-THEN rules drawn from the field experts, engineering knowledge and experimental work, in this paper, the fuzzy logic model is chosen as prior knowledge to leverage the predictive performance. Focusing on the fusion between rough fuzzy system and very scarce noisy samples, a simple but effective re-sampling algorithm based on piecewise relational transfer interpolation is presented and it is integrated with Gaussian processes regression (GPR) for WEDM process modeling. First, by using re-sampling algorithm encoded derivative regularization, the prior model is translated into a pseudo training dataset, and then the dataset is trained by the Gaussian processes. An empirical study on two benchmark datasets intuitively demonstrates the feasibility and effectiveness of this approach. Experiments on high-speed WEDM (DK7725B) are conducted for validation of nonlinear relationship between the design variables (i.e., workpiece thickness, peak current, on-time and off-time) and the responses (i.e., material removal rate and surface roughness). The experimental result shows that combining very rough fuzzy prior model with training examples still significantly improves the predictive performance of WEDM process modeling, even with very limited training dataset. That is, given the generalized prior model, the samples needed by GPR model could be reduced greatly meanwhile keeping precise.     

Cycle-breaking acceleration for support vector regression

Support vector regression (SVR) is a powerful tool in modeling and prediction tasks with widespread application in many areas. The most representative algorithms to train SVR models are Shevade et al.'s Modification 2 and Lin's WSS1 and WSS2 methods in the LIBSVM library. Both are variants of standard SMO in which the updating pairs selected are those that most violate the Karush-Kuhn-Tucker optimality conditions, to which LIBSVM adds a heuristic to improve the decrease in the objective function. In this paper, and after presenting a simple derivation of the updating procedure based on a greedy maximization of the gain in the objective function, we show how cycle-breaking techniques that accelerate the convergence of support vector machines (SVM) in classification can also be applied under this framework, resulting in significantly improved training times for SVR.     

Fuzzy nile-based support vector regression system

In this paper, we design a fuzzy rule-based support vector regression system. The proposed system utilizes the advantages of fuzzy model and support vector regression to extract support vectors to generate fuzzy if-then rules from the training data set. Based on the first-order hnear Tagaki-Sugeno （TS） model, the structure of rules is identified by the support vector regression and then the consequent parameters of rules are tuned by the global least squares method. Our model is applied to the real world regression task. The simulation results gives promising performances in terms of a set of fuzzy hales, which can be easily interpreted by humans.     

Adaptive simplification of solution for support vector machine

SVM has been receiving increasing interest in areas ranging from its original application in pattern recognition to other applications such as regression estimation due to its remarkable generalization performance. Unfortunately, SVM is currently considerably slower in test phase caused by number of the support vectors, which has been a serious limitation for some applications. To overcome this problem, we proposed an adaptive algorithm named feature vectors selection (FVS) to select the feature vectors from the support vector solutions, which is based on the vector correlation principle and greedy algorithm. Through the adaptive algorithm, the sparsity of solution is improved and the time cost in testing is reduced. To select the number of the feature vectors adaptively by the requirements, the generalization and complexity trade-off can be directly controlled. The computer simulations on regression estimation and pattern recognition show that FVS is a promising algorithm to simplify the solution for support vector machine.     

Rough support vector clustering

In this paper a novel kernel-based soft clustering method is proposed. This method incorporates rough set theoretic flavour in support vector clustering paradigm to achieve soft clustering. Empirical studies show that this method can find soft clusters having arbitrary shapes.     

支持向量机在字符识别中的应用研究

本文应用SVM对字符图像识别进行实验研究,并在此基础上,研究了SVM对含有高斯噪声的字符图像的识别问题。研究结果表明,SVM能够在有限样本的情况下,获得较高的识别率,是目前小样本学习的最佳解决方案。     

Parallel randomized sampling for support vector machine (SVM) and support vector regression (SVR)

A parallel randomized support vector machine (PRSVM) and a parallel randomized support vector regression (PRSVR) algorithm based on a randomized sampling technique are proposed in this paper. The proposed PRSVM and PRSVR have four major advantages over previous methods. (1) We prove that the proposed algorithms achieve an average convergence rate that is so far the fastest bounded convergence rate, among all SVM decomposition training algorithms to the best of our knowledge. The fast average convergence bound is achieved by a unique priority based sampling mechanism. (2) Unlike previous work (Provably fast training algorithm for support vector machines, 2001) the proposed algorithms work for general linear-nonseparable SVM and general non-linear SVR problems. This improvement is achieved by modeling new LP-type problems based on Karush–Kuhn–Tucker optimality conditions. (3) The proposed algorithms are the first parallel version of randomized sampling algorithms for SVM and SVR. Both the analytical convergence bound and the numerical results in a real application show that the proposed algorithm has good scalability. (4) We present demonstrations of the algorithms based on both synthetic data and data obtained from a real word application. Performance comparisons with SVM^light show that the proposed algorithms may be efficiently implemented.     

Performance improvement of character recognition in industrial applications using prior knowledge for more reliable segmentation

In industrial applications optical character recognition with smart cameras becomes more and more popular. Since these applications mostly have challenging environments for the systems it is most important to have very reliable character segmentation and classification algorithms. The investigations of several algorithms have shown that character segmentation is one if not the main bottleneck of character recognition. Furthermore, the requirements of robust and fast algorithms related to skew angle estimation and line segmentation, as well as tilt angle estimation, and character segmentation are high. This is the reason for introducing such algorithms that are specifically adapted to industrial applications. Additionally, a method is proposed that is based on the Bayes theorem to take account of prior knowledge for line and character segmentation. The main focus of the investigations of the character recognition system is recognition performance and speed, since real-time constraints are very hard in industrial application. Both requirements are evaluated on an image series captured with a smart camera in an industrial application.     

Regularized least squares support vector regression for the simultaneous learning of a function and its derivatives

In this paper, we propose a regularized least squares approach based support vector machine for simultaneously approximating a function and its derivatives. The proposed algorithm is simple and fast as no quadratic programming solver needs to be employed. Effectively, only the solution of a structured system of linear equations is needed.     

基于分组SVR和KNR的单帧图像超分辨

基于学习的图像超分辨是超分辨领域的一类新方法,该方法通过建立映射模型有针对性地对图像目标进行恢复,取得较好的超分辨效果,但往往需要大量学习样本,实际情况中一般难以满足。在无高分辨清晰图像库作为训练样本的前提下,从低分辨图像与其插值图像之间的关系出发,引入分组的思想,采用支持向量回归（SVR）或核非线性回归（KNR）对“组”建立局部映射模型,利用局部模型针对性地重新估计被插值的像素点。结果表明该方法有明显的超分辨效果。