Ginisupport vector machines for segmental minimum Bayes risk decoding of continuous speech

We describe the use of support vector machines (SVMs) for continuous speech recognition by incorporating them in segmental minimum Bayes risk decoding. Lattice cutting is used to convert the Automatic Speech Recognition search space into sequences of smaller recognition problems. SVMs are then trained as discriminative models over each of these problems and used in a rescoring framework. We pose the estimation of a posterior distribution over hypotheses in these regions of acoustic confusion as a logistic regression problem. We also show that GiniSVMs can be used as an approximation technique to estimate the parameters of the logistic regression problem. On a small vocabulary recognition task we show that the use of GiniSVMs can improve the performance of a well trained hidden Markov model system trained under the Maximum Mutual Information criterion. We also find that it is possible to derive reliable confidence scores over the GiniSVM hypotheses and that these can be used to good effect in hypothesis combination. We discuss the problems that we expect to encounter in extending this approach to large vocabulary continuous speech recognition and describe initial investigation of constrained estimation techniques to derive feature spaces for SVMs.     

Benchmarking Least Squares Support Vector Machine Classifiers

In Support Vector Machines (SVMs), the solution of the classification problem is characterized by a (convex) quadratic programming (QP) problem. In a modified version of SVMs, called Least Squares SVM classifiers (LS-SVMs), a least squares cost function is proposed so as to obtain a linear set of equations in the dual space. While the SVM classifier has a large margin interpretation, the LS-SVM formulation is related in this paper to a ridge regression approach for classification with binary targets and to Fisher's linear discriminant analysis in the feature space. Multiclass categorization problems are represented by a set of binary classifiers using different output coding schemes. While regularization is used to control the effective number of parameters of the LS-SVM classifier, the sparseness property of SVMs is lost due to the choice of the 2-norm. Sparseness can be imposed in a second stage by gradually pruning the support value spectrum and optimizing the hyperparameters during the sparse approximation procedure. In this paper, twenty public domain benchmark datasets are used to evaluate the test set performance of LS-SVM classifiers with linear, polynomial and radial basis function (RBF) kernels. Both the SVM and LS-SVM classifier with RBF kernel in combination with standard cross-validation procedures for hyperparameter selection achieve comparable test set performances. These SVM and LS-SVM performances are consistently very good when compared to a variety of methods described in the literature including decision tree based algorithms, statistical algorithms and instance based learning methods. We show on ten UCI datasets that the LS-SVM sparse approximation procedure can be successfully applied.     

Support vector machine applications in the field of hydrology: A review

In the recent few decades there has been very significant developments in the theoretical understanding of Support vector machines (SVMs) as well as algorithmic strategies for implementing them, and applications of the approach to practical problems. SVMs introduced by Vapnik and others in the early 1990s are machine learning systems that utilize a hypothesis space of linear functions in a high dimensional feature space, trained with optimization algorithms that implements a learning bias derived from statistical learning theory. This paper reviews the state-of-the-art and focuses over a wide range of applications of SVMs in the field of hydrology. To use SVM aided hydrological models, which have increasingly extended during the last years; comprehensive knowledge about their theory and modelling approaches seems to be necessary. Furthermore, this review provides a brief synopsis of the techniques of SVMs and other emerging ones (hybrid models), which have proven useful in the analysis of the various hydrological parameters. Moreover, various examples of successful applications of SVMs for modelling different hydrological processes are also provided.     

Improving the training time of support vector regression algorithms through novel hyper-parameters search space reductions

The selection of hyper-parameters in support vector machines (SVM) is a key point in the training process of these models when applied to regression problems. Unfortunately, an exact method to obtain the optimal set of SVM hyper-parameters is unknown, and search algorithms are usually applied to obtain the best possible set of hyper-parameters. In general these search algorithms are implemented as grid searches, which are time consuming, so the computational cost of the SVM training process increases considerably. This paper presents a novel study of the effect of including reductions in the range of SVM hyper-parameters, in order to reduce the SVM training time, but with the minimum possible impact in its performance. The paper presents reduction in parameter C, by considering its relation with the rest of SVM hyper-parameters (γ and ε), through an approximation of the SVM model. On the other hand, we use some characteristics of the Gaussian kernel function and a previous result in the literature to obtain novel bounds for γ and ε hyper-parameters. The search space reductions proposed are evaluated in different regression problems from UCI and StatLib databases. All the experiments carried out applying the popular LIBSVM solver have shown that our approach reduces the SVM training time, maintaining the SVM performance similar to when the complete range in SVM parameters is considered.     

MAML2: meta reinforcement learning via meta-learning for task categories

Meta-learning has been widely applied to solving few-shot reinforcement learning problems, where we hope to obtain an agent that can learn quickly in a new task. However, these algorithms often ignore some isolated tasks in pursuit of the average performance, which may result in negative adaptation in these isolated tasks, and they usually need sufficient learning in a stationary task distribution. In this paper, our algorithm presents a hierarchical framework of double meta-learning, and the whole framework includes classification, meta-learning, and re-adaptation. Firstly, in the classification process, we classify tasks into several task subsets, considered as some categories of tasks, by learned parameters of each task, which can separate out some isolated tasks thereafter. Secondly, in the meta-learning process, we learn category parameters in all subsets via meta-learning. Simultaneously, based on the gradient of each category parameter in each subset, we use meta-learning again to learn a new meta-parameter related to the whole task set, which can be used as an initial parameter for the new task. Finally, in the re-adaption process, we adapt the parameter of the new task with two steps, by the meta-parameter and the appropriate category parameter successively. Experimentally, we demonstrate our algorithm prevents the agent from negative adaptation without losing the average performance for the whole task set. Additionally, our algorithm presents a more rapid adaptation process within re-adaptation. Moreover, we show the good performance of our algorithm with fewer samples as the agent is exposed to an online meta-learning setting.     

A support vector machine with a hybrid kernel and minimal Vapnik-Chervonenkis dimension

We present a mechanism to train support vector machines (SVMs) with a hybrid kernel and minimal Vapnik-Chervonenkis (VC) dimension. After describing the VC dimension of sets of separating hyperplanes in a high-dimensional feature space produced by a mapping related to kernels from the input space, we proposed an optimization criterion to design SVMs by minimizing the upper bound of the VC dimension. This method realizes a structural risk minimization and utilizes a flexible kernel function such that a superior generalization over test data can be obtained. In order to obtain a flexible kernel function, we develop a hybrid kernel function and a sufficient condition to be an admissible Mercer kernel based on common Mercer kernels (polynomial, radial basis function, two-layer neural network, etc.). The nonnegative combination coefficients and parameters of the hybrid kernel are determined subject to the minimal upper bound of the VC dimension of the learning machine. The use of the hybrid kernel results in a better performance than those with a single common kernel. Experimental results are discussed to illustrate the proposed method and show that the SVM with the hybrid kernel outperforms that with a single common kernel in terms of generalization power.     

Levy distributed parameter control in differential evolution for numerical optimization

Differential evolution (DE) algorithm is a population based stochastic search technique widely applied in scientific and engineering fields for global optimization over real parameter space. The performance of DE algorithm highly depends on the selection of values of the associated control parameters. Therefore, finding suitable values of control parameters is a challenging task and researchers have already proposed several adaptive and self-adaptive variants of DE. In the paper control parameters are adapted by levy distribution, named as Levy distributed DE (LdDE) which efficiently handles exploration and exploitation dilemma in the search space. In order to assure a fair comparison with existing parameter controlled DE algorithms, we apply the proposed method on number of well-known unimodal, basic and expanded multimodal and hybrid composite benchmark optimization functions having different dimensions. The empirical study shows that the proposed LdDE algorithm exhibits an overall better performance in terms of accuracy and convergence speed compared to five prominent adaptive DE algorithms.     

Model selection for primal SVM

This paper introduces two types of nonsmooth optimization methods for selecting model hyperparameters in primal SVM models based on cross-validation. Unlike common grid search approaches for model selection, these approaches are scalable both in the number of hyperparameters and number of data points. Taking inspiration from linear-time primal SVM algorithms, scalability in model selection is achieved by directly working with the primal variables without introducing any dual variables. The proposed implicit primal gradient descent (ImpGrad) method can utilize existing SVM solvers. Unlike prior methods for gradient descent in hyperparameters space, all work is done in the primal space so no inversion of the kernel matrix is required. The proposed explicit penalized bilevel programming (PBP) approach optimizes both the hyperparameters and parameters simultaneously. It solves the original cross-validation problem by solving a series of least squares regression problems with simple constraints in both the hyperparameter and parameter space. Computational results on least squares support vector regression problems with multiple hyperparameters establish that both the implicit and explicit methods perform quite well in terms of generalization and computational time. These methods are directly applicable to other learning tasks with differentiable loss functions and regularization functions. Both the implicit and explicit algorithms investigated represent powerful new approaches to solving large bilevel programs involving nonsmooth loss functions.     

监督学习模型指导的函数级编译优化参数选择方法研究

基于机器学习的迭代编译方法可以在对新程序进行迭代编译时,有效预测新程序的最佳优化参数组合。现有方法在模型训练过程中存在优化参数组合搜索效率较低、程序特征表示不恰当、预测精度不高的问题。因此,基于机器学习的迭代编译方法是当前迭代编译领域内的一个研究热点,其研究挑战在于学习算法选择、优化参数搜索以及程序特征表示等问题。基于监督学习技术,提出了一种程序优化参数预测方法。该方法首先通过约束多目标粒子群算法对优化参数空间进行搜索,找到样本函数的最佳优化参数;然后,通过动静结合的程序特征表示技术,对函数特征进行抽取;最后,通过由函数特征和优化参数形成的样本构建监督学习模型,对新程序的优化参数进行预测。分别采用k近邻法和softmax回归建立统计模型,实验结果表明,新方法在NPB测试集和大型科学计算程序上实现了较好的预测性能。     

Texture classification using the support vector machines

In recent years, support vector machines (SVMs) have demonstrated excellent performance in a variety of pattern recognition problems. In this paper, we apply SVMs for texture classification, using translation-invariant features generated from the discrete wavelet frame transform. To alleviate the problem of selecting the right kernel parameter in the SVM, we use a fusion scheme based on multiple SVMs, each with a different setting of the kernel parameter. Compared to the traditional Bayes classifier and the learning vector quantization algorithm, SVMs, and, in particular, the fused output from multiple SVMs, produce more accurate classification results on the Brodatz texture album.     

Model selection for support vector machines via uniform design

The problem of choosing a good parameter setting for a better generalization performance in a learning task is the so-called model selection. A nested uniform design (UD) methodology is proposed for efficient, robust and automatic model selection for support vector machines (SVMs). The proposed method is applied to select the candidate set of parameter combinations and carry out a k-fold cross-validation to evaluate the generalization performance of each parameter combination. In contrast to conventional exhaustive grid search, this method can be treated as a deterministic analog of random search. It can dramatically cut down the number of parameter trials and also provide the flexibility to adjust the candidate set size under computational time constraint. The key theoretic advantage of the UD model selection over the grid search is that the UD points are “far more uniform”and “far more space filling” than lattice grid points. The better uniformity and space-filling phenomena make the UD selection scheme more efficient by avoiding wasteful function evaluations of close-by patterns. The proposed method is evaluated on different learning tasks, different data sets as well as different SVM algorithms.     

基于混沌更新策略的蜂群算法在SVM参数优化中的应用

针对支持向量机的参数寻优缺乏数学理论指导,传统人工蜂群算法易陷入长期停滞的不足,而混沌搜索算法具有很好的随机性和遍历性,提出了基于混沌更新策略人工蜂群支持向量机参数选择模型(IABC-SVM)。该模型利用混沌搜索对侦察蜂搜索方式进行改进,有效提高蜂群算法搜索效率。以UCI标准数据库中的数据进行数值实验,采用ACO-SVM、PSO-SVM、ABC-SVM作为对比模型,实验表明了IABC在SVM参数优化中的可行性和有效性,具有较高的预测准确率和较好的算法稳定性。     

Support vector machines for texture classification

This paper investigates the application of support vector machines (SVMs) in texture classification. Instead of relying on an external feature extractor, the SVM receives the gray-level values of the raw pixels, as SVMs can generalize well even in high-dimensional spaces. Furthermore, it is shown that SVMs can incorporate conventional texture feature extraction methods within their own architecture, while also providing solutions to problems inherent in these methods. One-against-others decomposition is adopted to apply binary SVMs to multitexture classification, plus a neural network is used as an arbitrator to make final classifications from several one-against-others SVM outputs. Experimental results demonstrate the effectiveness of SVMs in texture classification.     

A study of SVM using a combination of the online learning method and the midpoint-validation method

The support vector machine (SVM) is known as one of the most influential and powerful tools for solving classification and regression problems, but the original SVM does not have an online learning technique. Therefore, many researchers have introduced online learning techniques to the SVM. In a previous article, we proposed an unsupervised online learning method using the technique of the self-organized map for the SVM. In another article, we proposed the midpoint validation method for an improved SVM. We test the performance of the SVM using a combination of the two techniques in this article. In addition, we compare its performance with the original hard-margin SVM, the soft-margin SVM, and the k-NN method, and also experiment with our proposed method on surface electromyogram recognition problems with changes in the position of the electrode. These experiments showed that our proposed method gave a better performance than the other SVMs and corresponded to the changing data.     

基于RBF核的SVM的模型选择及其应用

使用RBF核的SVM(支持向量机)被广泛应用于模式识别中。此类SVM的模型选择取决于两个参数,其一是惩罚因子C,其二是核参数σ2。该文使用了网格搜索和双线性搜索两种方法进行参数选择,并将两者的优点综合,应用于脱机手写体英文字符识别。实验在NIST数据集上进行了验证,对搜索效率和推广识别率进行了比较。实验结果还表明使用最优参数的SVM在识别率上比使用ANN(人工神经元网络)的分类器有较大提高。     

基于均匀设计的支持向量机参数优化方法

在实际应用中,支持向量机的性能依赖于参数的选择。针对支持向量机的参数选择问题进行了研究和分析,提出了基于均匀设计的支持向量机参数优化方法。与基于网格搜索、粒子群算法、遗传算法等支持向量机参数优化方法进行了比较与分析,采用多个不同规模的标准的分类数据集进行测试,比较了四种方法的分类正确率和运行时间。仿真实验表明,四种方法都能找到最优参数,使支持向量机的分类正确率接近或超过分类数据集的理论精度,本文方法具有寻参时间短的特点。     

A flexible probabilistic framework for large-margin mixture of experts

Mixture-of-Experts (MoE) enable learning highly nonlinear models by combining simple expert models. Each expert handles a small region of the data space, as dictated by the gating network which generates the (soft) assignment of input to the corresponding experts. Despite their flexibility and renewed interest lately, existing MoE constructions pose several difficulties during model training. Crucially, neither of the two popular gating networks used in MoE, namely the softmax gating network and hierarchical gating network (the latter used in the hierarchical mixture of experts), have efficient inference algorithms. The problem is further exacerbated if the experts do not have conjugate likelihood and lack a naturally probabilistic formulation (e.g., logistic regression or large-margin classifiers such as SVM). To address these issues, we develop novel inference algorithms with closed-form parameter updates, leveraging some of the recent advances in data augmentation techniques. We also present a novel probabilistic framework for MoE, consisting of a range of gating networks with efficient inference made possible through our proposed algorithms. We exploit this framework by using Bayesian linear SVMs as experts on various classification problems (which has a non-conjugate likelihood otherwise generally), providing our final model with attractive large-margin properties. We show that our models are significantly more efficient than other training algorithms for MoE while outperforming other traditional non-linear models like Kernel SVMs and Gaussian Processes on several benchmark datasets.

     

基于元学习的小样本遥感图像分类

为改善传统分类算法在小样本遥感图像分类上效果差的缺陷,提升模型的快速学习能力,提出融合迁移学习和元学习的小样本分类算法.设计基于长短期记忆网络的元学习器,通过门控结构拟合网络参数更新方式最下化损失下界,具有自动学习分类器参数更新方式的机制,相比于传统方法,能够有效扩展优化算法的搜索空间;考虑样本的跨类别知识转移和训练时...     

ADAPTIVE MULTI-OBJECTIVE OPTIMIZATION BASED ON NONDOMINATED SOLUTIONS

An adaptive hybrid model (AHM) based on nondominated solutions is presented in this study for multi-objective optimization problems (MOPs). In this model, three search phases are devised according to the number of nondominated solutions in the current population: 1) emphasizing the dominated solutions when the population contains very few nondominated solutions; 2) maintaining the balance between nondominated and dominated solutions when nondominated ones become more; 3) when the population consists of adequate nondominated solutions, dominated ones could be ignored and the isolated nondominated ones are allocated more computational budget by their crowding distance values for heuristic search. To exploit local information efficiently, a local incremental search algorithm, LISA, is proposed and merged into the model. This model maintains the adaptive mechanism between the optimization process by the online discovered nondominated solutions. The proposed model is validated using five ZDT and five DTLZ problems. Compared with three other state-of-the-art multi-objective algorithms, namely NSGA-II, SPEA2, and PESA-II, AHM achieves comparable results in terms of convergence and diversity metrics. Finally, the sensitivity of introduced parameters and scalability to the number of objectives are investigated.