Global Convergence of Decomposition Learning Methods for Support Vector Machines

Decomposition methods are well-known techniques for solving quadratic programming (QP) problems arising in support vector machines (SVMs). In each iteration of a decomposition method, a small number of variables are selected and a QP problem with only the selected variables is solved. Since large matrix computations are not required, decomposition methods are applicable to large QP problems. In this paper, we will make a rigorous analysis of the global convergence of general decomposition methods for SVMs. We first introduce a relaxed version of the optimality condition for the QP problems and then prove that a decomposition method reaches a solution satisfying this relaxed optimality condition within a finite number of iterations under a very mild condition on how to select variables     

Random block coordinate descent method for multi-label support vector machine with a zero label

Multi-label support vector machine with a zero label (Rank-SVMz) is an effective SVM-type technique for multi-label classification, which is formulated as a quadratic programming (QP) problem with several disjoint equality constraints and lots of box ones, and then is solved by Frank–Wolfe method (FWM) embedded one-versus-rest (OVR) decomposition trick. However, it is still highly desirable to speed up the training and testing procedures of Rank-SVMz for many real world applications. Due to the special disjoint equality constraints, all variables to be solved in Rank-SVMz are naturally divided into several blocks via OVR technique. Therefore we propose a random block coordinate descent method (RBCDM) for Rank-SVMz in this paper. At each iteration, an entire QP problem is divided into a series of small-scale QP sub-problems, and then each QP sub-problem with a single equality constraint and many box ones is solved by sequential minimization optimization (SMO) used in binary SVM. The theoretical analysis shows that RBCDM has a much lower time complexity than FWM for Rank-SVMz. Our experimental results on six benchmark data sets demonstrate that, on the average, RBCDM runs 11 times faster, produces 12% fewer support vectors, and achieves a better classification performance than FWM for Rank-SVMz. Therefore Rank-SVMz with RBCDM is a powerful candidate for multi-label classification.     

复杂流程工业系统的优化操作

ＳＱＰ方法在中小规模非线性规划中已成为主流算法,但在求解大规模优化问题时存在Ｈｅｓｓｉａｎ矩阵规模过大,存储、计算困难,以及计算量随不等式约束数量呈指数上升等缺点,简约空间ＳＱＰ法将变量分解为独立变量和非猖变量两部分。优化时只考虑独立变量,从而大大降低了变量维数,减小了Ｈｅｓｓｉａｎ矩阵规模。内点法、修改障碍函数法在求解不等式约束问题时都具有迭代次数几乎不受不等式约束规模影响的特点,因此可以将它们集成入简约空间ＳＱＰ法,使之可以更有效地对大规模优化问题求解。     

Improved multiclass feature selection via list combination

Feature selection is a crucial machine learning technique aimed at reducing the dimensionality of the input space. By discarding useless or redundant variables, not only it improves model performance but also facilitates its interpretability. The well-known Support Vector Machines–Recursive Feature Elimination (SVM-RFE) algorithm provides good performance with moderate computational efforts, in particular for wide datasets. When using SVM-RFE on a multiclass classification problem, the usual strategy is to decompose it into a series of binary ones, and to generate an importance statistics for each feature on each binary problem. These importances are then averaged over the set of binary problems to synthesize a single value for feature ranking. In some cases, however, this procedure can lead to poor selection. In this paper we discuss six new strategies, based on list combination, designed to yield improved selections starting from the importances given by the binary problems. We evaluate them on artificial and real-world datasets, using both One–Vs–One (OVO) and One–Vs–All (OVA) strategies. Our results suggest that the OVO decomposition is most effective for feature selection on multiclass problems. We also find that in most situations the new K-First strategy can find better subsets of features than the traditional weight average approach.     

基于QPSO训练支持向量机的网络入侵检测

对于大规模入侵检测问题,分解算法是训练支持向量机的主要方法之一.在结构风险最小化的情况下,利用改进后的蚁群算法(QPSO)解决二次规划问题(QP),寻找最优解,并对 ArraySVM 算法进行了改进,同时对KDD入侵检测数据进行了检测.结果表明,算法精确度高于改进前的 ArraySVM 算法,并且减少了支持向量点数量.     

Combining DC Algorithms (DCAs) and Decomposition Techniques for the Training of Nonpositive–Semidefinite Kernels

Today, decomposition methods are one of the most popular methods for training support vector machines (SVMs). With the use of kernels that do not satisfy Mercer's condition, new techniques must be designed to handle nonpositive–semidefinite kernels resulting to this choice. In this work we incorporate difference of convex (DC functions) optimization techniques into decomposition methods to tackle this difficulty. The new approach needs no problem modification and we show that the only use of a truncated DC algorithms (DCAs) in the decomposition scheme produces a sufficient decrease of the objective function at each iteration. Thanks to this property, an asymptotic convergence proof of the new algorithm is produced without any blockwise convexity assumption on the objective function. We also investigate a working set selection rule using second-order information for sequential minimal optimization (SMO)-type decomposition in the spirit of DC optimization. Numerical results show the robustness and the efficiency of the new methods compared with state-of-the-art software.      

一种基于多策略差分进化的分解多目标进化算法

为了提高多目标优化问题非支配解集合的分布性和收敛性,根据不同差分进化策略的特点,基于切比雪夫分解机制,提出一种基于多策略差分进化的分解多目标进化算法(MOEA/D-WMSDE).该算法首先采用切比雪夫分解机制,将多目标优化问题转化为一系列单目标优化子问题;然后引入小波基函数和正态分布实现差分进化算法的参数控制,探究一种...     

The analysis of decomposition methods for support vector machines

The support vector machine (SVM) is a promising technique for pattern recognition. It requires the solution of a large dense quadratic programming problem. Traditional optimization methods cannot be directly applied due to memory restrictions. Up to now, very few methods can handle the memory problem and an important one is the "decomposition method." However, there is no convergence proof so far. We connect this method to projected gradient methods and provide theoretical proofs for a version of decomposition methods. An extension to bound-constrained formulation of SVM is also provided. We then show that this convergence proof is valid for general decomposition methods if their working set selection meets a simple requirement.     

Trajectory following for a flexible probe with state/input constraints: An approach based on model predictive control

This paper describes the trajectory following algorithm developed for a bio-inspired flexible probe, the direction of which can be controlled by means of an offset between interlocked probe segments which make up its body. The control approach employs model predictive control (MPC) to explicitly consider input and state constraints which arise from the unique mechanism of motion of the probe. For the sake of fast computation, a tracking error model is modified so that the nonlinear kinematic model of the probe is linearized, and the model is used to convert the optimization problem into a well-known quadratic programming (QP) problem. The input and state constraints are also converted into an inequality to be integrated into the QP problem. Simulated results demonstrate that the linearized tracking error model and the MPC control strategy are appropriate, handling large perturbations robustly, while satisfying all constraints. Experimental results in a gelatine sample, carried out with a 12 mm outer diameter prototype, demonstrate satisfactory two-dimensional trajectory following performance (0.52 mm average tracking error, with 1.49 mm STD). The experimental results also show that, given the probe’s constraints, the proposed controller provides more robust performance against large insertion perturbations than previously published control strategies developed for the probe.     

基于广义改进分解策略的多目标代理优化方法

为解决多目标代理优化方法中代理模型选择单一问题,提出基于广义改进函数分解策略的多目标代理优化方法.该方法充分利用模型预测信息构建广义改进多目标分解准则和广义改进R2指标准则,有效拓展多目标代理优化中代理模型的选择空间.所提两种准则通过随机均匀权重实现全局探索和局部搜索能力的自适应平衡.研究结果表明,所提方法在有限仿真条件下拥有良好的寻优性能,获得Pareto前沿在收敛性、多样性及空间分布性方面均具有一定优势.相比同类方法,该方法具有优势:1)不需要模型预测不确定性信息,适用于基于不同种类代理模型的代理优化方法; 2)实现简单且计算复杂度低,能够有效提升昂贵黑箱问题优化效率.     

高性能应用软件并行优化策略的研究

多核时代的来临对现有的应用软件提出了严重挑战,串行代码难以充分发挥硬件资源的性能;软件的并行优化成为亟待解决的重要问题。本文综合了 MPI,OpenMP,众核编程模型 CUDA 三个编程模型进行研究,讨论了适用于不同软件并行优化的方法,提出了适用于企业级应用的软件并行优化策略,最后总结和展望了软件并行优化的挑战和前景。     

Bandit problems and the exploration/exploitation tradeoff

We explore the two-armed bandit with Gaussian payoffs as a theoretical model for optimization. The problem is formulated from a Bayesian perspective, and the optimal strategy for both one and two pulls is provided. We present regions of parameter space where a greedy strategy is provably optimal. We also compare the greedy and optimal strategies to one based on a genetic algorithm. In doing so, we correct a previous error in the literature concerning the Gaussian bandit problem and the supposed optimality of genetic algorithms for this problem. Finally, we provide an analytically simple bandit model that is more directly applicable to optimization theory than the traditional bandit problem and determine a near-optimal strategy for that model     

Optimization based on bacterial chemotaxis

We present an optimization algorithm based on a model of bacterial chemotaxis. The original biological model is used to formulate a simple optimization algorithm, which is evaluated on a set of standard test problems. Based on this evaluation, several features are added to the basic algorithm using evolutionary concepts in order to obtain an improved optimization strategy, called the bacteria chemotaxis (BC) algorithm. This strategy is evaluated on a number of test functions for local and global optimization, compared with other optimization techniques, and applied to the problem of inverse airfoil design. The comparisons show that on average, BC performs similar to standard evolution strategies and worse than evolution strategies with enhanced convergence properties     

基于粒子群优化的有约束模型预测控制器

研究了模型预测控制(MPC)中解决带约束的优化问题时所用到的优化算法,针对传统的二次规划(QP)方法的不足,引入了一种带有混沌初始化的粒子群优化算法(CPSO),将其应用到模型预测控制中,用十解决同时带有输入约束和状态约束的控制问题.最后,引入了一个实际的带有约束的线性离散系统的优化控制问题,分别用二次规划和粒子群优化两种算法去解决,通过仿真结果的比较,说明了基于粒子群优化(PSO)的模型预测控制算法的优越性.     

Gibbs sampling based distributed OFDMA resource allocation

In this article, we present a distributed resource and power allocation scheme for muRip]e-resource wireless cellular networks. The global optimization of multi-cell multi-link resource allocation problem is known to be NP-hard in the general case. We use Gibbs sampling based algorithms to perform a distributed optimization that would lead to the global optimum of the problem. The objective of this article is to show how to use the Gibbs sampling （GS） algorithm and its variant the Metropolis-Hastings （MH） algorithm. We also propose an enhanced method of the MH algorithm, based on a priori known target state distribution, which improves the convergence speed without increasing the complexity. Also, we study different temperature cooling strategies and investigate their impact on the network optimization and convergence speed. Simulation results have also shown the effectiveness of the proposed methods.     

基于博弈策略强化学习的函数优化算法

该文提出了一种基于博弈论的函数优化算法。算法将优化问题的搜索空间映射为博弈的策略组合空间,优化目标函数映射为博弈的效用函数,通过博弈策略的强化学习过程智能地求解函数优化问题。文章给出了算法的形式定义及描述,然后在一组标准的函数优化测试集上进行了仿真运算,验证了算法的有效性。     

The query clustering problem: a set partitioning approach

In this research, we address the query clustering problem which involves determining globally optimal execution strategies for a set of queries. The need to process a set of queries together often arises in deductive database systems, scientific database systems, large bibliographic retrieval systems and several other database applications. We address the optimization problem from the perspective of overlaps in data requirements, and model the batched operations using a set-partitioning approach. In this model, we first consider the case of m queries each involving a two-way join operation. We develop a recursive methodology to determine all the processing strategies in this case. Next, we establish certain dominance properties among the strategies, and develop exact as well as heuristic algorithms for selecting an appropriate strategy. We extend this analysis to a clustering approach, and outline a framework for optimizing multiway joins. The results show that the proposed approach is viable and efficient, and can easily be incorporated into the query processing component of most database systems     

MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition

Decomposition is a basic strategy in traditional multiobjective optimization. However, it has not yet been widely used in multiobjective evolutionary optimization. This paper proposes a multiobjective evolutionary algorithm based on decomposition (MOEA/D). It decomposes a multiobjective optimization problem into a number of scalar optimization subproblems and optimizes them simultaneously. Each subproblem is optimized by only using information from its several neighboring subproblems, which makes MOEA/D have lower computational complexity at each generation than MOGLS and nondominated sorting genetic algorithm II (NSGA-II). Experimental results have demonstrated that MOEA/D with simple decomposition methods outperforms or performs similarly to MOGLS and NSGA-II on multiobjective 0-1 knapsack problems and continuous multiobjective optimization problems. It has been shown that MOEA/D using objective normalization can deal with disparately-scaled objectives, and MOEA/D with an advanced decomposition method can generate a set of very evenly distributed solutions for 3-objective test instances. The ability of MOEA/D with small population, the scalability and sensitivity of MOEA/D have also been experimentally investigated in this paper.     

Optimizing array-intensive applications for on-chip multiprocessors

With energy consumption becoming one of the first-class optimization parameters in computer system design, compilation techniques that consider performance and energy simultaneously are expected to play a central role. In particular, compiling a given application code under performance and energy constraints is becoming an important problem. In this paper, we focus on an on-chip multiprocessor architecture and present a set of code optimization strategies. We first evaluate an adaptive loop parallelization strategy (i.e., a strategy that allows each loop nest to execute using a different number of processors if doing so is beneficial) and measure the potential energy savings when unused processors during execution of a nested loop are shut down (i.e., placed into a power-down or sleep state). Our results show that shutting down unused processors can lead to as much as 67 percent energy savings at the expense of up to 17 percent performance loss in a set of array-intensive applications. To eliminate this performance penalty, we also discuss and evaluate a processor preactivation strategy based on compile-time analysis of nested loops. Based on our experiments, we conclude that an adaptive loop parallelization strategy combined with idle processor shut down and preactivation can be very effective in reducing energy consumption without increasing execution time. We then generalize our strategy and present an application parallelization strategy based on integer linear programming (ILP). Given an array-intensive application, our optimization strategy determines the number of processors to be used in executing each loop nest based on the objective function and additional compilation constraints provided by the user/programmer. Our initial experience with this constraint-based optimization strategy shows that it is very successful in optimizing array-intensive applications on on-chip multiprocessors under multiple energy and performance constraints.