Algorithmic Aspects of Using Small Instance Relaxations in Parallel Branch-and-Cut

Essential for the success of branch-and-cut algorithms for solving combinatorial optimization problems are the availability of reasonable tight relaxations and effective routines for solving the associated separation problems. In this paper we introduce the concept of small instance relaxations which can be particularly useful for problems with symmetric structure. Small instance relaxations are based on the facets of polytopes associated with small instances of the combinatorial optimization problem to be solved and can be generated automatically by facet enumeration. For a certain class of symmetric problems, we describe a general approach to the separation problem. Algorithmic aspects of using small instance relaxations effectively (parallel separation, facet selection, cutting plane selection) are discussed in detail. Extensive computational results are presented for the linear ordering problem and a certain betweenness problem. Received March 31, 1998; revised November 9, 1998.     

Greedy algorithms for a class of knapsack problems with binary weights

In this article we identify a class of two-dimensional knapsack problems with binary weights and related three-criteria unconstrained combinatorial optimization problems that can be solved in polynomial time by greedy algorithms. Starting from the knapsack problem with two equality constraints we show that this problem can be solved efficiently by using an appropriate partitioning of the items with respect to their binary weights. Based on the results for this problem we derive an algorithm for the three-criteria unconstrained combinatorial optimization problem with two binary objectives that explores the connectedness of the set of efficient knapsacks with respect to a combinatorial definition of adjacency. Furthermore, we prove that our approach is asymptotically optimal and provide extensive computational experiments that shows that we can solve the three-criteria problem with up to one million items in less than half an hour. Finally, we derive an efficient algorithm for the two-dimensional knapsack problems with binary constraints that only takes into account the results we obtained for the unconstrained three-criteria problem with binary weights.     

Approximate GCDs of polynomials and sparse SOS relaxations

The problem of computing approximate GCDs of several polynomials with real or complex coefficients can be formulated as computing the minimal perturbation such that the perturbed polynomials have an exact GCD of given degree. We present algorithms based on SOS (Sums Of Squares) relaxations for solving the involved polynomial or rational function optimization problems with or without constraints.     

无约束最优化问题的BFGS松弛异步并行算法

为解决大规模非线性最优化问题的串行求解速度慢的问题,提出应用松弛异步并行算法求解无约束最优化问题。根据无约束最优化问题的BFGS串行算法,在PC机群环境下将其并行化。利用CHOLESKY方法分解系数为对称正定矩阵的线性方程组,运用无序松弛异步并行方法求解解向量和Wolfe-Powell非线性搜索步长,并行求解BFGS修正公式,构建BFGS松弛异步并行算法,并对算法的时间复杂性、加速比进行分析。在PC机群的实验结果表明,该算法提高了无约束最优化问题的求解速度且负载均衡,算法具有线性加速比。     

A semidefinite method for tensor complementarity problems

In this paper, we study how to compute all real solutions of the tensor complimentary problem, if there are finite many ones. We formulate the problem as a sequence of polynomial optimization problems. The solutions can be computed sequentially. Each of them can be obtained by solving Lasserre's hierarchy of semidefinite relaxations. A semidefinite algorithm is proposed and its convergence properties are discussed. Some numerical experiments are also presented.     

Some algebraic methods for solving multiobjective polynomial integer programs

Multiobjective discrete programming is a well-known family of optimization problems with a large spectrum of applications. The linear case has been tackled by many authors during the past few years. However, the polynomial case has not been studied in detail due to its theoretical and computational difficulties. This paper presents an algebraic approach for solving these problems. We propose a methodology based on transforming the polynomial optimization problem to the problem of solving one or more systems of polynomial equations and we use certain Gröbner bases to solve these systems. Different transformations give different methodologies that are theoretically stated and compared by some computational tests via the algorithms that they induce.     

Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations

ABSTRACT

Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the limited-memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) update typically do not require manually tuning hyper-parameters but suffer from approximating a potentially indefinite Hessian with a positive-definite matrix. Hessian-free methods leverage the ability to perform Hessian-vector multiplication without needing the entire Hessian matrix, but each iteration's complexity is significantly greater than quasi-Newton methods. In this paper we propose an alternative approach for solving ML problems based on a quasi-Newton trust-region framework for solving large-scale optimization problems that allow for indefinite Hessian approximations. Numerical experiments on a standard testing data set show that with a fixed computational time budget, the proposed methods achieve better results than the traditional limited-memory BFGS and the Hessian-free methods.     

A first-order multigrid method for bound-constrained convex optimization

The aim of this paper is to design an efficient multigrid method for constrained convex optimization problems arising from discretization of some underlying infinite dimensional problems. Due to problem dependency of this approach, we only consider bound constraints with (possibly) a single equality constraint. As our aim is to target large-scale problems, we want to avoid computation of second derivatives of the objective function, thus excluding Newton-like methods. We propose a smoothing operator that only uses first-order information and study the computational efficiency of the resulting method.     

GpoSolver: a Matlab/C++ toolbox for global polynomial optimization

Global polynomial optimization can be a powerful tool when applied to engineering problems. One of the most successful methods for solving such problems is based on convex linear matrix inequality (LMI) relaxations. Software implementations of this approach can be found for example in Matlab toolboxes GloptiPoly and YALMIP. Matlab language makes it very easy when it comes to modelling polynomial problems. However, when using these toolboxes, Matlab is also required for the problem solving. GpoSolver aims at bridging this gap by providing a Matlab-based problem modelling toolbox supplemented by a problem-solving back end in a form of a C++ template library. Once a problem is conveniently modelled and parametrized in Matlab, a C++ class is automatically generated by GpoSolver. This class can be easily included into an existing codebase and used to solve different instances of the problem based on the supplied parameters.     

An exact penalty function algorithm for solving general constrained parameter optimization problems

An exact penalty function type of algorithm is proposed to solve a general class of constrained parameter optimization problems. The proposed algorithm has the property that any solution obtained by it will always satisfy the problem constraints, and that it will obtain a solution to the constrained problem, within a given specified tolerance, by solving a single unconstrained problem, i.e. it is not necessary to solve a sequence of unconstrained optimization problems. The algorithm applies a modification of Rosenbrock's (Rosenbrock, 1960) polynomial boundary penalty function, and a negative exponential penalty function with moving parameters, to modify the objective function in the neighborhood of the constrained region; a robust unconstrained algorithm (Davison and Wong, 1975) is then used to solve the resulting unconstrained optimization problem. Some standard test functions are included to show the performance of the algorithhm. Application of the algorithm is then made to solve some computer-aided design problems occurring in the area of control system synthesis.     

Generalized nonlinear ℋ︁∞ synthesis condition with its numerically efficient solution

In this paper, we will first derive a general synthesis condition for the output‐feedback ??_∞ control of smooth nonlinear systems. Computationally efficient ??_∞ control design procedure for a subclass of smooth nonlinear systems with polynomial vector field is then proposed by converting the resulting Hamilton‐Jacobi‐Isaacs inequalities from rational forms to their equivalent polynomial forms. Using quadratic Lyapunov functions, both the state‐feedback and output‐feedback problems will be reformulated as semi‐definite optimization conditions and locally tractable solutions can be obtained through sum‐of‐squares (SOS) programming. The proposed nonlinear ??_∞ design approach achieves significant relaxations on the plant structure compared with existing results in the literature. Moreover, the SOS‐based solution algorithm provides an effective computational scheme to break the bottleneck in solving nonlinear ??_∞ and optimal control problems. The proposed nonlinear ??_∞ control approach has been applied to several examples to demonstrate its advantages over existing nonlinear control techniques and its usefulness to engineering problems. Copyright © 2010 John Wiley & Sons, Ltd.     

A matrix-free line-search algorithm for nonconvex optimization

In this paper, we have developed a new algorithm for solving nonconvex large-scale problems. The new algorithm performs explicit matrix modifications adaptively, mimicing the implicit modifications used by trust-region methods. Thus, it shares the equivalent theoretical strength of trust-region approaches, without needing to accommodate an explicit step-size constraint. We show that the algorithm is well suited for solving very large-scale nonconvex problems whenever Hessian-vector products are available. The numerical results on the CUTEr problems demonstrate the effectiveness of this approach in the context of a line-search method for large-scale unconstrained nonconvex optimization. Moreover, applications in deep-learning problems further illustrate the usefulness of this algorithm. It does not share any of the prohibitive traits of popular matrix-free algorithms such as truncated conjugate gradient (CG) due to the difficult nature of deep-learning problems. Thus the proposed algorithm serves to bridge the gap between the needs of data-mining community and existing state-of-the-art approaches embraced foremost by the optimization community. Moreover, the proposed approach can be realized with minimal modification to the CG algorithm itself with negligible storage and computational overhead.     

DFP优化的数据点渐进迭代拟合方法

DFP方法(由Davidon,Fletcher和Powell 3人共同提出)是求解无约束优化问题的一种经典方法,文中指出数据点的拟合问题可转化为无约束优化问题的求解,并基于DFP优化方法给出了一种大规模数据点拟合方法,称之为DFP渐进迭代拟合方法.文中证明了该方法生成的极限曲线为初始数据点的最小二乘拟合曲线;它承袭了经典最小二乘渐进迭代逼近算法的众多优良性质,如具备直观的几何意义、可灵活地拟合大规模数据点、初始控制顶点的选择不影响最终迭代结果等.数值实例进一步表明,同等条件下,文中方法的收敛速度明显优于现有的几种数据点拟合方法.     

Global optimal solutions to general sensor network localization problem

Sensor network localization problem is to determine the position of the sensor nodes in a network given pairwise distance measurements. Such problem can be formulated as a quartic polynomial minimization via the least squares method. This paper presents a canonical duality theory for solving this challenging problem. It is shown that the nonconvex minimization problem can be reformulated as a concave maximization dual problem over a convex set in a symmetrical matrix space, and hence can be solved efficiently by combining a general (linear or quadratic) perturbation technique with existing optimization techniques. Applications are illustrated by solving some relatively large-scale problems. Our results show that the general sensor network localization problem is not NP-hard unless its canonical dual problem has no solution in its positive definite domain. Fundamental ideas for solving general NP-hard problems are discussed.     

A universal modification of the linear coupling method

In the late sixties, N. Shor and B. Polyak independently proposed optimal first-order methods for solving non-smooth convex optimization problems. In 1982 A. Nemirovski proposed optimal first-order methods for solving smooth convex optimization problems, which utilized auxiliary line search. In 1985 A. Nemirovski and Yu. Nesterov proposed a parametric family of optimal first-order methods for solving convex optimization problems with intermediate smoothness. In 2013 Yu. Nesterov proposed a universal gradient method which combined all good properties of the previous methods, except the possibility of using auxiliary line search. One can typically observe that in practice auxiliary line search improves performance for many tasks. In this paper, we propose the apparently first such method of non-smooth convex optimization allowing the use of the line search procedure. Moreover, it is based on the universal gradient method, which does not require any a priori information about the actual degree of smoothness of the problem. Numerical experiments demonstrate that the proposed method is, in some cases, considerably faster than Nesterov's universal gradient method.     

A Newton method for solving continuous multiple material minimum compliance problems

This paper presents an implementation of an active-set line-search Newton method intended for solving large-scale instances of a class of multiple material minimum compliance problems. The problem is modeled with a convex objective function and linear constraints. At each iteration of the Newton method, one or two linear saddle point systems are solved. These systems involve the Hessian of the objective function, which is both expensive to compute and completely dense. Therefore, the linear algebra is arranged such that the Hessian is not explicitly formed. The main concern is to solve a sequence of closely related problems appearing as the continuous relaxations in a nonlinear branch and bound framework for solving discrete minimum compliance problems. A test-set consisting of eight discrete instances originating from the design of laminated composite structures is presented. Computational experiments with a branch and bound method indicate that the proposed Newton method can, on most instances in the test-set, take advantage of the available starting point information in an enumeration tree and resolve the relaxations after branching with few additional function evaluations. Discrete feasible designs are obtained by a rounding heuristic. Designs with provably good objective functions are presented.     

一类非线性极大极小问题的极大熵社会认知算法

针对一类非线性极大极小问题目标函数非光滑的特点给求解带来的困难,利用社会认知算法并结合极大熵函数法给出了此类问题的一种新的有效算法。首先利用极大熵函数将原问题转化为一个光滑无约束优化问题,然后利用社会认知算法对其进行求解。该算法是基于社会认知理论,通过一系列的学习代理来模拟人类的社会性以及智能性从而完成对目标的优化。数值结果表明,该算法收敛快,数值稳定性好,是求解非线性极大极小问题的一种有效算法。     

An interval space reducing method for constrained problems with particle swarm optimization

In this paper, we propose a method for solving constrained optimization problems using interval analysis combined with particle swarm optimization. A set inverter via interval analysis algorithm is used to handle constraints in order to reduce constrained optimization to quasi unconstrained one. The algorithm is useful in the detection of empty search spaces, preventing useless executions of the optimization process. To improve computational efficiency, a space cleaning algorithm is used to remove solutions that are certainly not optimal. As a result, the search space becomes smaller at each step of the optimization procedure. After completing pre-processing, a modified particle swarm optimization algorithm is applied to the reduced search space to find the global optimum. The efficiency of the proposed approach is demonstrated through comprehensive experimentation involving 100 000 runs on a set of well-known benchmark constrained engineering design problems. The computational efficiency of the new method is quantified by comparing its results with other PSO variants found in the literature.     

一类非线性极小极大问题的改进粒子群算法

针对一类非线性极小极大问题目标函数非光滑的特点给求解带来的困难,利用改进的粒子群算法并结合极大熵函数法给出了此类问题的一种新的有效算法。首先利用极大熵函数将无约束和有约束极小极大问题转化为一个光滑函数的无约束最优化问题,将此光滑函数作为粒子群算法的适应值函数;然后用数学中的外推方法给出一个新的粒子位置更新公式,并应用这个改进的粒子群算法来优化此问题。数值结果表明,该算法收敛快﹑数值稳定性好,是求解非线性极小极大问题的一种有效算法。     

Applying multi-objective ant colony optimization algorithm for solving the unequal area facility layout problems

The unequal area facility layout problem (UA-FLP) which deals with the layout of departments in a facility comprises of a class of extremely difficult and widely applicable multi-objective optimization problems with constraints arising in diverse areas and meeting the requirements for real-world applications. Based on the heuristic strategy, the problem is first converted into an unconstrained optimization problem. Then, we use a modified version of the multi-objective ant colony optimization (MOACO) algorithm which is a heuristic global optimization algorithm and has shown promising performances in solving many optimization problems to solve the multi-objective UA-FLP. In the modified MOACO algorithm, the ACO with heuristic layout updating strategy which is proposed to update the layouts and add the diversity of solutions is a discrete ACO algorithm, with a difference from general ACO algorithms for discrete domains which perform an incremental construction of solutions but the ACO in this paper does not. We propose a novel pheromone update method and combine the Pareto optimization based on the local pheromone communication and the global search based on the niche technology to obtain Pareto-optimal solutions of the problem. In addition, the combination of the local search based on the adaptive gradient method and the heuristic department deformation strategy is applied to deal with the non-overlapping constraint between departments so as to obtain feasible solutions. Ten benchmark instances from the literature are tested. The experimental results show that the proposed MOACO algorithm is an effective method for solving the UA-FLP.