CONORBIT: constrained optimization by radial basis function interpolation in trust regions†

This paper presents CONORBIT (CONstrained Optimization by Radial Basis function Interpolation in Trust regions), a derivative-free algorithm for constrained black-box optimization where the objective and constraint functions are computationally expensive. CONORBIT employs a trust-region framework that uses interpolating radial basis function (RBF) models for the objective and constraint functions, and is an extension of the ORBIT algorithm [S.M. Wild, R.G. Regis and C.A. Shoemaker, ORBIT: optimization by radial basis function interpolation in trust-regions, SIAM J. Sci. Comput. 30 (2008), pp. 3197–3219]. It uses a small margin for the RBF constraint models to facilitate the generation of feasible iterates, and extensive numerical tests confirm that such a margin is helpful in improving performance. CONORBIT is compared with other algorithms on 27 test problems, a chemical process optimization problem, and an automotive application. Numerical results show that CONORBIT performs better than COBYLA (Powell 1994), a sequential penalty derivative-free method, an augmented Lagrangian method, a direct search method, and another RBF-based algorithm on the test problems and on the automotive application.     

Self-adjusting parameter control for surrogate-assisted constrained optimization under limited budgets

Constrained optimization of high-dimensional numerical problems plays an important role in many scientific and industrial applications. Function evaluations in many industrial applications are severely limited and often only little analytical information about objective function and constraint functions is available. For such expensive black-box optimization tasks, the constraint optimization algorithm COBRA (Constrained Optimization By Radial Basis Function Approximation) was proposed, making use of RBF (radial basis function) surrogate modeling for both objective and constraint functions. COBRA has shown remarkable success in solving reliably complex benchmark problems in less than 500 function evaluations. Unfortunately, COBRA requires careful adjustment of parameters in order to do so.In this work we present a new algorithm SACOBRA (Self-Adjusting COBRA), which is based on COBRA and capable of achieving high-quality results with very few function evaluations and no parameter tuning. It is shown with the help of performance profiles on a set of benchmark problems (G-problems, MOPTA08) that SACOBRA consistently outperforms COBRA algorithms with different fixed parameter settings. We analyze the importance of the new elements in SACOBRA and show that each element of SACOBRA plays a role to boost up the overall optimization performance. We discuss the reasons and get in this way a better understanding of high-quality RBF surrogate modeling.     

A surrogate-assisted evolution strategy for constrained multi-objective optimization

In many real-world optimization problems, several conflicting objectives must be achieved and optimized simultaneously and the solutions are often required to satisfy certain restrictions or constraints. Moreover, in some applications, the numerical values of the objectives and constraints are obtained from computationally expensive simulations. Many multi-objective optimization algorithms for continuous optimization have been proposed in the literature and some have been incorporated or used in conjunction with expert and intelligent systems. However, relatively few of these multi-objective algorithms handle constraints, and even fewer, use surrogates to approximate the objective or constraint functions when these functions are computationally expensive. This paper proposes a surrogate-assisted evolution strategy (ES) that can be used for constrained multi-objective optimization of expensive black-box objective functions subject to expensive black-box inequality constraints. Such an algorithm can be incorporated into an intelligent system that finds approximate Pareto optimal solutions to simulation-based constrained multi-objective optimization problems in various applications including engineering design optimization, production management and manufacturing. The main idea in the proposed algorithm is to generate a large number of trial offspring in each generation and use the surrogates to predict the objective and constraint function values of these trial offspring. Then the algorithm performs an approximate non-dominated sort of the trial offspring based on the predicted objective and constraint function values, and then it selects the most promising offspring (those with the smallest predicted ranks from the non-dominated sort) to become the actual offspring for the current generation that will be evaluated using the expensive objective and constraint functions. The proposed method is implemented using cubic radial basis function (RBF) surrogate models to assist the ES. The resulting RBF-assisted ES is compared with the original ES and to NSGA-II on 20 test problems involving 2–15 decision variables, 2–5 objectives and up to 13 inequality constraints. These problems include well-known benchmark problems and application problems in manufacturing and robotics. The numerical results showed that the RBF-assisted ES generally outperformed the original ES and NSGA-II on the problems used when the computational budget is relatively limited. These results suggest that the proposed surrogate-assisted ES is promising for computationally expensive constrained multi-objective optimization.     

A reliability-based optimization method using sequential surrogate model and Monte Carlo simulation

This paper presents a sequential surrogate model method for reliability-based optimization (SSRBO), which aims to reduce the number of the expensive black-box function calls in reliability-based optimization. The proposed method consists of three key steps. First, the initial samples are selected to construct radial basis function surrogate models for the objective and constraint functions, respectively. Second, by solving a series of special optimization problems in terms of the surrogate models, local samples are identified and added in the vicinity of the current optimal point to refine the surrogate models. Third, by solving the optimization problem with the shifted constraints, the current optimal point is obtained. Then, at the current optimal point, the Monte Carlo simulation based on the surrogate models is carried out to obtain the cumulative distribution functions (CDFs) of the constraints. The CDFs and target reliabilities are used to update the offsets of the constraints for the next iteration. Therefore, the original problem is decomposed to serial cheap surrogate-based deterministic problems and Monte Carlo simulations. Several examples are adopted to verify SSRBO. The results show that the number of the expensive black-box function calls is reduced exponentially without losing of precision compared to the alternative methods, which illustrates the efficiency and accuracy of the proposed method.

     

SO-MI: A surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems

This paper introduces a surrogate model based algorithm for computationally expensive mixed-integer black-box global optimization problems with both binary and non-binary integer variables that may have computationally expensive constraints. The goal is to find accurate solutions with relatively few function evaluations. A radial basis function surrogate model (response surface) is used to select candidates for integer and continuous decision variable points at which the computationally expensive objective and constraint functions are to be evaluated. In every iteration multiple new points are selected based on different methods, and the function evaluations are done in parallel. The algorithm converges to the global optimum almost surely. The performance of this new algorithm, SO-MI, is compared to a branch and bound algorithm for nonlinear problems, a genetic algorithm, and the NOMAD (Nonsmooth Optimization by Mesh Adaptive Direct Search) algorithm for mixed-integer problems on 16 test problems from the literature (constrained, unconstrained, unimodal and multimodal problems), as well as on two application problems arising from structural optimization, and three application problems from optimal reliability design. The numerical experiments show that SO-MI reaches significantly better results than the other algorithms when the number of function evaluations is very restricted (200–300 evaluations).     

Newton-conjugate gradient （CG） augmented Lagrangian method for path constrained dynamic process optimization

In this paper, a Newton-conjugate gradient (CG) augmented Lagrangian method is proposed for solving the path constrained dynamic process optimization problems. The path constraints are simplified as a single final time constraint by using a novel constraint aggregation function. Then, a control vector parameterization (CVP) approach is applied to convert the constraints simplified dynamic optimization problem into a nonlinear programming (NLP) problem with inequality constraints. By constructing an augmented Lagrangian function, the inequality constraints are introduced into the augmented objective function, and a box constrained NLP problem is generated. Then, a linear search Newton-CG approach, also known as truncated Newton (TN) approach, is applied to solve the problem. By constructing the Hamiltonian functions of objective and constraint functions, two adjoint systems are generated to calculate the gradients which are needed in the process of NLP solution. Simulation examples demonstrate the effectiveness of the algorithm.     

Numerical experience with a derivative-free trust-funnel method for nonlinear optimization problems with general nonlinear constraints

A trust-funnel method is proposed for solving nonlinear optimization problems with general nonlinear constraints. It extends the one presented by Gould and Toint [Nonlinear programming without a penalty function or a filter. Math. Prog. 122(1):155–196, 2010], originally proposed for equality-constrained optimization problems only, to problems with both equality and inequality constraints and where simple bounds are also considered. As the original one, our method makes use of neither filter nor penalty functions and considers the objective function and the constraints as independently as possible. To handle the bounds, an active-set approach is employed. We then exploit techniques developed for derivative-free optimization (DFO) to obtain a method that can also be used to solve problems where the derivatives are unavailable or are available at a prohibitive cost. The resulting approach extends the DEFT-FUNNEL algorithm presented by Sampaio and Toint [A derivative-free trust-funnel method for equality-constrained nonlinear optimization. Comput. Optim. Appl. 61(1):25–49, 2015], which implements a derivative-free trust-funnel method for equality-constrained problems. Numerical experiments with the extended algorithm show that our approach compares favourably to other well-known model-based algorithms for DFO.     

A RBF-based constrained global optimization algorithm for problems with computationally expensive objective and constraints

This paper presents a metamodel-based constrained optimization method, called Radial basis function-based Constrained Global Optimization (RCGO), to solve optimization problems involving computationally expensive objective function and inequality constraints. RCGO is an extension of the adaptive metamodel-based global optimization (AMGO) algorithm which can handle unconstrained black-box optimization problems. Firstly, a sequential sampling method is implemented to obtain the initial points for building the radial basis function (RBF) approximations to all computational expensive functions while enforcing a feasible solution. Then, an auxiliary objective function subject to the approximate constraints is constructed to determine the next iterative point and further improve the solution. During the process, a distance function with a group of exponents is introduced in the auxiliary function to balance the local exploitation and the global exploration. The RCGO method is tested on a series of benchmark problems, and the results demonstrate that RCGO needs fewer costly evaluations and can be applied for costly constrained problems with all infeasible start points. And the test results on the 30D problems demonstrate that RCGO has advantages in solving the problems. The proposed method is then applied to the design of a cycloid gear pump and desirable results are obtained.     

Distributed continuous‐time algorithm for a general nonsmooth monotropic optimization problem

This paper investigates a general monotropic optimization problem for continuous‐time networks, where the global objective function is a sum of local objective functions that are only known to individual agent, and general constraints are taken into account, including local inequality constraints, global equality constraint, and local feasible constraints. In addition, all functions involved in the objective functions and inequality constraints are not necessarily differentiable. To solve the problem, a distributed continuous‐time algorithm is designed using subgradient projections, and it is shown that the proposed algorithm is well defined in the sense that the existence of its solutions can be guaranteed. Furthermore, it is proved that the algorithm converges to an optimal solution for the general monotropic optimization problem. Finally, a simulation example is provided for validating the theoretical result.     

A derivative-free affine scaling trust region methods based on probabilistic models with new nonmonotone line search technique for linear inequality constrained minimization without strict complementarity

ABSTRACT

In this paper, a derivative-free trust region methods based on probabilistic models with new nonmonotone line search technique is considered for nonlinear programming with linear inequality constraints. The proposed algorithm is designed to build probabilistic polynomial interpolation models for the objective function. We build the affine scaling trust region methods which use probabilistic or random models within a classical trust region framework. The new backtracking linear search technique guarantee the descent of the objective function, and new iterative points are in the feasible region. In order to overcome the strict complementarity hypothesis, under some reasonable conditions which are weaker than strong second order sufficient condition, we give the new and more simple identification function to structure the affine matrix. The global and local fast convergence of the algorithm are shown and the results of numerical experiments are reported to show the effectiveness of the proposed algorithm.     

一种改进的RBF全局优化方法

径向基函数（Radial Basis Functions）由于具有良好的近似效果和运算简单的特点,被应用于全局优化中,成为解决黑箱函数全局优化问题的有效方法。然而现有的基于RBF的全局优化算法存在迭代过程中RBF模型重构效率低下,以及采样方法不合理导致函数估值次数过多等问题。在此提出几个改进思路：采用基于矩阵分块的增量RBF方法以减少模型重构时间提高效率;采用增量LHD采样方法以确保具有更好的空间填充性;采用算法重启策略以降低估值次数。通过实验验证改进方法的优势。     

A heuristic method for simultaneous tower and pattern-free field optimization on solar power systems

A heuristic method for optimizing a solar power tower system is proposed, in which both heliostat field (heliostat locations and number) and the tower (tower height and receiver size) are simultaneously considered.Maximizing the thermal energy collected per unit cost leads to a difficult optimization problem due to its characteristics: it has a nonconvex black-box objective function with computationally expensive evaluation and nonconvex constraints.The proposed method sequentially optimizes the field layout for a given tower configuration and then, the tower design is optimized for the previously obtained field layout. A greedy-based heuristic algorithm is presented to address the heliostat location problem. This algorithm follows a pattern-free method. The only constraints to be considered are the field region and the nonconvex constraints (which allow heliostats to not collide).The absence of a geometrical pattern to design the field and the simultaneous optimization of the field and the tower designs make this approach different from the existing ones. Our method is compared against other proposals in the literature of heliostat field optimization.     

An affine-scaling derivative-free trust-region method for solving nonlinear systems subject to linear inequality constraints

In this paper, an affine-scaling derivative-free trust-region method with interior backtracking line search technique is considered for solving nonlinear systems subject to linear inequality constraints. The proposed algorithm is designed to take advantage of the problem structured by building polynomial interpolation models for each function in the nonlinear system function F. The proposed approach is developed by forming a quadratic model with an appropriate quadratic function and scaling matrix: there is no need to handle the constraints explicitly. By using both trust-region strategy and interior backing line search technique, each iteration switches to backtracking step generated by the trust-region subproblem and satisfies strict interior point feasibility by line search backtracking technique. Under reasonable conditions, the global convergence and fast local convergence rate of the proposed algorithm are established. The results of numerical experiments are reported to show the effectiveness of the proposed algorithms.     

Multi-surrogate-based global optimization using a score-based infill criterion

This paper presents a new global optimization algorithm named MGOSIC to solve unconstrained expensive black-box optimization problems. In MGOSIC, three surrogate models Kriging, Radial Basis Function (RBF), and Quadratic Response Surfaces (QRS) are dynamically constructed, respectively. Additionally, a multi-point infill criterion is proposed to obtain new points in each cycle, where a score-based strategy is presented to mark cheap points generated by Latin hypercube sampling. According to their predictive values from the three surrogate models, the promising cheap points are assigned with different scores. In order to obtain the samples with diversity, a Max-Min approach is proposed to select promising sample points from the cheap point sets with higher scores. Simultaneously, the best solutions predicted by Kriging, RBF, and QRS are also recorded as supplementary samples, respectively. Once MGOSIC gets stuck in a local valley, the estimated mean square error of Kriging will be maximized to explore the sparsely sampled regions. Moreover, the whole optimization algorithm is carried out alternately in the global space and a reduced space. In summary, MGOSIC not only brings a new idea for multi-point sampling, but also builds a reasonable balance between exploitation and exploration. Finally, 19 mathematical benchmark cases and an engineering application of hydrofoil optimization are used to test MGOSIC. Furthermore, seven existing global optimization algorithms are also tested as contrast. The final results show that MGOSIC has high efficiency, strong stability, and better multi-point sampling capability in dealing with expensive black-box optimization problems.

     

广义支持向量机的多项式光滑函数法

为了求解广义支持向量机（GSVM）的优化问题,将带有不等式约束的原始优化问题转化为无约束优化问题,由于此无约束优化问题的目标函数不光滑,所以引入一族多项式光滑函数进行逼近,实验中可以根据不同的精度要求选择不同的逼近函数。用BFGS算法求解。实验结果表明,该算法和已有的GSVM的求解算法相比,更快地获得了更高的测试精度,更适合大规模数据集的训练。因此给出的GSVM的求解算法是有效的。     

异构集成代理辅助多目标粒子群优化算法

针对代理辅助进化算法在减少昂贵适应度评估时难以通过少量样本点构造高质量代理模型的问题,提出异构集成代理辅助多目标粒子群优化算法。该方法通过使用加权平均法将Kriging模型和径向基函数网络模型组合成高精度的异构集成模型,达到增强算法处理不确定性信息能力的目的。基于集成学习的两种代理模型分别应用于全局搜索和局部搜索,在多目标粒子群优化算法框架基础上,新提出的方法为每个目标函数自适应地构造了异构集成模型,利用其模型的非支配解来指导粒子群的更新,得出目标函数的最优解集。实验结果表明,所提方法提高了代理模型的搜索能力,减少了评估次数,并且随着搜索维度的增加,其计算复杂性也具有更好的可扩展性。     

Handling expensive multi-objective optimization problems with a cluster-based neighborhood regression model

This paper gives attention to multi-objective optimization in scenarios where objective function evaluation is expensive, that is, expensive multi-objective optimization. We firstly propose a cluster-based neighborhood regression model, which incorporates the linear regression technique to predict the descent direction and generate new potential offspring. Combining this model with the classical decomposition-based multi-objective optimization framework, we propose an efficient and effective algorithm for tackling computationally expensive multi-objective optimization problems. As opposed to the conventional approach of replacing the original time-consuming objective functions with the approximated ones obtained by surrogate model, the proposed algorithm incorporates the proposed regression model to serve as an operator producing higher-quality offspring so that the algorithm requires fewer iterations to reach a given solution quality. The proposed algorithm is compared with several state-of-the-art surrogate-assisted algorithms on a variety of well-known benchmark problems. Empirical results demonstrate that the proposed algorithm outperforms or is competitive with other peer algorithms, and has the ability to keep a good trade-off between solution quality and running time within a fairly small number of function evaluations. In particular, our proposed algorithm shows obvious superiority in terms of the computational time used for the algorithm components, and can obtain acceptable solutions for expensive problems with high efficiency.     

基于代理模型的差分进化约束优化

针对耗时计算目标函数的约束优化问题,提出用代理模型来代替耗时计算目标函数的方法,并结合目标函数的信息对约束个体进行选择,从而提出基于代理模型的差分进化约束优化算法。首先,采用拉丁超立方采样方法建立初始种群,用耗时计算目标函数对初始种群进行评估,并以此为样本数据建立目标函数的神经网络代理模型。然后,用差分进化方法为种群中的每一个亲本产生后代,并对后代使用代理模型进行评估,采用可行性规则来比较后代与其亲本并更新种群,根据替换机制将种群中较劣的个体替换为备用存档中较优的个体。最后,当达到最大适应度评估次数时算法停止,给出最优解。该算法与对比算法在10个测试函数上运行的结果表明,该算法得出的结果更精确。将该算法应用于工字梁优化问题的结果表明,相较于优化前的算法,该算法的适应度评估次数减少了80%;相对于FROFI（Feasibility Rule with the incorporation of Objective Function Information）算法,该算法的适应度评估次数减少了36%。运用所提算法进行优化可以有效减少调用耗时计算目标函数的次数,提升优化效率,节约计算成本。     

A statistical model-based algorithm for ‘black-box’ multi-objective optimisation

The problem of multi-objective optimisation with ‘expensive’ ‘black-box’ objective functions is considered. An algorithm is proposed that generalises the single objective P-algorithm constructed using the statistical model of multimodal functions and concepts of the theory of rational decisions under uncertainty. Computational examples are included demonstrating that the algorithm proposed possess several expected properties.     

Improving surrogate-assisted variable fidelity multi-objective optimization using a clustering algorithm

Surrogate-assisted evolutionary optimization has proved to be effective in reducing optimization time, as surrogates, or meta-models can approximate expensive fitness functions in the optimization run. While this is a successful strategy to improve optimization efficiency, challenges arise when constructing surrogate models in higher dimensional function space, where the trade space between multiple conflicting objectives is increasingly complex. This complexity makes it difficult to ensure the accuracy of the surrogates. In this article, a new surrogate management strategy is presented to address this problem. A k-means clustering algorithm is employed to partition model data into local surrogate models. The variable fidelity optimization scheme proposed in the author's previous work is revised to incorporate this clustering algorithm for surrogate model construction. The applicability of the proposed algorithm is illustrated on six standard test problems. The presented algorithm is also examined in a three-objective stiffened panel optimization design problem to show its superiority in surrogate-assisted multi-objective optimization in higher dimensional objective function space. Performance metrics show that the proposed surrogate handling strategy clearly outperforms the single surrogate strategy as the surrogate size increases.