Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions

This paper presents a new algorithm for derivative-free optimization of expensive black-box objective functions subject to expensive black-box inequality constraints. The proposed algorithm, called ConstrLMSRBF, uses radial basis function (RBF) surrogate models and is an extension of the Local Metric Stochastic RBF (LMSRBF) algorithm by Regis and Shoemaker (2007a) [1] that can handle black-box inequality constraints. Previous algorithms for the optimization of expensive functions using surrogate models have mostly dealt with bound constrained problems where only the objective function is expensive, and so, the surrogate models are used to approximate the objective function only. In contrast, ConstrLMSRBF builds RBF surrogate models for the objective function and also for all the constraint functions in each iteration, and uses these RBF models to guide the selection of the next point where the objective and constraint functions will be evaluated. Computational results indicate that ConstrLMSRBF is better than alternative methods on 9 out of 14 test problems and on the MOPTA08 problem from the automotive industry (Jones, 2008 [2]). The MOPTA08 problem has 124 decision variables and 68 inequality constraints and is considered a large-scale problem in the area of expensive black-box optimization. The alternative methods include a Mesh Adaptive Direct Search (MADS) algorithm (Abramson and Audet, 2006 [3]; Audet and Dennis, 2006 [4]) that uses a kriging-based surrogate model, the Multistart LMSRBF algorithm by Regis and Shoemaker (2007a) [1] modified to handle black-box constraints via a penalty approach, a genetic algorithm, a pattern search algorithm, a sequential quadratic programming algorithm, and COBYLA (Powell, 1994 [5]), which is a derivative-free trust-region algorithm. Based on the results of this study, the results in Jones (2008) [2] and other approaches presented at the ISMP 2009 conference, ConstrLMSRBF appears to be among the best, if not the best, known algorithm for the MOPTA08 problem in the sense of providing the most improvement from an initial feasible solution within a very limited number of objective and constraint function evaluations.     

A surrogate-assisted evolution strategy for constrained multi-objective optimization

In many real-world optimization problems, several conflicting objectives must be achieved and optimized simultaneously and the solutions are often required to satisfy certain restrictions or constraints. Moreover, in some applications, the numerical values of the objectives and constraints are obtained from computationally expensive simulations. Many multi-objective optimization algorithms for continuous optimization have been proposed in the literature and some have been incorporated or used in conjunction with expert and intelligent systems. However, relatively few of these multi-objective algorithms handle constraints, and even fewer, use surrogates to approximate the objective or constraint functions when these functions are computationally expensive. This paper proposes a surrogate-assisted evolution strategy (ES) that can be used for constrained multi-objective optimization of expensive black-box objective functions subject to expensive black-box inequality constraints. Such an algorithm can be incorporated into an intelligent system that finds approximate Pareto optimal solutions to simulation-based constrained multi-objective optimization problems in various applications including engineering design optimization, production management and manufacturing. The main idea in the proposed algorithm is to generate a large number of trial offspring in each generation and use the surrogates to predict the objective and constraint function values of these trial offspring. Then the algorithm performs an approximate non-dominated sort of the trial offspring based on the predicted objective and constraint function values, and then it selects the most promising offspring (those with the smallest predicted ranks from the non-dominated sort) to become the actual offspring for the current generation that will be evaluated using the expensive objective and constraint functions. The proposed method is implemented using cubic radial basis function (RBF) surrogate models to assist the ES. The resulting RBF-assisted ES is compared with the original ES and to NSGA-II on 20 test problems involving 2–15 decision variables, 2–5 objectives and up to 13 inequality constraints. These problems include well-known benchmark problems and application problems in manufacturing and robotics. The numerical results showed that the RBF-assisted ES generally outperformed the original ES and NSGA-II on the problems used when the computational budget is relatively limited. These results suggest that the proposed surrogate-assisted ES is promising for computationally expensive constrained multi-objective optimization.     

SO-MI: A surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems

This paper introduces a surrogate model based algorithm for computationally expensive mixed-integer black-box global optimization problems with both binary and non-binary integer variables that may have computationally expensive constraints. The goal is to find accurate solutions with relatively few function evaluations. A radial basis function surrogate model (response surface) is used to select candidates for integer and continuous decision variable points at which the computationally expensive objective and constraint functions are to be evaluated. In every iteration multiple new points are selected based on different methods, and the function evaluations are done in parallel. The algorithm converges to the global optimum almost surely. The performance of this new algorithm, SO-MI, is compared to a branch and bound algorithm for nonlinear problems, a genetic algorithm, and the NOMAD (Nonsmooth Optimization by Mesh Adaptive Direct Search) algorithm for mixed-integer problems on 16 test problems from the literature (constrained, unconstrained, unimodal and multimodal problems), as well as on two application problems arising from structural optimization, and three application problems from optimal reliability design. The numerical experiments show that SO-MI reaches significantly better results than the other algorithms when the number of function evaluations is very restricted (200–300 evaluations).     

Multipoint cubic surrogate function for sequential approximate optimization

Multipoint cubic approximations are investigated as surrogate functions for nonlinear objective and constraint functions in the context of sequential approximate optimization. The proposed surrogate functions match actual function and gradient values, including the current expansion point, thus satisfying the zero and first-order necessary conditions for global convergence to a local minimum of the original problem. Function and gradient information accumulated from multiple design points during the iteration history is used in estimating a reduced Hessian matrix and selected cubic terms in a design subspace appropriate for problems with many design variables. The resulting approximate response surface promises to accelerate convergence to an optimal design within the framework of a trust region algorithm. The hope is to realize computational savings in solving large numerical optimization problems. Numerical examples demonstrate the effectiveness of the new multipoint surrogate function in reducing errors over large changes in design variables.     

基于代理模型的差分进化约束优化

针对耗时计算目标函数的约束优化问题,提出用代理模型来代替耗时计算目标函数的方法,并结合目标函数的信息对约束个体进行选择,从而提出基于代理模型的差分进化约束优化算法。首先,采用拉丁超立方采样方法建立初始种群,用耗时计算目标函数对初始种群进行评估,并以此为样本数据建立目标函数的神经网络代理模型。然后,用差分进化方法为种群中的每一个亲本产生后代,并对后代使用代理模型进行评估,采用可行性规则来比较后代与其亲本并更新种群,根据替换机制将种群中较劣的个体替换为备用存档中较优的个体。最后,当达到最大适应度评估次数时算法停止,给出最优解。该算法与对比算法在10个测试函数上运行的结果表明,该算法得出的结果更精确。将该算法应用于工字梁优化问题的结果表明,相较于优化前的算法,该算法的适应度评估次数减少了80%;相对于FROFI（Feasibility Rule with the incorporation of Objective Function Information）算法,该算法的适应度评估次数减少了36%。运用所提算法进行优化可以有效减少调用耗时计算目标函数的次数,提升优化效率,节约计算成本。     

Surrogate-enhanced evolutionary annealing simplex algorithm for effective and efficient optimization of water resources problems on a budget

In water resources optimization problems, the objective function usually presumes to first run a simulation model and then evaluate its outputs. However, long simulation times may pose significant barriers to the procedure. Often, to obtain a solution within a reasonable time, the user has to substantially restrict the allowable number of function evaluations, thus terminating the search much earlier than required. A promising strategy to address these shortcomings is the use of surrogate modeling techniques. Here we introduce the Surrogate-Enhanced Evolutionary Annealing-Simplex (SEEAS) algorithm that couples the strengths of surrogate modeling with the effectiveness and efficiency of the evolutionary annealing-simplex method. SEEAS combines three different optimization approaches (evolutionary search, simulated annealing, downhill simplex). Its performance is benchmarked against other surrogate-assisted algorithms in several test functions and two water resources applications (model calibration, reservoir management). Results reveal the significant potential of using SEEAS in challenging optimization problems on a budget.     

A RBF-based constrained global optimization algorithm for problems with computationally expensive objective and constraints

This paper presents a metamodel-based constrained optimization method, called Radial basis function-based Constrained Global Optimization (RCGO), to solve optimization problems involving computationally expensive objective function and inequality constraints. RCGO is an extension of the adaptive metamodel-based global optimization (AMGO) algorithm which can handle unconstrained black-box optimization problems. Firstly, a sequential sampling method is implemented to obtain the initial points for building the radial basis function (RBF) approximations to all computational expensive functions while enforcing a feasible solution. Then, an auxiliary objective function subject to the approximate constraints is constructed to determine the next iterative point and further improve the solution. During the process, a distance function with a group of exponents is introduced in the auxiliary function to balance the local exploitation and the global exploration. The RCGO method is tested on a series of benchmark problems, and the results demonstrate that RCGO needs fewer costly evaluations and can be applied for costly constrained problems with all infeasible start points. And the test results on the 30D problems demonstrate that RCGO has advantages in solving the problems. The proposed method is then applied to the design of a cycloid gear pump and desirable results are obtained.     

贝叶斯优化方法和应用综述

设计类问题在科学研究和工业领域无处不在.作为一种十分有效的全局优化算法,近年来,贝叶斯优化方法在设计类问题上被广泛应用.通过设计恰当的概率代理模型和采集函数,贝叶斯优化框架只需经过少数次目标函数评估即可获得理想解,非常适用于求解目标函数表达式未知、非凸、多峰和评估代价高昂的复杂优化问题.从方法论和应用领域两方面深入分析、讨论和展望了贝叶斯优化的研究现状、面临的问题和应用领域,期望为相关领域的研究者提供有益的借鉴和参考.     

CONORBIT: constrained optimization by radial basis function interpolation in trust regions†

This paper presents CONORBIT (CONstrained Optimization by Radial Basis function Interpolation in Trust regions), a derivative-free algorithm for constrained black-box optimization where the objective and constraint functions are computationally expensive. CONORBIT employs a trust-region framework that uses interpolating radial basis function (RBF) models for the objective and constraint functions, and is an extension of the ORBIT algorithm [S.M. Wild, R.G. Regis and C.A. Shoemaker, ORBIT: optimization by radial basis function interpolation in trust-regions, SIAM J. Sci. Comput. 30 (2008), pp. 3197–3219]. It uses a small margin for the RBF constraint models to facilitate the generation of feasible iterates, and extensive numerical tests confirm that such a margin is helpful in improving performance. CONORBIT is compared with other algorithms on 27 test problems, a chemical process optimization problem, and an automotive application. Numerical results show that CONORBIT performs better than COBYLA (Powell 1994), a sequential penalty derivative-free method, an augmented Lagrangian method, a direct search method, and another RBF-based algorithm on the test problems and on the automotive application.     

求解约束优化问题的复合人工蜂群算法

针对约束优化问题,提出一种复合人工蜂群算法。该算法引入多维随机变异操作和最优引导变异操作平衡算法的探索能力和开发能力。将[ε]约束和可行性规则相结合平衡目标函数与约束,加快算法的收敛。通过对CEC 2006中20个测试函数和CEC 2010中18个测试函数及3个实际工程优化问题的实验结果分析表明,该算法对约束优化问题可行有效。     

Automatic design of marine structures by using successive response surface method

An automated optimization procedure based on successive response surface method is presented. The method is applied to weight optimization of a stiffened plate used in marine structures. In the design space the surrogate model is spanned sequentially into an optimally restricted subspace that is converging towards at least a local optimum. Both objective function and all constraint functions are modeled using linear response surface method enabling the use of a robust and efficient simplex algorithm for the optimizations. Special attention is paid to CAD and FEM-model linking that plays a central role in practical industrial applications. In this project SOLIDWORKS and ANSYS software are adopted for structural modeling and analysis, respectively, and the optimization is carried out in a MatLab environment. The reported results achieved in this project prove the robustness and effectiveness of the proposed approach.     

Handling expensive multi-objective optimization problems with a cluster-based neighborhood regression model

This paper gives attention to multi-objective optimization in scenarios where objective function evaluation is expensive, that is, expensive multi-objective optimization. We firstly propose a cluster-based neighborhood regression model, which incorporates the linear regression technique to predict the descent direction and generate new potential offspring. Combining this model with the classical decomposition-based multi-objective optimization framework, we propose an efficient and effective algorithm for tackling computationally expensive multi-objective optimization problems. As opposed to the conventional approach of replacing the original time-consuming objective functions with the approximated ones obtained by surrogate model, the proposed algorithm incorporates the proposed regression model to serve as an operator producing higher-quality offspring so that the algorithm requires fewer iterations to reach a given solution quality. The proposed algorithm is compared with several state-of-the-art surrogate-assisted algorithms on a variety of well-known benchmark problems. Empirical results demonstrate that the proposed algorithm outperforms or is competitive with other peer algorithms, and has the ability to keep a good trade-off between solution quality and running time within a fairly small number of function evaluations. In particular, our proposed algorithm shows obvious superiority in terms of the computational time used for the algorithm components, and can obtain acceptable solutions for expensive problems with high efficiency.     

约束进化算法及其应用研究综述

约束优化问题广泛存在于科学研究和工程实践中,其对应的约束优化进化算法也成为了进化领域的重要研究方向。约束优化进化算法的本质问题是如何有效地利用不可行解和可行解的信息,平衡目标函数和约束条件,使得算法更加高效。首先对约束优化问题进行定义;然后详细分析了目前主流的约束进化算法,同时,基于不同的约束处理机制,将这些机制分为约束和目标分离法、惩罚函数法、多目标优化法、混合法和其他算法,并对这些方法进行了详细的分析和总结;接着指出约束进化算法亟待解决的问题,并明确指出未来需要进一步研究的方向;最后对约束进化算法在工程优化、电子和通信工程、机械设计、环境资源配置、科研领域和管理分配等方面的应用进行了介绍。     

Surrogate modeling based on an adaptive network and granular computing

Reducing the number of evaluations of expensive fitness functions is one of the main concerns in evolutionary algorithms, especially when working with instances of contemporary engineering problems. As an alternative to this efficiency constraint, surrogate-based methods are grounded in the construction of approximate models that estimate the solutions’ fitness by modeling the relationships between solution variables and their performance. This paper proposes a methodology based on granular computing for the construction of surrogate models for evolutionary algorithms. Under the proposed method, granules are associated with representative solutions of the problem under analysis. New solutions are evaluated with the expensive (original) fitness function only if they are not already covered by an existing granule. The parameters defining granules are periodically adapted as the search goes on using a neuro-fuzzy network that does not only reduce the number of fitness function evaluations, but also provides better convergence capabilities. The proposed method is evaluated on classical benchmark functions and on a recent benchmark created to test large-scale optimization models. Our results show that the proposed method considerably reduces the actual number of fitness function evaluations without significantly degrading the quality of solutions.     

Modified harmony search optimization for constrained design problems

Constrained optimization is a major real-world problem. Constrained optimization problems consist of an objective function subjected to both linear and nonlinear constraints. Here a constraint handling procedure based on the fitness priority-based ranking method (FPBRM) is proposed. It is embedded into a harmony search (HS) algorithm that allows it to satisfy constraints. The HS algorithm is conceptualized using the musical process of searching for a perfect state of harmony. Here, the original heuristic HS was improved by combining both improved and global-best methods along with the FPBRM. The resulting modified harmony search (MHS) was then compared with the original HS technique and other optimization methods for several test problems.     

A new hybrid particle swarm and simulated annealing stochastic optimization method

A novel hybrid particle swarm and simulated annealing stochastic optimization method is proposed. The proposed hybrid method uses both PSO and SA in sequence and integrates the merits of good exploration capability of PSO and good local search properties of SA. Numerical simulation has been performed for selection of near optimum parameters of the method. The performance of this hybrid optimization technique was evaluated by comparing optimization results of thirty benchmark functions of different dimensions with those obtained by other numerical methods considering three criteria. These criteria were stability, average trial function evaluations for successful runs and the total average trial function evaluations considering both successful and failed runs. Design of laminated composite materials with required effective stiffness properties and minimum weight design of a three-bar truss are addressed as typical applications of the proposed algorithm in various types of optimization problems. In general, the proposed hybrid PSO-SA algorithm demonstrates improved performance in solution of these problems compared to other evolutionary methods The results of this research show that the proposed algorithm can reliably and effectively be used for various optimization problems.     

一种非参数惩罚函数的优化演化算法

对约束优化问题的处理通常使用惩罚函数法,使用普通惩罚函数法的困难存在于参数的选取。该文提出一种基于演化算法的非参数罚函数算法,对违反约束条件动态地进行惩罚,由适应值的设定来平衡群体中可行解和不可行解的比例,使群体较好地向最优解逼近,使用实数编码的多父体单形杂交演化策略来实现新算法,通过对测试函数的检验,该算法具有稳健、高效、简洁易于实现等特点。     

异构集成代理辅助多目标粒子群优化算法

针对代理辅助进化算法在减少昂贵适应度评估时难以通过少量样本点构造高质量代理模型的问题,提出异构集成代理辅助多目标粒子群优化算法。该方法通过使用加权平均法将Kriging模型和径向基函数网络模型组合成高精度的异构集成模型,达到增强算法处理不确定性信息能力的目的。基于集成学习的两种代理模型分别应用于全局搜索和局部搜索,在多目标粒子群优化算法框架基础上,新提出的方法为每个目标函数自适应地构造了异构集成模型,利用其模型的非支配解来指导粒子群的更新,得出目标函数的最优解集。实验结果表明,所提方法提高了代理模型的搜索能力,减少了评估次数,并且随着搜索维度的增加,其计算复杂性也具有更好的可扩展性。     

蒸馏与换热协同的约束多目标在线操作优化方法

针对蒸馏装置与换热网络间缺乏协同优化导致的分馏精度差和能耗高的问题,提出了一种基于代理模型的约束多目标在线协同操作优化方法.为了解决蒸馏装置与换热网络操作参数协同优化时存在的计算耗时和约束的问题,构建Kriging代理模型来近似目标函数和约束条件,提出了基于随机欠采样和Adaboost的分类代理模型（RUSBoost）来解决类别不平衡的收敛判定预测问题.提出了基于多阶段自适应约束处理的代理模型的模型管理方法,该方法采用基于参考向量激活状态的最大化改善期望准则和可行概率准则更新机制来平衡优化初始阶段种群的多样性和可行性,采用支配参考点的置信下限准则更新机制加快收敛速度.通过不断与机理模型交互来在线更新代理模型,实现在线操作优化.通过测试函数和仿真实例验证了本文方法的有效性.     

Surrogate assisted‐hybrid differential evolution algorithm using diversity control

Evolutionary algorithms (EAs) being a major optimization framework, typically require a considerable number of function evaluations to locate an optimal solution for computationally intensive real‐world optimization problems. In order to solve complex multimodal problems within a limited computational budget, surrogate models are integrated with EA. The overall performance of such algorithms not only depends on the construction and integration procedure of the model, but also on the efficiency of the underlying EA in overcoming premature convergence. This can be achieved through diversity control and parameter adaptation methodology in EAs. In this paper, an improved algorithm, namely Diversity Controlled Parameter adapted Differential Evolution with Local Search (DCPaDE‐LS) is integrated into two dynamic surrogate models and two variants, namely Surrogate Assisted Parameter adapted Differential Evolution with Artificial Neural Networks and Response Surface Methodology (SAPDE‐ANN and SAPDE‐RSM) are proposed. They reduce the exact function evaluations for complex, multimodal problems. The performance of the proposed variants are compared on a set of 26 bound‐constrained benchmark functions scalable with 10 and 30 dimensions, with respect to average number of function evaluations (NFE), success rate (SR) and % reduction in NFE in 30 independent trials. The SAPDE variants are compared with Self‐adaptive Differential Evolution, DCPaDE‐LS, Increasing Population Size Covariance Matrix Adaptation Evolution Strategy and Comprehensive Learning Particle Swarm Optimization. The SAPDE variants are able to reduce the NFE without loss in SR for all the functions. The algorithms are also validated using 12 solvable functions from CEC 2005. Of the two variants, SAPDE‐ANN reports reduced NFE in more functions compared with SAPDE‐RSM. Results reveal that the proposed SAPDE algorithm can be applied to real‐world optimization problems.