An algorithm for computationally expensive engineering optimization problems

Abstract

Modern engineering design often relies on computer simulations to evaluate candidate designs, a scenario which results in an optimization of a computationally expensive black-box function. In these settings, there will often exist candidate designs which cause the simulation to fail, and can therefore degrade the search effectiveness. To address this issue, this paper proposes a new metamodel-assisted computational intelligence optimization algorithm which incorporates classifiers into the optimization search. The classifiers predict which candidate designs are expected to cause the simulation to fail, and this prediction is used to bias the search towards designs predicted to be valid. To enhance the search effectiveness, the proposed algorithm uses an ensemble approach which concurrently employs several metamodels and classifiers. A rigorous performance analysis based on a set of simulation-driven design optimization problems shows the effectiveness of the proposed algorithm.     

Various approaches for constructing an ensemble of metamodels using local measures

Metamodels are approximate mathematical models used as surrogates for computationally expensive simulations. Since metamodels are widely used in design space exploration and optimization, there is growing interest in developing techniques to enhance their accuracy. It has been shown that the accuracy of metamodel predictions can be increased by combining individual metamodels in the form of an ensemble. Several efforts were focused on determining the contribution (or weight factor) of a metamodel in the ensemble using global error measures. In addition, prediction variance is also used as a local error measure to determine the weight factors. This paper investigates the efficiency of using local error measures, and also presents the use of the pointwise cross validation error as a local error measure as an alternative to using prediction variance. The effectiveness of ensemble models are tested on several problems with varying dimensionality: five mathematical benchmark problems, two structural mechanics problems and an automobile crash problem. It is found that the spatial ensemble models show better performances than the global ensemble for the low-dimensional problems, while the global ensemble is a more accurate model than the spatial ensembles for the high-dimensional problems. Ensembles based on pointwise cross validation error and prediction variance provide similar accuracy. The ensemble models based on local measures reduce cross validation errors drastically, but their performances are not that impressive in reducing the error evaluated at random test points, because the pointwise cross validation error is not a good surrogate for the error at a point.     

Ensemble of metamodels with optimized weight factors

Approximate mathematical models (metamodels) are often used as surrogates for more computationally intensive simulations. The common practice is to construct multiple metamodels based on a common training data set, evaluate their accuracy, and then to use only a single model perceived as the best while discarding the rest. This practice has some shortcomings as it does not take full advantage of the resources devoted to constructing different metamodels, and it is based on the assumption that changes in the training data set will not jeopardize the accuracy of the selected model. It is possible to overcome these drawbacks and to improve the prediction accuracy of the surrogate model if the separate stand-alone metamodels are combined to form an ensemble. Motivated by previous research on committee of neural networks and ensemble of surrogate models, a technique for developing a more accurate ensemble of multiple metamodels is presented in this paper. Here, the selection of weight factors in the general weighted-sum formulation of an ensemble is treated as an optimization problem with the desired solution being one that minimizes a selected error metric. The proposed technique is evaluated by considering one industrial and four benchmark problems. The effect of different metrics for estimating the prediction error at either the training data set or a few validation points is also explored. The results show that the optimized ensemble provides more accurate predictions than the stand-alone metamodels and for most problems even surpassing the previously reported ensemble approaches.     

Metamodel selection based on stepwise regression

Metamodels are often used to replace expensive simulations of engineering problems. When a training set is given, a series of metamodels can be constructed, and then there are two strategies to deal with these metamodels: (1) picking out the best one with the highest accuracy as an approximation of the computationally intensive simulation; and (2) combining all of them into an ensemble model. However, since the choice of approximate model depends on design of experiments (DOEs), employing of the first strategy thus increases the risk of adopting an inappropriate model. Nevertheless, the second strategy also seems not to be a good choice, since adding redundant metamodels may lead to loss of accuracy. Therefore, it is a necessary step to eliminate the redundant metamodels from the set of the candidates before constructing the final ensemble. Illuminated by the method of variable selection widely used in polynomial regression, a metamodel selection method based on stepwise regression is proposed. In our method, just a subset of n ones (n ≤ p, where p is the number of all of the candidate metamodels) is used. In addition, a new ensemble technique is proposed from the view of polynomial regression in this work. This new ensemble technique, combined with metamodel selection method, has been evaluated using six benchmark problems. The results show that eliminating the redundant metamodels before constructing the ensemble can provide more ideal prediction accuracy than directly constructing the ensemble by utilizing all of the candidates.     

Ensemble of metamodels: the augmented least squares approach

In this work we present an approach to create ensemble of metamodels (or weighted averaged surrogates) based on least squares (LS) approximation. The LS approach is appealing since it is possible to estimate the ensemble weights without using any explicit error metrics as in most of the existent ensemble methods. As an additional feature, the LS based ensemble of metamodels has a prediction variance function that enables the extension to the efficient global optimization. The proposed LS approach is a variation of the standard LS regression by augmenting the matrices in such a way that minimizes the effects of multicollinearity inherent to calculation of the ensemble weights. We tested and compared the augmented LS approach with different LS variants and also with existent ensemble methods, by means of analytical and real-world functions from two to forty-four variables. The augmented least squares approach performed with good accuracy and stability for prediction purposes, in the same level of other ensemble methods and has computational cost comparable to the faster ones.     

Ensemble of metamodels: extensions of the least squares approach to efficient global optimization

In this work we present LSEGO, an approach to drive efficient global optimization (EGO), based on LS (least squares) ensemble of metamodels. By means of LS ensemble of metamodels it is possible to estimate the uncertainty of the prediction with any kind of model (not only kriging) and provide an estimate for the expected improvement function. For the problems studied, the proposed LSEGO algorithm has shown to be able to find the global optimum with less number of optimization cycles than required by the classical EGO approach. As more infill points are added per cycle, the faster is the convergence to the global optimum (exploitation) and also the quality improvement of the metamodel in the design space (exploration), specially as the number of variables increases, when the standard single point EGO can be quite slow to reach the optimum. LSEGO has shown to be a feasible option to drive EGO with ensemble of metamodels as well as for constrained problems, and it is not restricted to kriging and to a single infill point per optimization cycle.     

Multi-objective process parameters optimization of hot-wire laser welding using ensemble of metamodels and NSGA-II

Hot-wire laser welding (HLW) can reduce the dispersion of laser energy and improve deposition efficiency by preheating filler wire during laser welding process. In order to obtain sound welding results, it is crucial to select appropriate process parameters. In this study, an optimization methodology based on ensemble metamodels (EM) which take the advantages of the prediction ability of stand-alone metamodels (Kriging, RBF and SVR), and Non-dominated sorting genetic algorithm (NSGA-II) are presented to obtain optimum process parameters during stainless steel 316L hot-wire laser welding. Firstly, EMs are developed through minimizing the generalized mean square Leave-one-out (LOO) errors to find the optimum weight factors of the used stand-alone Kriging, RBF and SVR metamodels. And then the EMs are applied to establish the relationships between process parameters (i,e., laser power (LP), welding speed (WS) and hot-wire current (I)) and welding results (i,e., welding depth-to-width ratio (DW), welding reinforcement (BR) and tensile strength (TS)). During optimization process, NSGA-II is employed to search for multi-objective Pareto optimal solutions based on EMs. In addition, the main effects of multiple process parameters on welding results are analyzed. The verification tests indicate that the obtained optimal process parameters are effective and reliable for producing expected welding results (maximized DW, maximized TS and desired BR value). In general, the proposed optimization method can provide a reliable guidance for HLW in engineering practice.     

On the ensemble of metamodels with multiple regional optimized weight factors

Metamodels are often used as surrogates for expensive high fidelity computational simulations (e.g., finite element analysis). Ensemble of metamodels (EM), which combines various types of individual metamodels in the form of a weighted average ensemble, is found to have improved accuracy over the individual metamodels used alone. Currently, there are mainly two kinds of EMs called as pointwise EM and average EM. The pointwise EM generally has better prediction accuracy than the average EM, but it is much more time-consuming than the average EM. In most cases, as a metamodel, EM is often used in the engineering design optimization which needs to invoke EM tens of thousands of times. Therefore, the average EM is still the most extensively used EM. To the authors’ best knowledge, the most accurate average EM is the EM with optimized weight factors proposed by Acar et al. However, the EM proposed by Acar et al. is often too “rigid” and may not have sufficient accuracy over some regions of the design space. In order to deal with this problem and further improve the prediction accuracy, a new EM with multiple regional optimized weight factors (EM-MROWF) is proposed in this study. In this new EM, the design space is divided into multiple subdomains each of which is assigned a set of optimized weight factors. This new EM was constructed by combining three typical individual metamodels, i.e., polynomial regression (PR), radial basis function (RBF), and Kriging (KRG). The proposed technique was evaluated by ten benchmark problems and two engineering application problems. The ten benchmark problems are typical mathematical functions for evaluating the approximation performance in previous studies. And, the two engineering application problems refer to the vehicular passive safety in the field of crashworthiness design. The study results showed that the EM-MROWF performed much better than the other existing average EMs as well as the three individual metamodels.     

Optimization of laser brazing onto galvanized steel based on ensemble of metamodels

Laser brazing (LB) provides a promising way to join the galvanized steel in automotive industry for its significant advantages including high speed, small heat-affected zone, and high welding seam quality. The process parameters of LB have significant effects on the bead profile and hence the quality of joint. Since the relationships between the process parameters and bead profile cannot be expressed explicitly, it is impractical to determine the optimal process parameters intuitively. This paper proposes an optimization methodology by combining genetic algorithm (GA) and ensemble of metamodels (EMs) to address the process parameters optimization of the bead profile in LB with crimping butt. Firstly, Taguchi experimental design is adopted to generate the experimental points. Secondly, the relationships between process parameters (i.e., welding speed, wire feed rate, gap) and the bead geometries are fitted using EMs based on the experimental data. The comparative results show that the EMs can take advantage of the prediction ability of each stand-alone metamodel and thus decrease the risk of adopting inappropriate metamodels. Then, the GA is used to facilitate design space exploration and global optimum search. Besides, the main effects and contribution rates of multiple process parameters on bead profile are analyzed. Eventually, the verification experiments are carried out to demonstrate the effectiveness and reliability of the obtained optimal parameters. Overall, the proposed hybrid approach, GA–EMs, exhibits great capability of guiding the actual LB processing and improving welding quality.     

基于Adaboost算法的回声状态网络预报器

把单个回声状态网络(echo state network,ESN)的预测模型作改进,对整体ESN预测精度的提高是有限的.针对以上问题,本文考虑整体ESN.首先利用Adaboost算法提升单个ESN的泛化性能及预测精度,并且根据Adaboost算法的结果,建立一种ESN预报器(Adaboost ESN,ABESN).这个ESN预报器根据拟合误差不断修正训练样本的权重,拟合误差越大,训练样本权重值就越大;因此,它在下一次迭代时,就会侧重在难以学习的样本.把单个ESN的预测模型经过加权,然后按照加法组合在一起,形成最终的ESN预测模型.将该预测模型应用于太阳黑子、Mackey-Glass时间序列的预测研究,仿真结果表明所提出的预测模型在实际时间序列预测领域的有效性.     

Effects of the correlation model,the trend model,and the number of training points on the accuracy of Kriging metamodels

This paper explores the effects of the correlation model, the trend model, and the number of training points on the accuracy of Kriging metamodels. Gaussian correlation models are found to be superior to exponential and linear correlation models. No particular trend model is found to be better than the other models. The number of training points used in constructing the Kriging metamodels is observed to change the relative performances of the trend and the correlation functions. The leave‐one‐out cross‐validation error is found to become a better surrogate for the actual error, as the number of training points is increased. Finally, the use of an ensemble of metamodels is discussed and it is found that using an ensemble may improve the accuracy.     

结合特征扰动与分配策略的集成辅助多目标优化算法

代理模型利用近似预测代替算法对多目标优化问题的真实评价,大幅减少了算法寻优所需的真实适应度评估次数。为提高代理模型在求解高维问题时的准确性并降低计算开销,提出一种基于特征扰动与分配策略的集成辅助多目标优化算法。将径向基函数网络代理模型与支持向量机回归代理模型作为集成过程中的基模型,降低算法在高维问题上的计算开销。结合特征扰动与基于记忆的影响因子分配策略构建集成代理模型,提高集成准确性。使用集成预测值与不确定信息加权辅助管理集成代理模型,平衡全局搜索与局部探索,增强算法在目标空间中的寻优能力。实验结果表明,该算法在ZDT1~ZDT3和ZDT6测试问题上所得解集的分布性与收敛性相比经典算法更好,并且当决策变量维数增加时,使用集成代理模型相比于Kriging代理模型约减少了90%的适应度评估次数,同时可获得更准确的预测结果。     

Ensemble of surrogates with recursive arithmetic average

Surrogate models are often used to replace expensive simulations of engineering problems. The common approach is to construct a series of metamodels based on a training set, and then, from these surrogates, pick out the best one with the highest accuracy as an approximation of the computationally intensive simulation. However, because the choice of approximate model depends on design of experiments (DOEs), the traditional strategy thus increases the risk of adopting an inappropriate model. Furthermore, in the design of complex product system, because of its feature of one-of-a-kind production, acquiring more samples is very expensive and intensively time-consuming, and sometimes even impossible. Therefore, in order to save sampling cost, it is a reasonable strategy to take full advantage of all the stand-alone surrogates and then combine them into an ensemble model. Ensemble technique is an effective way to make up for the shortfalls of traditional strategy. Motivated by the previous research on ensemble of surrogates, a new technique for constructing of a more accurate ensemble of surrogates is proposed in this paper. The weights are obtained using a recursive process, in which the values of these weights are updated in each iteration until the last ensemble achieves a desirable prediction accuracy. This technique has been evaluated using five benchmark problems and one reality problem. The results show that the proposed ensemble of surrogates with recursive arithmetic average provides more ideal prediction accuracy than the stand-alone surrogates and for most problems even exceeds the previously presented ensemble techniques. Finally, we should point out that the advantages of combination over selection are still difficult to illuminate. We are still using an “insurance policy” mode rather than offering significant improvements.     

基于多机器学习竞争策略的短时雷电预报

传统的雷电数据预测方法往往采用单一最优机器学习算法,较少考虑气象数据的时空变化等现象。针对该现象,提出一种基于集成策略的多机器学习短时雷电预报算法。首先,对气象数据进行属性约简,降低数据维度;其次,在数据集上训练多种异构机器学习分类器,并基于预测质量筛选最优基分类器;最后,通过对最优基分类器训练权重,并结合集成策略产生最终分类器。实验表明,该方法优于传统单最优方法,其平均预测准确率提高了9.5%。     

一种用于软测量建模的增量学习集成算法

针对软测量模型在实际应用中遇到的问题, 结合AdaBoost 集成学习思想, 提出适用于软测量回归的集成学习算法, 以提高传统软测量模型的精度. 为了克服模型更新技术对软测量实际应用的制约, 将增量学习机制加入软测量集成建模中, 使软测量模型具有在线实时更新的增量学习能力. 对浆纱过程使用新方法建立上浆率软测量模型, 并使用实际生产数据对模型进行检验, 检验结果表明, 该模型具有很好的预测精度, 并能够较好地实现在线更新.

     

利用集成分类器处理链路预测中的分类不平衡问题*

对于现实的复杂网络而言,有连边的节点对数目通常远小于无连边的节点对数目,在链路预测时,不同类别的样本数量不平衡会导致预测的分类结果与真实情况有较大的偏差。针对此问题,本文提出更优的链路预测算法,先对网络拓扑信息进行特征提取,再设计出一种集成分类器对数据样本进行平衡处理,然后基于网络的拓扑信息改进了分类器的集成规则,最后将训练出的集成分类器同现有的4个针对不平衡分类的链路预测学习算法进行对比研究。通过对4个不同规模的时序网络进行链路预测,结果表明：本文的链路预测学习算法具有更高的召回率,同时也保证了预测结果的准确性,从而更好地解决了链路预测中因类别不平衡导致的误分类问题。     

Adaptive sampling with automatic stopping for feasible region identification in engineering design

Engineering design is a complex process to find a suitable trade-off among different, and sometimes conflicting, design specifications. In reality, these requirements can be often considered as constraints of the design problem, that can be defined in terms of performance measures or geometrical characteristics of the device under study. In this paper, a new design space exploration methodology is presented for discovering feasible regions in the design space, where the term feasible region indicates the set of all design configurations satisfying all constraints of the design problem. The proposed method is based on Gaussian process metamodels to estimate the feasible region and leverages a information-based adaptive sampling technique to sequentially refine the prediction accuracy, which is applicable for multiple constraints problems. To efficiently stop the adaptive sampling process, a novel framework to estimate the metamodel’s prediction accuracy is proposed. The efficiency, accuracy and robustness of the proposed approach are compared with state-of-art techniques on suitable benchmark problems and practical engineering examples.

     

Towards an ensemble based system for predicting the number of software faults

Software fault prediction using different techniques has been done by various researchers previously. It is observed that the performance of these techniques varied from dataset to dataset, which make them inconsistent for fault prediction in the unknown software project. On the other hand, use of ensemble method for software fault prediction can be very effective, as it takes the advantage of different techniques for the given dataset to come up with better prediction results compared to individual technique. Many works are available on binary class software fault prediction (faulty or non-faulty prediction) using ensemble methods, but the use of ensemble methods for the prediction of number of faults has not been explored so far. The objective of this work is to present a system using the ensemble of various learning techniques for predicting the number of faults in given software modules. We present a heterogeneous ensemble method for the prediction of number of faults and use a linear combination rule and a non-linear combination rule based approaches for the ensemble. The study is designed and conducted for different software fault datasets accumulated from the publicly available data repositories. The results indicate that the presented system predicted number of faults with higher accuracy. The results are consistent across all the datasets. We also use prediction at level l (Pred(l)), and measure of completeness to evaluate the results. Pred(l) shows the number of modules in a dataset for which average relative error value is less than or equal to a threshold value l. The results of prediction at level l analysis and measure of completeness analysis have also confirmed the effectiveness of the presented system for the prediction of number of faults. Compared to the single fault prediction technique, ensemble methods produced improved performance for the prediction of number of software faults. Main impact of this work is to allow better utilization of testing resources helping in early and quick identification of most of the faults in the software system.     

Ensemble cluster pruning via convex-concave programming

Ensemble learning is the process of aggregating the decisions of different learners/models. Fundamentally, the performance of the ensemble relies on the degree of accuracy in individual learner predictions and the degree of diversity among the learners. The trade-off between accuracy and diversity within the ensemble needs to be optimized to provide the best grouping of learners as it relates to their performance. In this optimization theory article, we propose a novel ensemble selection algorithm which, focusing specifically on clustering problems, selects the optimal subset of the ensemble that has both accurate and diverse models. Those ensemble selection algorithms work for a given number of the best learners within the subset prior to their selection. The cardinality of a subset of the ensemble changes the prediction accuracy. The proposed algorithm in this study determines both the number of best learners and also the best ones. We compared our prediction results to recent ensemble clustering selection algorithms by the number of cardinalities and best predictions, finding better and approximated results to the optimum solutions.