Multiple surrogates: how cross-validation errors can help us to obtain the best predictor

Surrogate models are commonly used to replace expensive simulations of engineering problems. Frequently, a single surrogate is chosen based on past experience. This approach has generated a collection of papers comparing the performance of individual surrogates. Previous work has also shown that fitting multiple surrogates and picking one based on cross-validation errors (PRESS in particular) is a good strategy, and that cross-validation errors may also be used to create a weighted surrogate. In this paper, we discussed how PRESS (obtained either from the leave-one-out or from the k-fold strategies) is employed to estimate the RMS error, and whether to use the best PRESS solution or a weighted surrogate when a single surrogate is needed. We also studied the minimization of the integrated square error as a way to compute the weights of the weighted average surrogate. We found that it pays to generate a large set of different surrogates and then use PRESS as a criterion for selection. We found that (1) in general, PRESS is good for filtering out inaccurate surrogates; and (2) with sufficient number of points, PRESS may identify the best surrogate of the set. Hence the use of cross-validation errors for choosing a surrogate and for calculating the weights of weighted surrogates becomes more attractive in high dimensions (when a large number of points is naturally required). However, it appears that the potential gains from using weighted surrogates diminish substantially in high dimensions. We also examined the utility of using all the surrogates for forming the weighted surrogates versus using a subset of the most accurate ones. This decision is shown to depend on the weighting scheme. Finally, we also found that PRESS as obtained through the k-fold strategy successfully estimates the RMSE.     

Ensemble of surrogates with recursive arithmetic average

Surrogate models are often used to replace expensive simulations of engineering problems. The common approach is to construct a series of metamodels based on a training set, and then, from these surrogates, pick out the best one with the highest accuracy as an approximation of the computationally intensive simulation. However, because the choice of approximate model depends on design of experiments (DOEs), the traditional strategy thus increases the risk of adopting an inappropriate model. Furthermore, in the design of complex product system, because of its feature of one-of-a-kind production, acquiring more samples is very expensive and intensively time-consuming, and sometimes even impossible. Therefore, in order to save sampling cost, it is a reasonable strategy to take full advantage of all the stand-alone surrogates and then combine them into an ensemble model. Ensemble technique is an effective way to make up for the shortfalls of traditional strategy. Motivated by the previous research on ensemble of surrogates, a new technique for constructing of a more accurate ensemble of surrogates is proposed in this paper. The weights are obtained using a recursive process, in which the values of these weights are updated in each iteration until the last ensemble achieves a desirable prediction accuracy. This technique has been evaluated using five benchmark problems and one reality problem. The results show that the proposed ensemble of surrogates with recursive arithmetic average provides more ideal prediction accuracy than the stand-alone surrogates and for most problems even exceeds the previously presented ensemble techniques. Finally, we should point out that the advantages of combination over selection are still difficult to illuminate. We are still using an “insurance policy” mode rather than offering significant improvements.     

Ensemble of metamodels with optimized weight factors

Approximate mathematical models (metamodels) are often used as surrogates for more computationally intensive simulations. The common practice is to construct multiple metamodels based on a common training data set, evaluate their accuracy, and then to use only a single model perceived as the best while discarding the rest. This practice has some shortcomings as it does not take full advantage of the resources devoted to constructing different metamodels, and it is based on the assumption that changes in the training data set will not jeopardize the accuracy of the selected model. It is possible to overcome these drawbacks and to improve the prediction accuracy of the surrogate model if the separate stand-alone metamodels are combined to form an ensemble. Motivated by previous research on committee of neural networks and ensemble of surrogate models, a technique for developing a more accurate ensemble of multiple metamodels is presented in this paper. Here, the selection of weight factors in the general weighted-sum formulation of an ensemble is treated as an optimization problem with the desired solution being one that minimizes a selected error metric. The proposed technique is evaluated by considering one industrial and four benchmark problems. The effect of different metrics for estimating the prediction error at either the training data set or a few validation points is also explored. The results show that the optimized ensemble provides more accurate predictions than the stand-alone metamodels and for most problems even surpassing the previously reported ensemble approaches.     

Various approaches for constructing an ensemble of metamodels using local measures

Metamodels are approximate mathematical models used as surrogates for computationally expensive simulations. Since metamodels are widely used in design space exploration and optimization, there is growing interest in developing techniques to enhance their accuracy. It has been shown that the accuracy of metamodel predictions can be increased by combining individual metamodels in the form of an ensemble. Several efforts were focused on determining the contribution (or weight factor) of a metamodel in the ensemble using global error measures. In addition, prediction variance is also used as a local error measure to determine the weight factors. This paper investigates the efficiency of using local error measures, and also presents the use of the pointwise cross validation error as a local error measure as an alternative to using prediction variance. The effectiveness of ensemble models are tested on several problems with varying dimensionality: five mathematical benchmark problems, two structural mechanics problems and an automobile crash problem. It is found that the spatial ensemble models show better performances than the global ensemble for the low-dimensional problems, while the global ensemble is a more accurate model than the spatial ensembles for the high-dimensional problems. Ensembles based on pointwise cross validation error and prediction variance provide similar accuracy. The ensemble models based on local measures reduce cross validation errors drastically, but their performances are not that impressive in reducing the error evaluated at random test points, because the pointwise cross validation error is not a good surrogate for the error at a point.     

Ensemble of surrogate based global optimization methods using hierarchical design space reduction

For computationally expensive black-box problems, surrogate models are widely employed to reduce the needed computation time and efforts during the search of the global optimum. However, the construction of an effective surrogate model over a large design space remains a challenge in many cases. In this work, a new global optimization method using an ensemble of surrogates and hierarchical design space reduction is proposed to deal with the optimization problems with computation-intensive, black-box objective functions. During the search, an ensemble of three representative surrogate techniques with optimized weight factors is used for selecting promising sample points, narrowing down space exploration and identifying the global optimum. The design space is classified into: Original Global Space (OGS), Promising Joint Space (PJS), Important Local Space (ILS), using the newly proposed hierarchical design space reduction (HSR). Tested using eighteen representative benchmark and two engineering design optimization problems, the newly proposed global optimization method shows improved capability in identifying promising search area and reducing design space, and superior search efficiency and robustness in identifying the global optimum.     

Toward an optimal ensemble of kernel-based approximations with engineering applications

This paper presents a general approach toward the optimal selection and ensemble (weighted average) of kernel-based approximations to address the issue of model selection. That is, depending on the problem under consideration and loss function, a particular modeling scheme may outperform the others, and, in general, it is not known a priori which one should be selected. The surrogates for the ensemble are chosen based on their performance, favoring non-dominated models, while the weights are adaptive and inversely proportional to estimates of the local prediction variance of the individual surrogates. Using both well-known analytical test functions and, in the surrogate-based modeling of a field scale alkali-surfactant-polymer enhanced oil recovery process, the ensemble of surrogates, in general, outperformed the best individual surrogate and provided among the best predictions throughout the domains of interest. This work was supported in part by the Fondo Nacional de Ciencia, Tecnología e Innovación (FONACIT), Venezuela under Grant F-2005000210. N. Q. Author also acknowledges that this material is based upon work supported by National Science Foundation under Grant DDM-423280.     

An adaptive hybrid surrogate model

The determination of complex underlying relationships between system parameters from simulated and/or recorded data requires advanced interpolating functions, also known as surrogates. The development of surrogates for such complex relationships often requires the modeling of high dimensional and non-smooth functions using limited information. To this end, the hybrid surrogate modeling paradigm, where different surrogate models are combined, offers an effective solution. In this paper, we develop a new high fidelity surrogate modeling technique that we call the Adaptive Hybrid Functions (AHF). The AHF formulates a reliable Crowding Distance-Based Trust Region (CD-TR), and adaptively combines the favorable characteristics of different surrogate models. The weight of each contributing surrogate model is determined based on the local measure of accuracy for that surrogate model in the pertinent trust region. Such an approach is intended to exploit the advantages of each component surrogate. This approach seeks to simultaneously capture the global trend of the function as well as the local deviations. In this paper, the AHF combines four component surrogate models: (i) the Quadratic Response Surface Model (QRSM), (ii) the Radial Basis Functions (RBF), (iii) the Extended Radial Basis Functions (E-RBF), and (iv) the Kriging model. The AHF is applied to standard test problems and to a complex engineering design problem. Subsequent evaluations of the Root Mean Squared Error (RMSE) and the Maximum Absolute Error (MAE) illustrate the promising potential of this hybrid surrogate modeling approach.     

An approximate optimization strategy using refined hybrid metamodel

An approximate model called metamodel or surrogate model is a mathematical model that numerically approximates response of a system during an engineering simulation process or test. The introduction of a metamodel makes it possible to express response defined in the design problem as a simple mathematical function of design variables. A metamodel can be built with response surface method (RSM), kriging, neural network, radial basis function, and so on. Each method has its advantages and disadvantages. A combined metamodel called hybrid model, ensemble model, or multiple surrogates has been developed to maximize each metamodel's strength. The hybrid model of this research includes RSM and kriging. Besides, a strategy to refine the hybrid metamodel is implemented by reducing design space. In this process, information related to Hessian is utilized for an unconstrained optimization problem, on the contrary feasibility for a constrained optimization problem. This research presents a new hybrid metamodel-based optimization strategy called refined hybrid metamodel. Five mathematical test problems, two-bar design, spring design, and propeller shaft design problems are solved with the suggested method, verifying its usefulness. Most of the optimal results with the proposed method are closer to exact solutions with smaller function evaluations than existing methods.     

Analysis of dataset selection for multi-fidelity surrogates for a turbine problem

Multi-fidelity surrogates (MFS) have become a popular way to combine small number of expensive high-fidelity (HF) samples and many cheap low-fidelity (LF) samples. In some situations LF samples can come from multiple sources and sometimes the HF samples alone can obtain a more accurate surrogate than the combination (HF&LF). Therefore this paper considers using maximum likelihood (ML) and cross validation (CV) to select the dataset leading to best surrogate accuracy, when multiple sample sources are available. The kriging and co-kriging techniques were employed to build surrogates. Unlike conventional model selection, the multi-fidelity datasets selection by ML and CV has to compare the surrogate accuracy of different true functions. The effectiveness of ML and CV is examined through a two-variable turbine problem, where samples can come from one HF and two LF models. The indicators were used to select between using only HF samples or combining them with one set of LF samples or the other. The best selection proved to depend on the design of experiments (DOE), and so datasets were generated for a large number of DOEs. It was found the CV and ML worked relatively well in selection between two LF sample sources for MFS. When selecting between only HF and HF&LF, the ML, which is frequently used in co-kriging hyper-parameter estimation, failed in detecting when the surrogate accuracy of only HF was better than HF & LF. The CV was successful only part of the time. The reasons behind the poor performance are analyzed with the help of a 1D example.     

Sparsity-promoting polynomial response surface: A new surrogate model for response prediction

Computation-intensive analyses/simulations are becoming increasingly common in engineering design problems. To improve the computation efficiency, surrogate models are used to replace expensive simulations of engineering problems. This paper proposes a new high-fidelity surrogate modeling approach which is called the Sparsity-promoting Polynomial Response Surface (SPPRS). In the SPPRS model, a series of Legendre polynomials is selected as basis functions, and its number is compatible with the sample size so as to enhance the expression ability for complex functional relationships. The coefficients associated with basis functions are estimated using a “sparsity-promoting” regression approach which is an ensemble of two techniques: least squares and ℓ₁-norm regularization. As a result, only these basis functions relevant to explain the function relationship are picked out, and that dedicates to ease the problem of overfitting for training points. With the sparsity-promoting regression approach, such a surrogate model intends to capture both the global trend of the functional variation and a reasonable local accuracy in the neighborhood of training points. Additionally, Latin hypercube design (LHD) is proved conducive to improving the predictive capability of our model. The SPPRS is applied to seven benchmark test functions and a complex engineering problem. The results illustrate the promising benefits of this novel surrogate modeling technique.     

Efficient global optimization with ensemble and selection of kernel functions for engineering design

In this paper, we investigate the use of multiple kernel functions for assisting single-objective Kriging-based efficient global optimization (EGO). The primary objective is to improve the robustness of EGO in terms of the choice of kernel function for solving a variety of black-box optimization problems in engineering design. Specifically, three widely used kernel functions are studied, that is, Gaussian, Matérn-3/2, and Matérn-5/2 function. We investigate both model selection and ensemble techniques based on Akaike information criterion (AIC) and cross-validation error on a set of synthetic (noiseless and noisy) and non-algebraic (aerodynamic and parameter tuning) optimization problems; in addition, the use of cross-validation-based local (i.e., pointwise) ensemble is also studied. Since all the constituent surrogate models in the ensemble scheme are Kriging models, it is possible to perform EGO since the Kriging uncertainty structure is still preserved. Through analyses of empirical experiments, it is revealed that the ensemble techniques improve the robustness and performance of EGO. It is also revealed that the use of Matérn-kernels yields better results than those of the Gaussian kernel when EGO with a single kernel is considered. Furthermore, we observe that model selection methods do not yield any substantial improvement over single kernel EGO. When averaged across all types of problem (i.e., noise level, dimensionality, and synthetic/non-algebraic), the local ensemble technique achieves the best performance.

     

Improving surrogate-assisted variable fidelity multi-objective optimization using a clustering algorithm

Surrogate-assisted evolutionary optimization has proved to be effective in reducing optimization time, as surrogates, or meta-models can approximate expensive fitness functions in the optimization run. While this is a successful strategy to improve optimization efficiency, challenges arise when constructing surrogate models in higher dimensional function space, where the trade space between multiple conflicting objectives is increasingly complex. This complexity makes it difficult to ensure the accuracy of the surrogates. In this article, a new surrogate management strategy is presented to address this problem. A k-means clustering algorithm is employed to partition model data into local surrogate models. The variable fidelity optimization scheme proposed in the author's previous work is revised to incorporate this clustering algorithm for surrogate model construction. The applicability of the proposed algorithm is illustrated on six standard test problems. The presented algorithm is also examined in a three-objective stiffened panel optimization design problem to show its superiority in surrogate-assisted multi-objective optimization in higher dimensional objective function space. Performance metrics show that the proposed surrogate handling strategy clearly outperforms the single surrogate strategy as the surrogate size increases.     

Performance of an ensemble of ordinary, universal, non-stationary and limit Kriging predictors

The selection of stationary or non-stationary Kriging to create a surrogate model of a black box function requires apriori knowledge of the nature of response of the function as these techniques are better at representing some types of responses than others. While an adaptive technique has been previously proposed to adjust the level of stationarity within the surrogate model such a model can be prohibitively expensive to construct for high dimensional problems. An alternative approach is to employ a surrogate model constructed from an ensemble of stationary and non-stationary Kriging models. The following paper assesses the accuracy and optimization performance of such a modelling strategy using a number of analytical functions and engineering design problems.     

Robust springback optimization of a dual phase steel seven-flange die assembly

This paper investigates robust springback optimization of a DP600 dual phase steel seven-flange die assembly composed of different flange designs. The optimum values of the die radius and the punch radius are sought to minimize the mean and the standard deviation of springback using surrogate based optimization. Springback values at the training points of surrogate models are evaluated using the finite element analysis code LS-DYNA. In this work, four different surrogate modeling types are considered: polynomial response surfaces (PRS) approximations, stepwise regression (SWR), radial basis functions (RBF) and Kriging (KR). Two sets of surrogate models are constructed in this study. The first set is constructed to relate the springback to the design variables as well as the random variables. It is found for the first set of surrogate models that KR provides more accurate springback predictions than PRS, SWR and RBF. The mean and the standard deviation of springback are calculated using Monte Carlo simulations, where the first set of surrogate models is utilized. The second set of surrogate models is generated to relate the mean and the standard deviation of springback to the design variables. It is found for the second set of surrogate models that PRS provides more accurate springback predictions than SWR, RBF and KR. It is also found that introducing beads increases the mean performance and the robustness. The robust optimization is performed and significant springback reductions are obtained for all flanges ranging between 7% and 85% compared to the nominal design. It is also found that a design change that decreases the mean springback also reduces the springback variation. It is observed that the optimization results heavily dependent on the bounds of the die and punch radii. In addition, optimization with multiple surrogates is investigated. Finding multiple candidates of optimum with multiple surrogates and selecting the one with the best actual performance is found to be a better strategy than optimizing using the most accurate surrogate model.     

A rule-based method for automated surrogate model selection

Surrogate models have been widely used in engineering design because of their capability to approximate computationally complex engineering systems. In practice, the choice of surrogate models is extremely important since there are many types of surrogate models, and they also have different hyper-parameters. Traditional manual selection approaches are very time-consuming and cannot be generalized. To address these challenges, an evolutionary algorithm (EA)-based approaches are proposed and studied. However, they lack interpretability and are computationally expensive. To address these gaps, we create a rule-based method for an automatic surrogate model selection called AutoSM. The drastic increase in the selection pace by pre-screening of surrogate model types based on selection rule extraction is the scientific contribution of our proposed method. First, an interpretable decision tree is built to map four critical features, including problem scale, noise, size of sample and nonlinearity, to the types of surrogate model and select the promising surrogate model; then, a genetic algorithm (GA) is used to find the appropriate hyper-parameters for each selected surrogate model. The AutoSM is tested with three theoretical problems and two engineering problems, including a hot rod rolling and a blowpipe design problem. According to the empirical results, using the proposed AutoSM, we can find the promising surrogate model and associated hyper-parameter in 9 times less than other automatic selection approaches such as concurrent surrogate model selection (COSMOS) while maintaining the same accuracy and robustness in surrogate model selection. Besides, the proposed AutoSM, unlike previous EA-based automatic surrogate model selection methods, is not a black box and is interpretable.     

A holistic multi-objective optimization design procedure for ensemble member generation and selection

In the last few years, machine learning techniques have been successfully applied to solve engineering problems. However, owing to certain complexities found in real-world problems, such as class imbalance, classical learning algorithms may not reach a prescribed performance. There can be situations where a good result on different conflicting objectives is desirable, such as true positive and true negative ratios, or it is important to balance model’s complexity and prediction score. To solve such issues, the application of multi-objective optimization design procedures can be used to analyze various trade-offs and build more robust machine learning models. Thus, the creation of ensembles of predictive models using such procedures is addressed in this work. First, a set of diverse predictive models is built by employing a multi-objective evolutionary algorithm. Next, a second multi-objective optimization step selects the previous models as ensemble members, resulting on several non-dominated solutions. A final multi-criteria decision making stage is applied to rank and visualize the resulting ensembles. To analyze the proposed methodology, two different experiments are conducted for binary classification. The first case study is a famous classification problem through which the proposed procedure is illustrated. The second one is a challenging real-world problem related to water quality monitoring, where the proposed procedure is compared to four classical ensemble learning algorithms. Results on this second experiment show that the proposed technique is able to create robust ensembles that can outperform other ensemble methods. Overall, the authors conclude that the proposed methodology for ensemble generation creates competitive models for real-world engineering problems.     

Uncertainty quantification in reliability estimation with limit state surrogates

Model-based reliability analysis is affected by different types of epistemic uncertainty, due to inadequate data and modeling errors. When the physics-based simulation model is computationally expensive, a surrogate has often been used in reliability analysis, introducing additional uncertainty due to the surrogate. This paper proposes a framework to include statistical uncertainty and model uncertainty in surrogate-based reliability analysis. Two types of surrogates have been considered: (1) general-purpose surrogate models that compute the system model output over the desired ranges of the random variables; and (2) limit-state surrogates. A unified approach to connect the model calibration analysis using the Kennedy and O’Hagan (KOH) framework to the construction of limit state surrogate and to estimating the uncertainty in reliability analysis is developed. The Gaussian Process (GP) general-purpose surrogate of the physics-based simulation model obtained from the KOH calibration analysis is further refined at the limit state (local refinement) to construct the limit state surrogate, which is used for reliability analysis. An efficient single-loop sampling approach using the probability integral transform is used for sampling the input variables with statistical uncertainty. The variability in the GP prediction (surrogate uncertainty) is included in reliability analysis through correlated sampling of the model predictions at different inputs. The Monte Carlo sampling (MCS) error, which represents the error due to limited Monte Carlo samples, is quantified by constructing a probability density function. All the different sources of epistemic uncertainty are quantified and aggregated to estimate the uncertainty in the reliability analysis. Two examples are used to demonstrate the proposed techniques.     

A multi-fidelity surrogate modeling approach for incorporating multiple non-hierarchical low-fidelity data

Multi-fidelity (MF) surrogate models have been widely used in simulation-based design problems to reduce the computational cost by integrating the data with different fidelity levels. Most of the existing MF modeling methods are only applicable to the problems with hierarchical low-fidelity (LF) models, namely the fidelity levels of multiple LF models can be identified. However, the fidelity levels of the LF models that are obtained from different simplification methods often vary over the design space. To address this challenge, a non-hierarchical Co-Kriging modeling (NHLF-Co-Kriging) method that can flexibly handle multiple non-hierarchical LF models is developed in this work. In the proposed method, multiple LF models are scaled by different scale factors, and a discrepancy model is utilized to depict the differences between the HF model and the ensembled LF models. To make the discrepancy Gaussian process (GP) model easy to be fitted, an optimization problem whose objective is to minimize the second derivative of the prediction values of the discrepancy GP model is defined to obtain optimal scale factors of the LF models. The performance of the NHLF-Co-Kriging method is compared with the extended Co-Kriging model and linear regression MF surrogate model through several analytical examples and an engineering case. Results show that the proposed method selects more reasonable scale factors for the multiple LF models and provides more accurate MF surrogate models under a limited computational budget.     

A methodology for building neural networks models from empirical engineering data

Neural networks (NN) are general tools for modeling functional relationships in engineering. They are used to model the behavior of products and the properties of processes. Nevertheless, their use is often ad hoc. This paper provides a sound basis for using NN as tools for modeling functional relationships implicit in empirical engineering data. First, a clear definition of a modeling task is given, followed by reviewing the theoretical modeling capabilities of NN and NN model estimation. Subsequently, a procedure for using NN in engineering practice is described and illustrated with an example of modeling marine propeller behavior. Particular attention is devoted to better estimation of model quality, insight into the influence of measurement errors on model quality, and the use of advanced methods such as stacked generalization and ensemble modeling to further improve model quality. Using a new method of ensemble of SG(k-NN), one could improve the quality of models even if they are close to being optimal.