Ensemble of metamodels with optimized weight factors

Approximate mathematical models (metamodels) are often used as surrogates for more computationally intensive simulations. The common practice is to construct multiple metamodels based on a common training data set, evaluate their accuracy, and then to use only a single model perceived as the best while discarding the rest. This practice has some shortcomings as it does not take full advantage of the resources devoted to constructing different metamodels, and it is based on the assumption that changes in the training data set will not jeopardize the accuracy of the selected model. It is possible to overcome these drawbacks and to improve the prediction accuracy of the surrogate model if the separate stand-alone metamodels are combined to form an ensemble. Motivated by previous research on committee of neural networks and ensemble of surrogate models, a technique for developing a more accurate ensemble of multiple metamodels is presented in this paper. Here, the selection of weight factors in the general weighted-sum formulation of an ensemble is treated as an optimization problem with the desired solution being one that minimizes a selected error metric. The proposed technique is evaluated by considering one industrial and four benchmark problems. The effect of different metrics for estimating the prediction error at either the training data set or a few validation points is also explored. The results show that the optimized ensemble provides more accurate predictions than the stand-alone metamodels and for most problems even surpassing the previously reported ensemble approaches.     

Various approaches for constructing an ensemble of metamodels using local measures

Metamodels are approximate mathematical models used as surrogates for computationally expensive simulations. Since metamodels are widely used in design space exploration and optimization, there is growing interest in developing techniques to enhance their accuracy. It has been shown that the accuracy of metamodel predictions can be increased by combining individual metamodels in the form of an ensemble. Several efforts were focused on determining the contribution (or weight factor) of a metamodel in the ensemble using global error measures. In addition, prediction variance is also used as a local error measure to determine the weight factors. This paper investigates the efficiency of using local error measures, and also presents the use of the pointwise cross validation error as a local error measure as an alternative to using prediction variance. The effectiveness of ensemble models are tested on several problems with varying dimensionality: five mathematical benchmark problems, two structural mechanics problems and an automobile crash problem. It is found that the spatial ensemble models show better performances than the global ensemble for the low-dimensional problems, while the global ensemble is a more accurate model than the spatial ensembles for the high-dimensional problems. Ensembles based on pointwise cross validation error and prediction variance provide similar accuracy. The ensemble models based on local measures reduce cross validation errors drastically, but their performances are not that impressive in reducing the error evaluated at random test points, because the pointwise cross validation error is not a good surrogate for the error at a point.     

Simultaneous optimization of shape parameters and weight factors in ensemble of radial basis functions

Radial basis functions (RBFs) are approximate mathematical models that can mimic the behavior of fast changing responses. Different formulations of RBFs can be combined in the form of an ensemble model to improve prediction accuracy. The conventional approach in constructing an RBF ensemble is based on a two-step procedure. In the first step, the optimal values of the shape parameters of each stand-alone RBF model are determined. In the second step, the shape parameters are fixed to these optimal values and the weight factors of each stand-alone RBF model in the ensemble are optimized. In this paper, simultaneous optimization of shape parameters and weight factors is proposed as an alternative to this two-step procedure for further improvement of prediction accuracy. Gaussian, multiquadric and inverse multiquadric RBF formulations are combined in the ensemble model. The efficiency of the proposed method is evaluated through example problems of varying dimensions from two to twelve. It is found that the proposed method improves the prediction accuracy of the ensemble compared to the conventional two-step procedure for the example problems considered.     

Metamodel selection based on stepwise regression

Metamodels are often used to replace expensive simulations of engineering problems. When a training set is given, a series of metamodels can be constructed, and then there are two strategies to deal with these metamodels: (1) picking out the best one with the highest accuracy as an approximation of the computationally intensive simulation; and (2) combining all of them into an ensemble model. However, since the choice of approximate model depends on design of experiments (DOEs), employing of the first strategy thus increases the risk of adopting an inappropriate model. Nevertheless, the second strategy also seems not to be a good choice, since adding redundant metamodels may lead to loss of accuracy. Therefore, it is a necessary step to eliminate the redundant metamodels from the set of the candidates before constructing the final ensemble. Illuminated by the method of variable selection widely used in polynomial regression, a metamodel selection method based on stepwise regression is proposed. In our method, just a subset of n ones (n ≤ p, where p is the number of all of the candidate metamodels) is used. In addition, a new ensemble technique is proposed from the view of polynomial regression in this work. This new ensemble technique, combined with metamodel selection method, has been evaluated using six benchmark problems. The results show that eliminating the redundant metamodels before constructing the ensemble can provide more ideal prediction accuracy than directly constructing the ensemble by utilizing all of the candidates.     

Ensemble of surrogates

The custom in surrogate-based modeling of complex engineering problems is to fit one or more surrogate models and select the one surrogate model that performs best. In this paper, we extend the utility of an ensemble of surrogates to (1) identify regions of possible high errors at locations where predictions of surrogates widely differ, and (2) provide a more robust approximation approach. We explore the possibility of using the best surrogate or a weighted average surrogate model instead of individual surrogate models. The weights associated with each surrogate model are determined based on the errors in surrogates. We demonstrate the advantages of an ensemble of surrogates using analytical problems and one engineering problem. We show that for a single problem the choice of test surrogate can depend on the design of experiments.     

Pointwise ensemble of meta-models using v nearest points cross-validation

As the use of meta-models to replace computationally-intensive simulations for estimating real system behaviors increases, there is an increasing need to select appropriate meta-models that well represent real system behaviors. Since in most cases designers do not know the behavior of the real system a priori, however, they often have trouble selecting a suitable meta-model. In order to provide robust prediction performance, ensembles of meta-models have been developed which linearly combines stand-alone meta-models. In this study, we propose a new pointwise ensemble of meta-models whose weights vary according to the prediction point of interest. The suggested method can include all kinds of stand-alone meta-models for ensemble construction, and can interpolate real system response values at training points, even if regression models are included as stand-alone meta-models. To evaluate the effectiveness of the proposed method, its prediction performance is compared with those of existing ensembles of meta-models using well-known mathematical functions. The results show that our pointwise ensemble of meta-models provides more robust and accurate predictions than existing models for a majority of test problems.     

Prediction of Chaotic Time-Series with a Resource-Allocating RBF Network

One of the main problems associated with artificial neural networks on-line learning methods is the estimation of model order. In this paper, we report about a new approach to constructing a resource-allocating radial basis function network exploiting weights adaptation using recursive least-squares technique based on Givens QR decomposition. Further, we study the performance of pruning strategy we introduced to obtain the same prediction accuracy of the network with lower model order. The proposed methods were tested on the task of Mackey-Glass time-series prediction. Order of resulting networks and their prediction performance were superior to those previously reported by Platt [12].     

基于Adaboost算法的回声状态网络预报器

把单个回声状态网络(echo state network,ESN)的预测模型作改进,对整体ESN预测精度的提高是有限的.针对以上问题,本文考虑整体ESN.首先利用Adaboost算法提升单个ESN的泛化性能及预测精度,并且根据Adaboost算法的结果,建立一种ESN预报器(Adaboost ESN,ABESN).这个ESN预报器根据拟合误差不断修正训练样本的权重,拟合误差越大,训练样本权重值就越大;因此,它在下一次迭代时,就会侧重在难以学习的样本.把单个ESN的预测模型经过加权,然后按照加法组合在一起,形成最终的ESN预测模型.将该预测模型应用于太阳黑子、Mackey-Glass时间序列的预测研究,仿真结果表明所提出的预测模型在实际时间序列预测领域的有效性.     

Ensemble of metamodels: the augmented least squares approach

In this work we present an approach to create ensemble of metamodels (or weighted averaged surrogates) based on least squares (LS) approximation. The LS approach is appealing since it is possible to estimate the ensemble weights without using any explicit error metrics as in most of the existent ensemble methods. As an additional feature, the LS based ensemble of metamodels has a prediction variance function that enables the extension to the efficient global optimization. The proposed LS approach is a variation of the standard LS regression by augmenting the matrices in such a way that minimizes the effects of multicollinearity inherent to calculation of the ensemble weights. We tested and compared the augmented LS approach with different LS variants and also with existent ensemble methods, by means of analytical and real-world functions from two to forty-four variables. The augmented least squares approach performed with good accuracy and stability for prediction purposes, in the same level of other ensemble methods and has computational cost comparable to the faster ones.     

基于个体选择的动态权重神经网络集成方法研究

神经网络集成技术能有效地提高神经网络的预测精度和泛化能力,已成为机器学习和神经计算领域的一个研究热点。该文针对回归分析问题提出了一种结合应用遗传算法进行个体选择和动态确定结果合成权重的神经网络集成构造方法。在训练出个体神经网络之后,应用遗传算法对个体网络进行选择,然后根据被选择的各个体网络在输入空间上对训练样本的预测误差,应用广义回归网络来动态地确定各个体网络在特定输入空间上的合成权重。实验结果表明,与仅应用个体网络选择或动态确定权重的方法相比,该集成方法基本上能取得更好地预测精度和相近的稳定性。     

Performance of an ensemble of ordinary, universal, non-stationary and limit Kriging predictors

The selection of stationary or non-stationary Kriging to create a surrogate model of a black box function requires apriori knowledge of the nature of response of the function as these techniques are better at representing some types of responses than others. While an adaptive technique has been previously proposed to adjust the level of stationarity within the surrogate model such a model can be prohibitively expensive to construct for high dimensional problems. An alternative approach is to employ a surrogate model constructed from an ensemble of stationary and non-stationary Kriging models. The following paper assesses the accuracy and optimization performance of such a modelling strategy using a number of analytical functions and engineering design problems.     

On-line transfer learning for multi-fidelity data fusion with ensemble of deep neural networks

Deep Neural Network (DNN) is widely used in engineering applications for its ability to handle problems with almost any nonlinearities. However, it is generally difficult to obtain sufficient high-fidelity (HF) sample points for expensive optimization tasks, which may affect the generalization performance of DNN and result in inaccurate predictions. To solve this problem and improve the prediction accuracy of DNN, this paper proposes an on-line transfer learning based multi-fidelity data fusion (OTL-MFDF) method including two parts. In the first part, the ensemble of DNNs is established. Firstly, a large number of low-fidelity sample points and a few HF sample points are generated, which are used as the source dataset and target dataset, respectively. Then, the Bayesian Optimization (BO) is utilized to obtain several groups of hyperparameters, based on which DNNs are pre-trained using the source dataset. Next, these pre-trained DNNs are re-trained by fine-tuning on the target dataset, and the ensemble of DNNs is established by assigning different weights to each pre-trained DNN. In the second part, the on-line learning system is developed for adaptive updating of the ensemble of DNNs. To evaluate the uncertainty error of the predicted values of DNN and determine the location of the updated HF sample point, the query-by-committee strategy based on the ensemble of DNNs is developed. The Covariance Matrix Adaptation Evolutionary Strategies is employed as the optimizer to find out the location where the maximal disagreement is achieved by the ensemble of DNNs. The design space is partitioned by the Voronoi diagram method, and then the selected point is moved to its nearest Voronoi cell boundary to avoid clustering between the updated point and the existing sample points. Three different types of test problems and an engineering example are adopted to illustrate the effectiveness of the OTL-MFDF method. Results verify the outstanding efficiency, global prediction accuracy and applicability of the OTL-MFDF method.     

Toward an optimal ensemble of kernel-based approximations with engineering applications

This paper presents a general approach toward the optimal selection and ensemble (weighted average) of kernel-based approximations to address the issue of model selection. That is, depending on the problem under consideration and loss function, a particular modeling scheme may outperform the others, and, in general, it is not known a priori which one should be selected. The surrogates for the ensemble are chosen based on their performance, favoring non-dominated models, while the weights are adaptive and inversely proportional to estimates of the local prediction variance of the individual surrogates. Using both well-known analytical test functions and, in the surrogate-based modeling of a field scale alkali-surfactant-polymer enhanced oil recovery process, the ensemble of surrogates, in general, outperformed the best individual surrogate and provided among the best predictions throughout the domains of interest. This work was supported in part by the Fondo Nacional de Ciencia, Tecnología e Innovación (FONACIT), Venezuela under Grant F-2005000210. N. Q. Author also acknowledges that this material is based upon work supported by National Science Foundation under Grant DDM-423280.     

Random forests-based extreme learning machine ensemble for multi-regime time series prediction

Accurate and timely predicting values of performance parameters are currently strongly needed for important complex equipment in engineering. In time series prediction, two problems are urgent to be solved. One problem is how to achieve the accuracy, stability and efficiency together, and the other is how to handle time series with multiple regimes. To solve these two problems, random forests-based extreme learning machine ensemble model and a novel multi-regime approach are proposed respectively, and these two approaches can be integrated to achieve better performance. First, the extreme learning machine (ELM) is used in the proposed model because of its efficiency. Then the regularized ELM and ensemble learning strategy are used to improve generalization performance and prediction accuracy. The bootstrap sampling technique is used to generate training sample sets for multiple base-level ELM models, and then the random forests (RF) model is used as the combiner to aggregate these ELM models to achieve more accurate and stable performance. Next, based on the specific properties of turbofan engine time series, a multi-regime approach is proposed to handle it. Regimes are first separated, then the proposed RF-based ELM ensemble model is used to learn models of all regimes, individually, and last, all the learned regime models are aggregated to predict performance parameter at the future timestamp. The proposed RF-based ELM ensemble model and multi-regime approaches are evaluated by using NN3 time series and NASA turbofan engine time series, and then the proposed model is applied to the exhaust gas temperature prediction of CFM engine. The results demonstrate that the proposed RF-based ELM ensemble model and multi-regime approach can be accurate, stable and efficient in predicting multi-regime time series, and it can be robust against overfitting.     

Multiple surrogates: how cross-validation errors can help us to obtain the best predictor

Surrogate models are commonly used to replace expensive simulations of engineering problems. Frequently, a single surrogate is chosen based on past experience. This approach has generated a collection of papers comparing the performance of individual surrogates. Previous work has also shown that fitting multiple surrogates and picking one based on cross-validation errors (PRESS in particular) is a good strategy, and that cross-validation errors may also be used to create a weighted surrogate. In this paper, we discussed how PRESS (obtained either from the leave-one-out or from the k-fold strategies) is employed to estimate the RMS error, and whether to use the best PRESS solution or a weighted surrogate when a single surrogate is needed. We also studied the minimization of the integrated square error as a way to compute the weights of the weighted average surrogate. We found that it pays to generate a large set of different surrogates and then use PRESS as a criterion for selection. We found that (1) in general, PRESS is good for filtering out inaccurate surrogates; and (2) with sufficient number of points, PRESS may identify the best surrogate of the set. Hence the use of cross-validation errors for choosing a surrogate and for calculating the weights of weighted surrogates becomes more attractive in high dimensions (when a large number of points is naturally required). However, it appears that the potential gains from using weighted surrogates diminish substantially in high dimensions. We also examined the utility of using all the surrogates for forming the weighted surrogates versus using a subset of the most accurate ones. This decision is shown to depend on the weighting scheme. Finally, we also found that PRESS as obtained through the k-fold strategy successfully estimates the RMSE.     

基于广义回归网络的动态权重回归型神经网络集成方法研究*

神经网络集成技术能有效地提高神经网络的预测精度和泛化能力,已成为机器学习和神经计算领域的一个研究热点。针对回归分析问题提出了一种动态确定结果合成权重的神经网络集成构造方法,在训练出个体神经网络之后,根据各个体网络在输入空间上对训练样本的预测误差,应用广义回归网络来动态地确定各个体网络在特定输入空间上的权重。实验结果表明,与传统的简单平均和加权平均方法相比,本集成方法能取得更好的预测精度。     

数据驱动的贝叶斯SVR自适应建模及昂贵约束多目标代理优化

实际工程中的多目标优化问题往往具有黑箱特性且需要耗时的功能性评估,采用传统的进化优化方法求解,存在计算成本高昂且难以实现的问题.考虑代理优化方法在处理需要功能性评估工程设计问题中的高效性,提出一种小样本数据驱动下的贝叶斯SVR自适应建模及昂贵约束多目标代理优化方法.该方法在实现过程中选取贝叶斯SVR模型以减少功能性评估过程的昂贵仿真成本,利用最大化约束期望改进矩阵聚合策略进行新设计方案选取,并通过小样本信息的不断更新实现数据驱动下的贝叶斯SVR模型自适应更新和逐步优化.贝叶斯SVR模型具有强的边界刻画能力及预测不确定性度量功能,可为新样本挑选提供预测精度保障及潜在的改进方向.所提出的切比雪夫距离和曼哈顿距离聚合策略从样本填充的改进范围考虑,使其具有较强的改进边界探索能力,在多变量优化问题中具有计算复杂度低、适用性强的特点.测试函数及工程实例结果表明:1)所提出的方法可在小样本条件下有效减少昂贵仿真成本,提升昂贵约束多目标问题的优化效率;2)获取昂贵约束多目标问题的Pareto前沿在收敛性、多样性及空间分布性方面均具有一定优势.     

基于广义回归网络的动态权重回归型神经网络集成方法研究

神经网络集成技术能有效地提高神经网络的预测精度和泛化能力,已成为机器学习和神经计算领域的一个研究热点。针对回归分析问题提出了一种动态确定结果合成权重的神经网络集成构造方法,在训练出个体神经网络之后,根据各个体网络在输入空间上对训练样本的预测误差,应用广义回归网络来动态地确定各个体网络在特定输入空间上的权重。实验结果表明,与传统的简单平均和加权平均方法相比,本集成方法能取得更好的预测精度。     

Mid-range metamodel assembly building based on linear regression for large scale optimization problems

In this work an approach to building a high accuracy approximation valid in a larger range of design variables is investigated. The approach is based on an assembly of multiple surrogates into a single surrogate using linear regression. The coefficients of the model assembly are not weights of the individual models but tuning parameters determined by the least squares method. The approach was utilized in the Multipoint Approximation Method (MAM) method within the mid-range approximation framework. The developed technique has been tested on several benchmark problems with up to 1000 design variables and constraints. The obtained results show a high degree of accuracy of the built approximations and the efficiency of the technique when applied to large-scale optimization problems.     

一种新的基于多样性赋权证据推理的集成学习方法

在集成学习中使用平均法、投票法作为结合策略无法充分利用基分类器的有效信息,且根据波动性设置基分类器的权重不精确、不恰当。以上问题会降低集成学习的效果,为了进一步提高集成学习的性能,提出将证据推理(evidence reasoning, ER)规则作为结合策略,并使用多样性赋权法设置基分类器的权重。首先,由多个深度学习模型作为基分类器、ER规则作为结合策略,构建集成学习的基本结构;然后,通过多样性度量方法计算每个基分类器相对于其他基分类器的差异性;最后,将差异性归一化实现基分类器的权重设置。通过多个图像数据集的分类实验,结果表明提出的方法较实验选取的其他方法准确率更高且更稳定,证明了该方法可以充分利用基分类器的有效信息,且多样性赋权法更精确。