Ensemble of metamodels with optimized weight factors

Approximate mathematical models (metamodels) are often used as surrogates for more computationally intensive simulations. The common practice is to construct multiple metamodels based on a common training data set, evaluate their accuracy, and then to use only a single model perceived as the best while discarding the rest. This practice has some shortcomings as it does not take full advantage of the resources devoted to constructing different metamodels, and it is based on the assumption that changes in the training data set will not jeopardize the accuracy of the selected model. It is possible to overcome these drawbacks and to improve the prediction accuracy of the surrogate model if the separate stand-alone metamodels are combined to form an ensemble. Motivated by previous research on committee of neural networks and ensemble of surrogate models, a technique for developing a more accurate ensemble of multiple metamodels is presented in this paper. Here, the selection of weight factors in the general weighted-sum formulation of an ensemble is treated as an optimization problem with the desired solution being one that minimizes a selected error metric. The proposed technique is evaluated by considering one industrial and four benchmark problems. The effect of different metrics for estimating the prediction error at either the training data set or a few validation points is also explored. The results show that the optimized ensemble provides more accurate predictions than the stand-alone metamodels and for most problems even surpassing the previously reported ensemble approaches.     

Various approaches for constructing an ensemble of metamodels using local measures

Metamodels are approximate mathematical models used as surrogates for computationally expensive simulations. Since metamodels are widely used in design space exploration and optimization, there is growing interest in developing techniques to enhance their accuracy. It has been shown that the accuracy of metamodel predictions can be increased by combining individual metamodels in the form of an ensemble. Several efforts were focused on determining the contribution (or weight factor) of a metamodel in the ensemble using global error measures. In addition, prediction variance is also used as a local error measure to determine the weight factors. This paper investigates the efficiency of using local error measures, and also presents the use of the pointwise cross validation error as a local error measure as an alternative to using prediction variance. The effectiveness of ensemble models are tested on several problems with varying dimensionality: five mathematical benchmark problems, two structural mechanics problems and an automobile crash problem. It is found that the spatial ensemble models show better performances than the global ensemble for the low-dimensional problems, while the global ensemble is a more accurate model than the spatial ensembles for the high-dimensional problems. Ensembles based on pointwise cross validation error and prediction variance provide similar accuracy. The ensemble models based on local measures reduce cross validation errors drastically, but their performances are not that impressive in reducing the error evaluated at random test points, because the pointwise cross validation error is not a good surrogate for the error at a point.     

Performance of an ensemble of ordinary, universal, non-stationary and limit Kriging predictors

The selection of stationary or non-stationary Kriging to create a surrogate model of a black box function requires apriori knowledge of the nature of response of the function as these techniques are better at representing some types of responses than others. While an adaptive technique has been previously proposed to adjust the level of stationarity within the surrogate model such a model can be prohibitively expensive to construct for high dimensional problems. An alternative approach is to employ a surrogate model constructed from an ensemble of stationary and non-stationary Kriging models. The following paper assesses the accuracy and optimization performance of such a modelling strategy using a number of analytical functions and engineering design problems.     

Metamodel selection based on stepwise regression

Metamodels are often used to replace expensive simulations of engineering problems. When a training set is given, a series of metamodels can be constructed, and then there are two strategies to deal with these metamodels: (1) picking out the best one with the highest accuracy as an approximation of the computationally intensive simulation; and (2) combining all of them into an ensemble model. However, since the choice of approximate model depends on design of experiments (DOEs), employing of the first strategy thus increases the risk of adopting an inappropriate model. Nevertheless, the second strategy also seems not to be a good choice, since adding redundant metamodels may lead to loss of accuracy. Therefore, it is a necessary step to eliminate the redundant metamodels from the set of the candidates before constructing the final ensemble. Illuminated by the method of variable selection widely used in polynomial regression, a metamodel selection method based on stepwise regression is proposed. In our method, just a subset of n ones (n ≤ p, where p is the number of all of the candidate metamodels) is used. In addition, a new ensemble technique is proposed from the view of polynomial regression in this work. This new ensemble technique, combined with metamodel selection method, has been evaluated using six benchmark problems. The results show that eliminating the redundant metamodels before constructing the ensemble can provide more ideal prediction accuracy than directly constructing the ensemble by utilizing all of the candidates.     

A novel evolutionary-negative correlated mixture of experts model in tourism demand estimation

Mixtures of experts (ME) model are widely used in many different areas as a recognized ensemble learning approach to account for nonlinearities and other complexities in the data, such as time series estimation. With the aim of developing an accurate tourism demand time series estimation model, a mixture of experts model called LSPME (Lag Space Projected ME) is presented by combining ideas from subspace projection methods and negative correlation learning (NCL). The LSPME uses a new cluster-based lag space projection (CLSP) method to automatically obtain input space to train each expert focused on the difficult instances at each step of the boosting approach. For training experts of the LSPME, a new NCL algorithm called Sequential Evolutionary NCL algorithm (SENCL) is proposed that uses a moving average for the correlation penalty term in the error function of each expert to measure the error correlation between it and its previous experts. The LSPME model was compared with other ensemble models using monthly tourist arrivals to Japan from four markets: The United States, United Kingdom, Hong Kong and Taiwan. The experimental results show that the estimation accuracy of the proposed LSPME model is significantly better than the other ensemble models and can be considered to be a promising alternative for time series estimation problems.     

On polynomial response surfaces and Kriging for use in structural optimization of crashworthiness

The accuracy of different approximating response surfaces is investigated. In the classical response surface methodology (CRSM) the true response function is usually replaced with a low-order polynomial. In Kriging the true response function is replaced with a low-order polynomial and an error correcting function. In this paper the error part of the approximating response surface is obtained from simple point Kriging theory. The combined polynomial and error correcting function will be addressed as a Kriging surface approximation.To be able to use Kriging the spatial correlation or covariance must be known. In this paper the error is assumed to have a normal distribution and the covariance to depend only on one parameter. The maximum-likelihood method is used to find the latter parameter. A weighted least-square procedure is used to determine the trend before simple point Kriging is used for the error function. In CRSM the surface approximation is determined through an ordinary least-square fit. In both cases the D-optimality criterion has been used to distribute the design points.From this investigation we have found that a low-ordered polynomial assumption should be made with the Kriging approach. We have also concluded that Kriging better than CRSM resolves abrupt changes in the response, e.g. due to buckling, contact or plastic deformation.     

Ensemble of metamodels: extensions of the least squares approach to efficient global optimization

In this work we present LSEGO, an approach to drive efficient global optimization (EGO), based on LS (least squares) ensemble of metamodels. By means of LS ensemble of metamodels it is possible to estimate the uncertainty of the prediction with any kind of model (not only kriging) and provide an estimate for the expected improvement function. For the problems studied, the proposed LSEGO algorithm has shown to be able to find the global optimum with less number of optimization cycles than required by the classical EGO approach. As more infill points are added per cycle, the faster is the convergence to the global optimum (exploitation) and also the quality improvement of the metamodel in the design space (exploration), specially as the number of variables increases, when the standard single point EGO can be quite slow to reach the optimum. LSEGO has shown to be a feasible option to drive EGO with ensemble of metamodels as well as for constrained problems, and it is not restricted to kriging and to a single infill point per optimization cycle.     

On the ensemble of metamodels with multiple regional optimized weight factors

Metamodels are often used as surrogates for expensive high fidelity computational simulations (e.g., finite element analysis). Ensemble of metamodels (EM), which combines various types of individual metamodels in the form of a weighted average ensemble, is found to have improved accuracy over the individual metamodels used alone. Currently, there are mainly two kinds of EMs called as pointwise EM and average EM. The pointwise EM generally has better prediction accuracy than the average EM, but it is much more time-consuming than the average EM. In most cases, as a metamodel, EM is often used in the engineering design optimization which needs to invoke EM tens of thousands of times. Therefore, the average EM is still the most extensively used EM. To the authors’ best knowledge, the most accurate average EM is the EM with optimized weight factors proposed by Acar et al. However, the EM proposed by Acar et al. is often too “rigid” and may not have sufficient accuracy over some regions of the design space. In order to deal with this problem and further improve the prediction accuracy, a new EM with multiple regional optimized weight factors (EM-MROWF) is proposed in this study. In this new EM, the design space is divided into multiple subdomains each of which is assigned a set of optimized weight factors. This new EM was constructed by combining three typical individual metamodels, i.e., polynomial regression (PR), radial basis function (RBF), and Kriging (KRG). The proposed technique was evaluated by ten benchmark problems and two engineering application problems. The ten benchmark problems are typical mathematical functions for evaluating the approximation performance in previous studies. And, the two engineering application problems refer to the vehicular passive safety in the field of crashworthiness design. The study results showed that the EM-MROWF performed much better than the other existing average EMs as well as the three individual metamodels.     

基于C-AdaBoost模型的乳腺癌预测研究

机器学习和深度学习技术可用于解决医学分类预测中的许多问题,其中一些分类算法的预测精度较高,而另一些算法的精度有限。提出了基于C-AdaBoost模型的集成学习算法,对乳腺癌疾病进行预测,发现了判断乳腺癌是否复发、乳腺癌肿瘤是否为良性的最优特征组合。通过逐步回归方法对现有特征进行二次选取,并结合C-AdaBoost模型使得预测效果更优。大量实验表明,基于C-AdaBoost模型的算法的预测准确率比SVM、Naive Bayes、RandomForest以及传统的集成学习模型等机器学习分类器的准确率最多可提高19.5%,从而可以更好地帮助医生进行临床决策。     

面向汽车外形空气动力学优化的代理模型方法

针对代理模型在汽车外形气动优化上的适应性研究较少的现状,运用不同数量样本点构建径向基函数(Radial Basis Function,RBF)模型、多项式模型和Kriging模型等3种常用代理模型.对比发现,在样本点相同的情况下,RBF模型的精度最高,最优解更好.在样本点增加的基础上,多项式和Kriging模型的精度提高,但计算量也大幅增加;多项式最优解更接近RBF模型的最优解,而Kriging模型最优解仍不理想.综合评估可知RBF模型更适用于汽车外气动优化.     

Update strategies for kriging models used in variable fidelity optimization

Many optimization methods for simulation-based design rely on the sequential use of metamodels to reduce the associated computational burden. In particular, kriging models are frequently used in variable fidelity optimization. Nevertheless, such methods may become computationally inefficient when solving problems with large numbers of design variables and/or sampled data points due to the expensive process of optimizing the kriging model parameters in each iteration. One solution to this problem would be to replace the kriging models with traditional Taylor series response surface models. Kriging models, however, were shown to provide good approximations of computer simulations that incorporate larger amounts of data, resulting in better global accuracy. In this paper, a metamodel update management scheme (MUMS) is proposed to reduce the cost of using kriging models sequentially by updating the kriging model parameters only when they produce a poor approximation. The scheme uses the trust region ratio (TR-MUMS), which is a ratio that compares the approximation to the true model. Two demonstration problems are used to evaluate the proposed method: an internal combustion engine sizing problem and a control-augmented structural design problem. The results indicate that the TR-MUMS approach is very effective; on the demonstration problems, it reduced the number of likelihood evaluations by three orders of magnitude compared to using a global optimizer to find the kriging parameters in every iteration. It was also found that in trust region-based method, the kriging model parameters need not be updated using a global optimizer—local methods perform just as well in terms of providing a good approximation without affecting the overall convergence rate, which, in turn, results in a faster execution time.     

Ensemble of metamodels: the augmented least squares approach

In this work we present an approach to create ensemble of metamodels (or weighted averaged surrogates) based on least squares (LS) approximation. The LS approach is appealing since it is possible to estimate the ensemble weights without using any explicit error metrics as in most of the existent ensemble methods. As an additional feature, the LS based ensemble of metamodels has a prediction variance function that enables the extension to the efficient global optimization. The proposed LS approach is a variation of the standard LS regression by augmenting the matrices in such a way that minimizes the effects of multicollinearity inherent to calculation of the ensemble weights. We tested and compared the augmented LS approach with different LS variants and also with existent ensemble methods, by means of analytical and real-world functions from two to forty-four variables. The augmented least squares approach performed with good accuracy and stability for prediction purposes, in the same level of other ensemble methods and has computational cost comparable to the faster ones.     

普通Kriging方法建立宏观地层的验证与评价

空间插值方法对空间变异性和空间相关性的反映程度及其准确性与精度直接影响到三维地质模型的真实性。普通Kriging方法应用于数字地层中,其变异函数模型和参数选择对结果影响很大,利用最值、方差、相关系数、平均误差以及误差分布统计为主要比较指标,采用交叉验证方法结合误差分析,对球状模型、指数模型、线性模型进行比较研究,初步得出线性有块金模型优于其他变异函数模型的结论,对数字地层的工程应用及其真实性评价有一定指导意义。     

Ensemble data mining modeling in corrosion of concrete sewer: A comparative study of network-based (MLPNN & RBFNN) and tree-based (RF,CHAID, & CART) models

This research aims to evaluate ensemble learning (bagging, boosting, and modified bagging) potential in predicting microbially induced concrete corrosion in sewer systems from the data mining (DM) perspective. Particular focus is laid on ensemble techniques for network-based DM methods, including multi-layer perceptron neural network (MLPNN) and radial basis function neural network (RBFNN) as well as tree-based DM methods, such as chi-square automatic interaction detector (CHAID), classification and regression tree (CART), and random forests (RF). Hence, an interdisciplinary approach is presented by combining findings from material sciences and hydrochemistry as well as data mining analyses to predict concrete corrosion. The effective factors on concrete corrosion such as time, gas temperature, gas-phase H₂S concentration, relative humidity, pH, and exposure phase are considered as the models’ inputs. All 433 datasets are randomly selected to construct an individual model and twenty component models of boosting, bagging, and modified bagging based on training, validating, and testing for each DM base learners. Considering some model performance indices, (e.g., Root mean square error, RMSE; mean absolute percentage error, MAPE; correlation coefficient, r) the best ensemble predictive models are selected. The results obtained indicate that the prediction ability of the random forests DM model is superior to the other ensemble learners, followed by the ensemble Bag-CHAID method. On average, the ensemble tree-based models acted better than the ensemble network-based models; nevertheless, it was also found that taking the advantages of ensemble learning would enhance the general performance of individual DM models by more than 10%.     

A study into the potential of GPUs for the efficient construction and evaluation of Kriging models

The surrogate modelling technique known as Kriging, and its various derivatives, requires an optimization process to effectively determine the model’s defining parameters. This optimization typically involves the maximisation of a likelihood function which requires the construction and inversion of a correlation matrix dependent on the selected modelling parameters. The construction of such models in high dimensions and with a large numbers of sample points can, therefore, be considerably expensive. Similarly, once such a model has been constructed the evaluation of the predictor, error and other related design and model improvement criteria can also be costly. The following paper investigates the potential for graphical processing units to be used to accelerate the evaluation of the Kriging likelihood, predictor and error functions. Five different Kriging formulations are considered including, ordinary, universal, non-stationary, gradient-enhanced and multi-fidelity Kriging. Other key contributions include the derivation of the adjoint of the likelihood function for a fully and partially gradient-enhanced Kriging model as well as the presentation of novel schemes to accelerate the likelihood optimization via a mixture of single and double precision calculations and by automatically selecting the best hardware to perform the evaluations on.     

Performance study of gradient-enhanced Kriging

The use of surrogate models for approximating computationally expensive simulations has been on the rise for the last two decades. Kriging-based surrogate models are popular for approximating deterministic computer models. In this work, the performance of Kriging is investigated when gradient information is introduced for the approximation of computationally expensive black-box simulations. This approach, known as gradient-enhanced Kriging, is applied to various benchmark functions of varying dimensionality (2D-20D). As expected, results from the benchmark problems show that additional gradient information can significantly enhance the accuracy of Kriging. Gradient-enhanced Kriging provides a better approximation even when gradient information is only partially available. Further comparison between gradient-enhanced Kriging and an alternative formulation of gradient-enhanced Kriging, called indirect gradient-enhanced Kriging, highlights various advantages of directly employing gradient information, such as improved surrogate model accuracy, better conditioning of the correlation matrix, etc. Finally, gradient-enhanced Kriging is used to model 6- and 10-variable fluid–structure interaction problems from bio-mechanics to identify the arterial wall’s stiffness.     

Sparsity-promoting polynomial response surface: A new surrogate model for response prediction

Computation-intensive analyses/simulations are becoming increasingly common in engineering design problems. To improve the computation efficiency, surrogate models are used to replace expensive simulations of engineering problems. This paper proposes a new high-fidelity surrogate modeling approach which is called the Sparsity-promoting Polynomial Response Surface (SPPRS). In the SPPRS model, a series of Legendre polynomials is selected as basis functions, and its number is compatible with the sample size so as to enhance the expression ability for complex functional relationships. The coefficients associated with basis functions are estimated using a “sparsity-promoting” regression approach which is an ensemble of two techniques: least squares and ℓ₁-norm regularization. As a result, only these basis functions relevant to explain the function relationship are picked out, and that dedicates to ease the problem of overfitting for training points. With the sparsity-promoting regression approach, such a surrogate model intends to capture both the global trend of the functional variation and a reasonable local accuracy in the neighborhood of training points. Additionally, Latin hypercube design (LHD) is proved conducive to improving the predictive capability of our model. The SPPRS is applied to seven benchmark test functions and a complex engineering problem. The results illustrate the promising benefits of this novel surrogate modeling technique.     

On model typing

Where object-oriented languages deal with objects as described by classes, model-driven development uses models, as graphs of interconnected objects, described by metamodels. A number of new languages have been and continue to be developed for this model-based paradigm, both for model transformation and for general programming using models. Many of these use single-object approaches to typing, derived from solutions found in object-oriented systems, while others use metamodels as model types, but without a clear notion of polymorphism. Both of these approaches lead to brittle and overly restrictive reuse characteristics. In this paper we propose a simple extension to object-oriented typing to better cater for a model-oriented context, including a simple strategy for typing models as a collection of interconnected objects. We suggest extensions to existing type system formalisms to support these concepts and their manipulation. Using a simple example we show how this extended approach permits more flexible reuse, while preserving type safety.     

基于深度LightGBM集成学习模型的谷歌商店顾客购买力预测

针对轻量化梯度促进机（LightGBM）等集成学习模型只对数据信息进行一次挖掘,无法自动地细化数据挖掘粒度或通过深入挖掘得到更多的数据中潜在内部关联信息的问题,提出了深度LightGBM集成学习模型,该模型由滑动窗口和加深两部分组成。首先,通过滑动窗口使得集成学习模型能够自动地细化数据挖掘粒度,从而更加深入地挖掘数据中潜在的内部关联信息,同时赋予模型一定的表示学习能力。然后,基于滑动窗口,用加深步骤进一步地提升模型的表示学习能力。最后,结合特征工程对数据集进行处理。在谷歌商店数据集上进行的实验结果表明,所提深度集成学习模型相较原始集成学习模型的预测精度高出6.16个百分点。所提方法能够自动地细化数据挖掘粒度,从而获取更多数据集中的潜在信息,并且深度LightGBM集成学习模型与传统深度神经网络相比是非神经网络的深度模型,参数更少,可解释性更强。