On the ensemble of metamodels with multiple regional optimized weight factors

Metamodels are often used as surrogates for expensive high fidelity computational simulations (e.g., finite element analysis). Ensemble of metamodels (EM), which combines various types of individual metamodels in the form of a weighted average ensemble, is found to have improved accuracy over the individual metamodels used alone. Currently, there are mainly two kinds of EMs called as pointwise EM and average EM. The pointwise EM generally has better prediction accuracy than the average EM, but it is much more time-consuming than the average EM. In most cases, as a metamodel, EM is often used in the engineering design optimization which needs to invoke EM tens of thousands of times. Therefore, the average EM is still the most extensively used EM. To the authors’ best knowledge, the most accurate average EM is the EM with optimized weight factors proposed by Acar et al. However, the EM proposed by Acar et al. is often too “rigid” and may not have sufficient accuracy over some regions of the design space. In order to deal with this problem and further improve the prediction accuracy, a new EM with multiple regional optimized weight factors (EM-MROWF) is proposed in this study. In this new EM, the design space is divided into multiple subdomains each of which is assigned a set of optimized weight factors. This new EM was constructed by combining three typical individual metamodels, i.e., polynomial regression (PR), radial basis function (RBF), and Kriging (KRG). The proposed technique was evaluated by ten benchmark problems and two engineering application problems. The ten benchmark problems are typical mathematical functions for evaluating the approximation performance in previous studies. And, the two engineering application problems refer to the vehicular passive safety in the field of crashworthiness design. The study results showed that the EM-MROWF performed much better than the other existing average EMs as well as the three individual metamodels.     

A rule-based method for automated surrogate model selection

Surrogate models have been widely used in engineering design because of their capability to approximate computationally complex engineering systems. In practice, the choice of surrogate models is extremely important since there are many types of surrogate models, and they also have different hyper-parameters. Traditional manual selection approaches are very time-consuming and cannot be generalized. To address these challenges, an evolutionary algorithm (EA)-based approaches are proposed and studied. However, they lack interpretability and are computationally expensive. To address these gaps, we create a rule-based method for an automatic surrogate model selection called AutoSM. The drastic increase in the selection pace by pre-screening of surrogate model types based on selection rule extraction is the scientific contribution of our proposed method. First, an interpretable decision tree is built to map four critical features, including problem scale, noise, size of sample and nonlinearity, to the types of surrogate model and select the promising surrogate model; then, a genetic algorithm (GA) is used to find the appropriate hyper-parameters for each selected surrogate model. The AutoSM is tested with three theoretical problems and two engineering problems, including a hot rod rolling and a blowpipe design problem. According to the empirical results, using the proposed AutoSM, we can find the promising surrogate model and associated hyper-parameter in 9 times less than other automatic selection approaches such as concurrent surrogate model selection (COSMOS) while maintaining the same accuracy and robustness in surrogate model selection. Besides, the proposed AutoSM, unlike previous EA-based automatic surrogate model selection methods, is not a black box and is interpretable.     

The integration of design of experiments, surrogate modeling and optimization for thermoscience research

This paper presents an integrated approach for the solution of complex optimization problems in thermoscience research. The cited approach is based on the design of computational experiments (DOE), surrogate modeling, and optimization. The DOE/surrogate modeling techniques under consideration include: A-optimal/classical linear regression, Latin hypercube/artificial neural networks, and Latin hypercube/Sugeno-type fuzzy models. These techniques are coupled with both local (modified Newtons method) and global (genetic algorithms) optimization methods. The proposed approach proved to be an effective, efficient and robust modeling and optimization tool in the context of a case study, and holds promise for use in larger scale optimization problems in thermoscience research.     

Ensemble of metamodels: the augmented least squares approach

In this work we present an approach to create ensemble of metamodels (or weighted averaged surrogates) based on least squares (LS) approximation. The LS approach is appealing since it is possible to estimate the ensemble weights without using any explicit error metrics as in most of the existent ensemble methods. As an additional feature, the LS based ensemble of metamodels has a prediction variance function that enables the extension to the efficient global optimization. The proposed LS approach is a variation of the standard LS regression by augmenting the matrices in such a way that minimizes the effects of multicollinearity inherent to calculation of the ensemble weights. We tested and compared the augmented LS approach with different LS variants and also with existent ensemble methods, by means of analytical and real-world functions from two to forty-four variables. The augmented least squares approach performed with good accuracy and stability for prediction purposes, in the same level of other ensemble methods and has computational cost comparable to the faster ones.     

神经网络集成的设计与应用

传统的神经网络一般采用个体网络,其应用效果很大程度上取决于使用者的经验,且网络的泛化能力不强.一种改进的神经网络集成方法,为传统神经网络存在的问题提供了一个简易的解决方案.由理论分析和实验结果可以得出结论,神经网络集成方法比传统的个体网络方法的效果更好.     

基于半监督回归的选择性集成算法

为了提高小数据量的有标记样本问题中学习器的性能,结合半监督学习和选择性集成学习,提出了基于半监督回归的选择性集成算法SSRES。算法基于半监督学习的基本思想,同时使用有标记样本和未标记样本训练学习器从而减少对有标记样本的需求,使用选择性集成算法GRES对不同学习器进行适当的选择,并将选择的结果结合提高学习器的泛化能力。实验结果表明,在小数据量的有标记样本问题中,该算法能够有效地提高学习器的性能。     

Ensemble methods for multi-label classification

Ensemble methods have been shown to be an effective tool for solving multi-label classification tasks. In the RAndom k-labELsets (RAKEL) algorithm, each member of the ensemble is associated with a small randomly-selected subset of k labels. Then, a single label classifier is trained according to each combination of elements in the subset. In this paper we adopt a similar approach, however, instead of randomly choosing subsets, we select the minimum required subsets of k labels that cover all labels and meet additional constraints such as coverage of inter-label correlations. Construction of the cover is achieved by formulating the subset selection as a minimum set covering problem (SCP) and solving it by using approximation algorithms. Every cover needs only to be prepared once by offline algorithms. Once prepared, a cover may be applied to the classification of any given multi-label dataset whose properties conform with those of the cover. The contribution of this paper is two-fold. First, we introduce SCP as a general framework for constructing label covers while allowing the user to incorporate cover construction constraints. We demonstrate the effectiveness of this framework by proposing two construction constraints whose enforcement produces covers that improve the prediction performance of random selection by achieving better coverage of labels and inter-label correlations. Second, we provide theoretical bounds that quantify the probabilities of random selection to produce covers that meet the proposed construction criteria. The experimental results indicate that the proposed methods improve multi-label classification accuracy and stability compared to the RAKEL algorithm and to other state-of-the-art algorithms.     

Ensemble strategies with adaptive evolutionary programming

Mutation operators such as Gaussian, Lévy and Cauchy have been used with evolutionary programming (EP). According to the no free lunch theorem, it is impossible for EP with a single mutation operator to outperform always. For example, Classical EP (CEP) with Gaussian mutation is better at searching in a local neighborhood while the Fast EP (FEP) with the Cauchy mutation performs better over a larger neighborhood. Motivated by these observations, we propose an ensemble approach where each mutation operator has its associated population and every population benefits from every function call. This approach enables us to benefit from different mutation operators with different parameter values whenever they are effective during different stages of the search process. In addition, the recently proposed Adaptive EP (AEP) using Gaussian (ACEP) and Cauchy (AFEP) mutations is also evaluated. In the AEP, the strategy parameter values are adapted based on the search performance in the previous few generations. The performance of ensemble is compared with a mixed mutation strategy, which integrates several mutation operators into a single algorithm as well as against the AEP with a single mutation operator. Improved performance of the ensemble over the single mutation-based algorithms and mixed mutation algorithm is verified using statistical tests.     

Ensemble classification of paired data

In many medical applications, data are taken from paired organs or from repeated measurements of the same organ or subject. Subject based as opposed to observation based evaluation of these data results in increased efficiency of the estimation of the misclassification rate. A subject based approach for classification in the generation of bootstrap samples of bagging and bundling methods is analyzed. A simulation model is used to compare the performance of different strategies to create the bootstrap samples which are used to grow individual trees. The proposed approach is compared to linear discriminant analysis, logistic regression, random forests and gradient boosting. Finally, the simulation results are applied to glaucoma diagnosis using both eyes of glaucoma patients and healthy controls. It is demonstrated that the proposed subject based resampling reduces the misclassification rate.     

基于成对差异性度量的选择性集成方法

有效地产生泛化能力强、差异大的个体学习器,是集成学习算法的关键。为了提高学习器的差异性和精度,文中提出一种基于成对差异性度量的选择性集成方法。同时研究一种改进方法,进一步提高方法的运算速度,且支持并行计算。最后通过使用BP神经网络作为基学习器,在UCI数据集上进行实验,并与Bagging、基于遗传算法的选择性集成(GASEN)算法进行比较。实验结果表明,该改进算法在性能上与GASEN算法相近的前提下,训练速度得到大幅提高。     

一种基于非负矩阵分解的聚类集成算法

为了解决通过原始数据集获得的基聚类结果存在一定的信息丢失,从而使得集成阶段的有效信息减少的问题,提出了一种基于非负矩阵分解的K-means聚类集成算法。该算法先利用K-means聚类算法获得集成信息矩阵,然后从原始数据集获取数据相关性,将两者结合后通过非负矩阵分解(NMF)技术构建共识函数以获得最终结果。实验证明,所提算法可以有效获取原始数据的潜在信息,并提高聚类质量。     

基于正则化互信息和差异度的集成特征选择

如何构造差异性大的基分类器是集成学习研究的重点,为此提出迭代循环选择法:以最大化正则互信息为准则提取最优特征子集,进而基于此训练得到基分类器;同时以错分样本个数作为差异性度量准则来评价所得基分类器的性能,若满足条件则停止,反之则循环迭代直至结束.最后用加权投票法融合所选基分类器的识别结果.通过仿真实验验证算法的有效性,以支持向量机为分类器,在公共数据集UCI上进行实验,并与单SVM及经典的Bagging集成算法和特征Bagging集成算法进行对比.实验结果显示,该方法可获得较高的分类精度.     

基于分割集成的行人检测方法

为提高行人检测的准确率,提出基于分割集成的方法用于静态图片中的行人检测。先将每个训练样本均匀分割成若干区域,提取特征后利用AdaBoost算法对每个区域建立一个局部分类器,这些局部分类器加权组成一个全局分类器。采用不同的分割方法重复上述过程,得到多个全局分类器。为进一步提高检测效果,得到更好的平均性能,对每种分割方法分别使用方向梯度直方图、多尺度方向梯度直方图特征建立2个全局分类器。当检测新的窗口时,集成上述全局分类器,通过加权投票的方式决定最终的检测结果。在INRIA公共测试集上的实验表明,文中方法有效提高检测效果。     

一种可并行的贝叶斯集合在线学习算法

无论是Boosting还是Bagging算法，在使用连续样本集进行分类器集合学习时，均需缓存大量数据，这对大容量样本集的应用不可行。本文提出一种基于贝叶斯集合的在线学习算法BEPOL，在保持Boosting算法加权采样思想的前提下，只需对样本集进行一次扫描，就可实现对贝叶斯集合的在线更新学习。算法针对串行训练时间长、成员相关性差的缺点，采用了并行学习的思想，通过将各贝叶斯分量映射到并行计算结构上，提高集合学习的效率。通过UCI数据集的实验表明，算法BEPOL具有与批量学习算法相近的分类性能和更小的时间开销，这使得算法对某些具有时间和空间限制的应用，如大型数据集或连续型数据集应用尤其有效。     

Ensemble member generation for sequential data assimilation

Using an ensemble of model forecasts to describe forecast error covariance extends linear sequential data assimilation schemes to nonlinear applications. This approach forms the basis of the Ensemble Kalman Filter and derivative filters such as the Ensemble Square Root Filter. While ensemble data assimilation approaches are commonly reported in the scientific literature, clear guidelines for effective ensemble member generation remain scarce. As the efficiency of the filter is reliant on the accurate determination of forecast error covariance from the ensemble, this paper describes an approach for the systematic determination of random error. Forecast error results from three factors: errors in initial condition, forcing data and model equations. The method outlined in this paper explicitly acknowledges each of these sources in the generation of an ensemble. The initial condition perturbation approach presented optimally spans the dynamic range of the model states and allows an appropriate ensemble size to be determined. The forcing data perturbation approach treats forcing observations differently according to their nature. While error from model physics is not dealt with in detail, discussion of some commonly used approaches and their limitations is provided. The paper concludes with an example application for a synthetic coastal hydrodynamic experiment assimilating sea surface temperature (SST) data, which shows better prediction capability when contrasted with standard approaches in the literature.     

基于广义回归网络的动态权重回归型神经网络集成方法研究*

神经网络集成技术能有效地提高神经网络的预测精度和泛化能力,已成为机器学习和神经计算领域的一个研究热点。针对回归分析问题提出了一种动态确定结果合成权重的神经网络集成构造方法,在训练出个体神经网络之后,根据各个体网络在输入空间上对训练样本的预测误差,应用广义回归网络来动态地确定各个体网络在特定输入空间上的权重。实验结果表明,与传统的简单平均和加权平均方法相比,本集成方法能取得更好的预测精度。     

Ensemble control of linear systems with parameter uncertainties

In this paper, we study the optimal control problem for a class of four-dimensional linear systems based on quaternionic and Fourier analysis. When the control is unconstrained, the optimal ensemble controller for this linear ensemble control systems is given in terms of prolate spheroidal wave functions. For the constrained convex optimisation problem of such systems, the quadratic programming is presented to obtain the optimal control laws. Simulations are given to verity the effectiveness of the proposed theory.     

ModEnPBT: A Modified Backtracking Ensemble Pruning algorithm

This paper proposes a new Modified Backtracking Ensemble Pruning algorithm (ModEnPBT), which is based upon the design idea of our previously proposed Ensemble Pruning via Backtracking algorithm (EnPBT), and however, aiming at overcoming its drawback of redundant solution space definition. Solution space of ModEnPBT is compact with no repeated solution vectors, therefore it possesses relatively higher searching efficiency compared with EnPBT algorithm. ModEnPBT still belongs to the category of Backtracking algorithm, which can systematically search for the solutions of a problem in a manner of depth-first, which is suitable for solving all those large-scale combinatorial optimization problems. Experimental results on three benchmark classification tasks demonstrate the validity and effectiveness of the proposed ModEnPBT.     

基于Ensemble的增量分类方法

针对在维护数据挖掘模型过程中须反复计算数据集、效率较低的问题,基于Ensembles学习思想,研究增量数据集的弱分类器生成方法,根据增量数据集分类器之间的相异度提出新的组合分类算法,分析组合分类器的出错率。实验结果表明,该分类方法是有效的。     

基于交叉分组技术的集成算法研究

集成学习主要通过扰动训练数据集来产生较强泛化能力.研究者们提出了各种各样的方法来实现这一目标,但如何扰动训练数据集以达到最佳的泛化能力并没有被深入研究.本文中,提出了对训练数据集进行扰动的交叉分组(cross-grouping)方法,通过改变交叉因子以实现对训练数据集不同程度的扰动,从而实现当集成规模较小时,得到更强的泛化能力.实验表明,当选择合适的交叉因子时,GG-Bagging泛化能力要强于Bagging和Boosting,略优于Decorate和Random Forests.