基于估值不确定度排序顺序均值采样的昂贵高维多目标进化算法

模型管理,特别是训练样本的选择和填充采样准则,是影响昂贵多目标优化算法求解性能的重要因素.为此,选择样本库中具有较好目标函数值的若干个体作为样本训练目标函数的代理模型,使用基于参考向量的进化算法搜索模型的最优解集,并提出一种基于个体目标函数估值不确定度排序顺序均值的采样策略,从该最优解集中选择两个个体进行真实的目标函数评价.为了验证算法的有效性,将所提出算法在DTLZ和WFG多目标优化测试问题和两个实际工程优化问题上进行测试,并与其他5种优秀的同类型算法进行结果对比.实验结果表明,所提出算法在求解昂贵高维多目标优化问题上是有效的.     

Variable selection in model-based clustering: A general variable role modeling

The currently available variable selection procedures in model-based clustering assume that the irrelevant clustering variables are all independent or are all linked with the relevant clustering variables. A more versatile variable selection model is proposed, taking into account three possible roles for each variable: The relevant clustering variables, the irrelevant clustering variables dependent on a part of the relevant clustering variables and the irrelevant clustering variables totally independent of all the relevant variables. A model selection criterion and a variable selection algorithm are derived for this new variable role modeling. The model identifiability and the consistency of the variable selection criterion are also established. Numerical experiments highlight the interest of this new modeling.     

An efficient variable screening method for effective surrogate models for reliability-based design optimization

In the reliability-based design optimization (RBDO) process, surrogate models are frequently used to reduce the number of simulations because analysis of a simulation model takes a great deal of computational time. On the other hand, to obtain accurate surrogate models, we have to limit the dimension of the RBDO problem and thus mitigate the curse of dimensionality. Therefore, it is desirable to develop an efficient and effective variable screening method for reduction of the dimension of the RBDO problem. In this paper, requirements of the variable screening method for deterministic design optimization (DDO) and RBDO are compared, and it is found that output variance is critical for identifying important variables in the RBDO process. An efficient approximation method based on the univariate dimension reduction method (DRM) is proposed to calculate output variance efficiently. For variable screening, the variables that induce larger output variances are selected as important variables. To determine important variables, hypothesis testing is used in this paper so that possible errors are contained in a user-specified error level. Also, an appropriate number of samples is proposed for calculating the output variance. Moreover, a quadratic interpolation method is studied in detail to calculate output variance efficiently. Using numerical examples, performance of the proposed method is verified. It is shown that the proposed method finds important variables efficiently and effectively     

基于回归系数的变量筛选方法用于近红外光谱分析

提出了一种基于回归系数的变量逐步筛选方法。对光谱中各变量计算其回归系数后,按其绝对值由大到小将相应变量排列,采用PLS交互检验按前向选择法逐步选择最佳变量子集。用该方法对玉米和柴油近红外光谱数据进行分析,对玉米蛋白质、柴油十六烷值和粘度分别选择出了14、12以及30个最佳变量用于建模,所得预测结果均优于全谱变量建模的预测结果。可见本方法是一种有效实用的近红外光谱变量选择方法。     

一种基于互信息变量选择的极端学习机算法

针对回归问题中存在的变量选择和网络结构设计问题, 提出一种基于互信息的极端学习机(ELM) 训练算法, 同时实现输入变量的选择和隐含层的结构优化. 该算法将互信息输入变量选择嵌入到ELM网络的学习过程之中, 以网络的学习性能作为衡量输入变量与输出变量相关与否的指标, 并以增量式的方法确定隐含层节点的规模.在Lorenz、Gas Furnace 和10 组标杆数据上的仿真结果表明了所提出算法的有效性. 该算法不仅可以简化网络结构, 还可以提高网络的泛化性能.

     

A rule-based method for automated surrogate model selection

Surrogate models have been widely used in engineering design because of their capability to approximate computationally complex engineering systems. In practice, the choice of surrogate models is extremely important since there are many types of surrogate models, and they also have different hyper-parameters. Traditional manual selection approaches are very time-consuming and cannot be generalized. To address these challenges, an evolutionary algorithm (EA)-based approaches are proposed and studied. However, they lack interpretability and are computationally expensive. To address these gaps, we create a rule-based method for an automatic surrogate model selection called AutoSM. The drastic increase in the selection pace by pre-screening of surrogate model types based on selection rule extraction is the scientific contribution of our proposed method. First, an interpretable decision tree is built to map four critical features, including problem scale, noise, size of sample and nonlinearity, to the types of surrogate model and select the promising surrogate model; then, a genetic algorithm (GA) is used to find the appropriate hyper-parameters for each selected surrogate model. The AutoSM is tested with three theoretical problems and two engineering problems, including a hot rod rolling and a blowpipe design problem. According to the empirical results, using the proposed AutoSM, we can find the promising surrogate model and associated hyper-parameter in 9 times less than other automatic selection approaches such as concurrent surrogate model selection (COSMOS) while maintaining the same accuracy and robustness in surrogate model selection. Besides, the proposed AutoSM, unlike previous EA-based automatic surrogate model selection methods, is not a black box and is interpretable.     

Development of a variable selection method for soft sensor using artificial neural network and nonnegative garrote

This paper developed a new variable selection method for soft sensor applications using the nonnegative garrote (NNG) and artificial neural network (ANN). The proposed method employs the ANN to generate a well-trained network, and then uses the NNG to conduct the accurate shrinkage of input weights of the ANN. This paper took Bayesian information criterion as the model evaluation criterion, and the optimal garrote parameter s was determined by v-fold cross-validation. The performance of the proposed algorithm was compared to existing state-of-art variable selection methods. Two artificial dataset examples and a real industrial application for air separation process were applied to demonstrate the performance of the methods. The experimental results showed that the proposed method presented better model accuracy with fewer variables selected, compared to other state-of-art methods.     

Sparse modeling using orthogonal forward regression with PRESS statistic and regularization

The paper introduces an efficient construction algorithm for obtaining sparse linear-in-the-weights regression models based on an approach of directly optimizing model generalization capability. This is achieved by utilizing the delete-1 cross validation concept and the associated leave-one-out test error also known as the predicted residual sums of squares (PRESS) statistic, without resorting to any other validation data set for model evaluation in the model construction process. Computational efficiency is ensured using an orthogonal forward regression, but the algorithm incrementally minimizes the PRESS statistic instead of the usual sum of the squared training errors. A local regularization method can naturally be incorporated into the model selection procedure to further enforce model sparsity. The proposed algorithm is fully automatic, and the user is not required to specify any criterion to terminate the model construction procedure. Comparisons with some of the existing state-of-art modeling methods are given, and several examples are included to demonstrate the ability of the proposed algorithm to effectively construct sparse models that generalize well.     

一种新的回归变量选择方法

本文就多元线性回归模型,提出一种回归变量选择方法,其基本思想是利用矩阵的扫描运算和变量间的相关性分析,根据自变量（回归变量）与因变量之间相关性越强对因变量的解释作用越强的原则,选择回归变量。用本文提出的方法选择回归变量,具有降低回归模型复杂度、提高模型精确性和可靠性、便于工程实现的优点。仿真算例证实了该方法的有效性和可行性。     

一种基于多代理模型的混合整数规划优化方法

提出一种基于多代理模型的优化方法,求解混合整数规划问题.首先,基于群智能优化策略提出一种基于多群体协作模型的采样方法,保证候选解的正确性和多样性;其次,采用基于数据并行的高斯过程建模方法,在线构造局部代理模型;再次,通过多代理模型对候选解进行预筛选,实现与粒子群算法的协同优化;最后,通过14个测试问题和一个基于数据驱动的模型参数选取问题,验证所提出方法的有效性.     

Incorporating feature selection method into support vector regression for stock index forecasting

Stock index forecasting is one of the most difficult tasks that financial organizations, firms and private investors have to face. Support vector regression (SVR) has become a popular alternative in stock index forecasting tasks due to its generalization capability in obtaining a unique solution. However, the major limitation of SVR is that it cannot capture the relative importance of independent variables to the dependent variable when many potential independent variables are considered. This study incorporates feature selection method and SVR for building stock index forecasting model. The proposed model uses multivariate adaptive regression splines (MARS), an effective nonlinear and nonparametric regression methodology, to identify important forecasting variables. The obtained significant predictor variables are then served as the inputs for the SVR model. Experimental results reveal that the obtained important variables from MARS can improve the forecasting performance of the SVR models. Moreover, the MARS results provide useful information about the relationship between the selected predictor variables and stock index through the obtained basis functions, important predictor variables and the MARS prediction function. Hence, the proposed stock index forecasting model can generate good forecasting performance and exhibits the capability of identifying significant predictor variables, which provide valuable information for further investment decisions/strategies.     

A backward elimination discrete optimization algorithm for model selection in spatio-temporal regression models

Regression models are used in geosciences to extrapolate data and identify significant predictors of a response variable. Criterion approaches based on the residual sum of squares (RSS), such as the Akaike Information Criterion, Bayesian Information Criterion (BIC), Deviance Information Criterion, or Mallows' Cp can be used to compare non-nested models to identify an optimal subset of covariates. Computational limitations arise when the number of observations or candidate covariates is large in comparing all possible combinations of the available covariates, and in characterizing the covariance of the residuals for each examined model when the residuals are autocorrelated, as is often the case in spatial and temporal regression analysis. This paper presents computationally efficient algorithms for identifying the optimal model as defined using any RSS-based model selection criterion. The proposed dual criterion optimal branch and bound (DCO B&B) algorithm is guaranteed to identify the optimal model, while a single criterion heuristic (SCH) B&B algorithm provides further computational savings and approximates the optimal solution. These algorithms are applicable both to multiple linear regression (MLR) and to response variables with correlated residuals. We also propose an approach for iterative model selection, where a single set of covariance parameters is used in each iteration rather than a different set of parameters being used for each examined model. Simulation experiments are performed to evaluate the performance of the algorithms for regression models, using MLR and geostatistical regression as prototypical regression tools and BIC as a prototypical model selection approach. Results show massive computational savings using the DCO B&B algorithm relative to performing an exhaustive search. The SCH B&B is shown to provide a good approximation of the optimal model in most cases, while the DCO B&B with iterative covariance parameter optimization yields the closest approximation to the DCO B&B algorithm while also providing additional computational savings.     

一种基于自适应代理模型的并行贝叶斯优化方法

提出一种基于自适应代理模型的并行贝叶斯优化方法,用于求解计算成本高的复杂优化问题.该方法基于多点期望改进判据,通过批次采样实现并行优化.针对并行优化产生的大量历史数据会导致全局代理模型建模成本高的问题,提出一种改进的基于数据并行的高斯过程建模方法,在线构造局部代理模型.此外,针对多点期望改进判据计算成本高的问题,提出一种启发式的分层优化策略,通过序贯优化基于自适应代理模型的单点期望改进判据,近似计算多点期望改进判据.最后通过5个测试问题验证所提出方法的有效性.     

An improved model-based evolutionary algorithm for multi-objective optimization

The basic idea in the estimation of distribution algorithms is the replacement of heuristic operators with machine learning models such as regression models, clustering models, or classification models. So, recently, the model-based evolutionary algorithms (MBEAs) have been suggested in three groups: The estimation of distribution algorithms (EDAs), surrogate assisted evolutionary algorithms, and the inversed models to map from the objective space to the decision space. In this article, a new approach, based on an inversed model of Gaussian process and random forest framework, is proposed. The main idea is applying the process of random forest variable importance with a random grouping that determines some of the best assignment of decision variables to objective functions in order to form a Gaussian process in inverse models that maps to decision space the rich solutions which are discovered from objective space. Then these inverse models through sampling the objective space generate offspring. The proposed algorithm has been tested on the benchmark test suite for evolutionary algorithms (modified Deb K, Thiele L, Laumanns M, Zitzler E (DTLZ), and Walking Fish Group (WFG)) and indicates that the proposed method is a competitive and promising approach.     

An inferential modeling method using enumerative PLS based nonnegative garrote regression

In this paper a robust linear regression method with variable selection is proposed for predicting desirable end-of-line quality variables in complex industrial processes. The development of such prediction models is challenging because there is usually a large pool of candidate explanatory variables, limited sample data, and multicollinearity among explanatory variables. The proposed method is named as the enumerative partial least square based nonnegative garrote regression. It employs partial least square regression in enumerative manner to generate initial model coefficients and then uses a nonnegative garrote method to shrink original coefficients so that irrelevant variables can be eliminated implicitly. Analysis about the advantages of the proposed method is provided compared to existing state-of-art model construction methods. Two simulation examples as well as an industrial application in a local semiconductor factory unit are used to validate the proposed method. These examples witness substantial improvement in terms of accuracy and robustness in variable selection compared to existing methods. Specifically, for the industrial case the percentages of improvement in terms of root mean squared error is up to 24.3% compared with the previous work.     

基于动态MANCOVA的混合制程虚拟测量算法

在晶圆/液晶面板等批次加工过程中,产品质量的及时估计与品质管制是提高产能和降低成本的有效途径.针对"少量多样"的混合制程,利用逐步回归算法挑选该制程的关毽变量,引入产品的效益因子,建立混合制程的虚拟测量模型;为克服系统扰动对模型精度的影响,以产品效益因子为状态量建立该制程的状态方程,利用Kalman滤波器递归估计模型参数得到动态的MANCOVA模型;最后通过某湿式蚀刻制程的工程应用验证了该算法的有效性.     

A random-effect model approach for group variable selection

We consider regression models with a group structure in explanatory variables. This structure is commonly seen in practice, but it is only recently realized that taking the information into account in the modeling process may improve both the interpretability and accuracy of the model. In this paper, we study a new approach to group variable selection using random-effect models. Specific distributional assumptions on random effects pertaining to a given structure lead to a new class of penalties that include some existing penalties. We also develop an efficient computational algorithm. Numerical studies are provided to demonstrate better sensitivity and specificity properties without sacrificing the prediction accuracy. Finally, we present some real-data applications of the proposed approach.     

Jackknife and bootstrap methods in the identification of dynamic models

A new criterion based on a Jackknife or a Bootstrap statistic is proposed for identifying non-parsimonious dynamic models (FIR, ARX). It is applicable for selecting the number of components in latent variable regression methods or the constraining parameter in regularized least squares regression methods. These meta parameters are used to overcome ill-conditioning caused by model over-parameterization, when fitted using prediction error or least squares methods. In all cases studied, using PLS for parameter estimation, the proposed criterion led to the selection of better models, in the mean square error sense, than when selected via cross-validation. The methodology also provides approximate confidence intervals for the model parameters and the step and impulse response of the system.     

Applying input variables selection technique on input weighted support vector machine modeling for BOF endpoint prediction

Basic oxygen furnace (BOF) steelmaking is a complex process and dynamic model is very important for endpoint control. It is usually difficult to build a precise BOF endpoint dynamic model because many input variables affect the endpoint carbon content and temperature. For this problem, two effective variables selection steps: mechanism analysis and mutual information calculation are proposed to choose appropriate input variables according to a variable selection algorithm. Then, the selected inputs are weighted on the basis of mutual information values. Finally, two input weighted support vector machine BOF endpoint dynamic models are constructed to predict endpoint carbon content and temperature. Results show that the variable selection for BOF endpoint prediction model is essential and effective. The complexity and precise of two endpoint prediction models are improved.     

Kernelized LARS–LASSO for constructing radial basis function neural networks

Model structure selection is of crucial importance in radial basis function (RBF) neural networks. Existing model structure selection algorithms are essentially forward selection or backward elimination methods that may lead to sub-optimal models. This paper proposes an alternative selection procedure based on the kernelized least angle regression (LARS)–least absolute shrinkage and selection operator (LASSO) method. By formulating the RBF neural network as a linear-in-the-parameters model, we derive a l ₁-constrained objective function for training the network. The proposed algorithm makes it possible to dynamically drop a previously selected regressor term that is insignificant. Furthermore, inspired by the idea of LARS, the computing of output weights in our algorithm is greatly simplified. Since our proposed algorithm can simultaneously conduct model structure selection and parameter optimization, a network with better generalization performance is built. Computational experiments with artificial and real world data confirm the efficacy of the proposed algorithm.