Forward stage‐wise ensemble regression algorithm to improve base regressors prediction ability: an empirical study

Analysis of scientific data requires accurate regressor algorithms to decrease prediction errors. Lots of machine learning algorithms, that is, neural networks, rule‐based algorithms, regression trees and some kinds of lazy learners, are used to realize this need. In recent years, different ensemble regression strategies were improved to obtain enhanced predictors with lower forecasting errors. Ensemble algorithms combine good models that make errors in different parts of analyzed data. There are mainly two approaches in ensemble regression algorithm generation; boosting and bagging. The aim of this article is to evaluate a boosting‐based ensemble approach, forward stage‐wise additive modelling (FSAM), to improve some widely used base regressors’ prediction ability. We used 10 regression algorithms in four different types to make predictions on 10 diverse data from different scientific areas and we compared the experimental results in terms of correlation coefficient, mean absolute error, and root mean squared error metrics. Furthermore, we made use of scatter plots to demonstrate the effect of ensemble modelling on the prediction accuracies of evaluated algorithms. We empirically obtained that in general FSAM enhances the accuracies of base regressors or it at least maintains the base regressor performance.     

Wind power prediction using deep neural network based meta regression and transfer learning

An innovative short term wind power prediction system is proposed which exploits the learning ability of deep neural network based ensemble technique and the concept of transfer learning. In the proposed DNN-MRT scheme, deep auto-encoders act as base-regressors, whereas Deep Belief Network is used as a meta-regressor. Employing the concept of ensemble learning facilitates robust and collective decision on test data, whereas deep base and meta-regressors ultimately enhance the performance of the proposed DNN-MRT approach. The concept of transfer learning not only saves time required during training of a base-regressor on each individual wind farm dataset from scratch but also stipulates good weight initialization points for each of the wind farm for training. The effectiveness of the proposed, DNN-MRT technique is expressed by comparing statistical performance measures in terms of root mean squared error (RMSE), mean absolute error (MAE), and standard deviation error (SDE) with other existing techniques.     

Jointly Optimized Regressors for Image Super‐resolution

Learning regressors from low‐resolution patches to high‐resolution patches has shown promising results for image super‐resolution. We observe that some regressors are better at dealing with certain cases, and others with different cases. In this paper, we jointly learn a collection of regressors, which collectively yield the smallest super‐resolving error for all training data. After training, each training sample is associated with a label to indicate its ‘best’ regressor, the one yielding the smallest error. During testing, our method bases on the concept of ‘adaptive selection’ to select the most appropriate regressor for each input patch. We assume that similar patches can be super‐resolved by the same regressor and use a fast, approximate kNN approach to transfer the labels of training patches to test patches. The method is conceptually simple and computationally efficient, yet very effective. Experiments on four datasets show that our method outperforms competing methods.     

基于分类与回归混合模型的人脸年龄估计方法

针对现有人脸年龄数据库样本数量少、各年龄段分布不均匀的问题,提出了一种基于分类与回归混合模型的人脸年龄估计方法。该方法主要包含两个方面：特征学习和估计模式。在特征学习方面,利用已有的深度卷积神经网络（CNN）,先在粗糙年龄标注数据集上预训练,再在现有的精确年龄标注数据库上微调,分别得到一个年龄段判别模型和两个年龄估计模型;在估计模式方面,该方法采用由粗到细的策略：首先,将人脸分入青少年、中年、老年和两个重叠区域这五个年龄段;然后,对于青少年和老年采用分类模型估计,对于中年采用回归模型估计,对于重叠区域采用两个模型估计的均值。所提方法在测试集上的平均绝对误差（MAE）为2.56。实验结果表明该方法受不同肤色和性别的影响较小,有较低的误差。     

Development of a fully automated RULA assessment system based on computer vision

The purpose of this study was to develop an automated, RULA-based posture assessment system using a deep learning algorithm to estimate RULA scores, including scores for wrist posture, based on images of workplace postures. The proposed posture estimation system reported a mean absolute error (MAE) of 2.86 on the validation dataset obtained by randomly splitting 20% of the original training dataset before data augmentation. The results of the proposed system were compared with those of two experts’ manual evaluation by computing the intraclass correlation coefficient (ICC), which yielded index values greater than 0.75, thereby confirming good agreement between manual raters and the proposed system. This system will reduce the time required for postural evaluation while producing highly reliable RULA scores that are consistent with those generated by manual approach. Thus, we expect that this study will aid ergonomic experts in conducting RULA-based surveys of occupational postures in workplace conditions.     

基于降噪自编码器的社会化推荐算法

现有的社会化推荐算法未考虑信任用户对目标用户深层的偏好影响。针对这一问题,提出了一种基于深度学习的混合推荐算法,利用降噪自编码器学习用户及其信任用户的评分偏好,使用加权隐藏层来平衡这些表示的重要性,有效建模用户间的潜在偏好交互。在此基础上,通过用户聚类和个性化权重区分不同类的用户受其信任用户的影响程度。在开放数据集上的实验结果表明,该算法优于现有的社会化推荐算法,与主要的推荐算法SoRec、RSTE、SocialMF、TrustMF相比,其平均绝对误差(MAE)和均方根误差(RMSE)显著降低,获得了较好的推荐效果。     

An improved long short-term memory networks with Takagi-Sugeno fuzzy for traffic speed prediction considering abnormal traffic situation

Traffic speed prediction is an emerging paradigm for achieving a better transportation system in smart cities and improving the heavy traffic management in the intelligent transportation system (ITS). The accurate traffic speed prediction is affected by many contextual factors such as abnormal traffic conditions, traffic incidents, lane closures due to construction or events, and traffic congestion. To overcome these problems, we propose a new method named fuzzy optimized long short-term memory (FOLSTM) neural network for long-term traffic speed prediction. FOLSTM technique is a hybrid method composed of computational intelligence (CI), machine learning (ML), and metaheuristic techniques, capable of predicting the speed for macroscopic traffic key parameters. First, the proposed hybrid unsupervised learning method, agglomerated hierarchical K-means (AHK) clustering, divides the input samples into a group of clusters. Second, based on parameters the Gaussian bell-shaped fuzzy membership function calculates the degree of membership (high, low, and medium) for each cluster using Takagi-Sugeno fuzzy rules. Finally, the whale optimization algorithm (WOA) is used in LSTM to optimize the parameters obtained by fuzzy rules and calculate the optimal weight value. FOLSTM evaluates the accurate traffic speed from the abnormal traffic data to overcome the nonlinear characteristics. Experimental results demonstrated that our proposed method outperforms the state-of-the-art approaches in terms of metrics such as mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE).     

Machine learning aided high-throughput first-principles calculations to predict the formation enthalpy of σ phase

The σ phase is a topologically close-packed phase and can significantly influence the performance and properties of materials. Accurate prediction the formation enthalpy of the σ phase is crucial for the development of high-performance materials. First-principles calculations based on density functional theory (DFT) have been employed to study the formation enthalpy of the σ phase, but this approach requires a amount of computational resources and time. In this study, we propose a machine learning (ML) method to predict the formation enthalpy of the σ phase. This method employs a first-principles dataset containing 1342 configurations of the binary σ phases for model training and testing. Among the algorithms used, the Multi-Layer Perceptron algorithm demonstrated the highest predictive accuracy, with the mean absolute error (MAE) of 22.881 meV/atom, which is comparable to the existing ML prediction model based on first-principles calculations. The trained model was then utilized to predict the formation enthalpy of the 1177 untrained ternary configurations, achieving a significant reduction in computational time of over 59% compared to traditional first-principles calculations. Furthermore, the model was validated for lattice parameters prediction, achieving the MAE of 0.073 Å and 0.048 Å for the a and c, respectively. A Graphical User Interface (GUI) was developed. Finally, we predicted the formation enthalpy of all the possible ternary configurations, which is comparable to the MAE of DFT-calculations itself.     

Defect Prediction Using Akaike and Bayesian Information Criterion

Data available in software engineering for many applications contains variability and it is not possible to say which variable helps in the process of the prediction. Most of the work present in software defect prediction is focused on the selection of best prediction techniques. For this purpose, deep learning and ensemble models have shown promising results. In contrast, there are very few researches that deals with cleaning the training data and selection of best parameter values from the data. Sometimes data available for training the models have high variability and this variability may cause a decrease in model accuracy. To deal with this problem we used the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) for selection of the best variables to train the model. A simple ANN model with one input, one output and two hidden layers was used for the training instead of a very deep and complex model. AIC and BIC values are calculated and combination for minimum AIC and BIC values to be selected for the best model. At first, variables were narrowed down to a smaller number using correlation values. Then subsets for all the possible variable combinations were formed. In the end, an artificial neural network (ANN) model was trained for each subset and the best model was selected on the basis of the smallest AIC and BIC value. It was found that combination of only two variables’ ns and entropy are best for software defect prediction as it gives minimum AIC and BIC values. While, nm and npt is the worst combination and gives maximum AIC and BIC values.     

Towards Improving Predictive Statistical Learning Model Accuracy by Enhancing Learning Technique

The accuracy of the statistical learning model depends on the learning technique used which in turn depends on the dataset’s values. In most research studies, the existence of missing values (MVs) is a vital problem. In addition, any dataset with MVs cannot be used for further analysis or with any data driven tool especially when the percentage of MVs are high. In this paper, the authors propose a novel algorithm for dealing with MVs depending on the feature selection (FS) of similarity classifier with fuzzy entropy measure. The proposed algorithm imputes MVs in cumulative order. The candidate feature to be manipulated is selected using similarity classifier with Parkash’s fuzzy entropy measure. The predictive model to predict MVs within the candidate feature is the Bayesian Ridge Regression (BRR) technique. Furthermore, any imputed features will be incorporated within the BRR equation to impute the MVs in the next chosen incomplete feature. The proposed algorithm was compared against some practical state-of-the-art imputation methods by conducting an experiment on four medical datasets which were gathered from several databases repository with MVs generated from the three missingness mechanisms. The evaluation metrics of mean absolute error (MAE), root mean square error (RMSE) and coefficient of determination (R² score) were used to measure the performance. The results exhibited that performance vary depending on the size of the dataset, amount of MVs and the missingness mechanism type. Moreover, compared to other methods, the results showed that the proposed method gives better accuracy and less error in most cases.     

Improved universal quantitative steganalysis in spatial domain using ELM ensemble

Quantitative steganalysis seeks to extract the additional information about the hidden message in the covert communications. Most of the quantitative steganalyzers available in the literature target a specific embedding algorithm and generally extract the payload information using structural paradigm. Modern steganalyzers use supervised machine learning to estimate the stego payload using sophisticated feature sets. In this paper, an Ensemble Framework based universal quantitative steganalyzer for digital images is proposed which employs optimised Extreme Learning Machines as the base regressors. The framework exploits inherent diversity of the base regressor and the use of random subspaces of the image features further reduces the prediction error. The proposed ensemble regressor exhibits improved payload predictions when evaluated vis-à-vis the individual base regressor and other state-of-the-art algorithms. The experimental results across different embedding algorithms, image datasets and variedly sized feature sets demonstrate the robustness and wide applicability of the proposed framework.

     

Adaptive transfer learning in deep neural networks: Wind power prediction using knowledge transfer from region to region and between different task domains

Transfer learning (TL) in deep neural networks is gaining importance because, in most of the applications, the labeling of data is costly and time consuming. Additionally, TL also provides an effective weight initialization strategy for deep neural networks. This paper introduces the idea of adaptive TL in deep neural networks (ATL‐DNN) for wind power prediction. Specifically, we show in case of wind power prediction that adaptive TL of the deep neural networks system can be adaptively modified as regards training on a different wind farm is concerned. The proposed ATL‐DNN technique is tested for short‐term wind power prediction, where continuously arriving information has to be exploited. Adaptive TL not only helps in providing good weight initialization, but also in utilizing the incoming data for effective learning. Additionally, the proposed ATL‐DNN technique is shown to transfer knowledge between different task domains (wind power to wind speed prediction) and from one region to another region. The simulation results show that the proposed ATL‐DNN technique achieves average values of 0.0637, 0.0986, and 0.0984 for the mean absolute error, root mean squared error, and standard deviation error, respectively.     

Daily Urban Water Demand Forecasting Based on Chaotic Theory and Continuous Deep Belief Neural Network

The prediction of daily water demands is a crucial part of the effective functioning of the water supply system. This work proposed that a continuous deep belief neural network (CDBNN) model based on the chaotic theory should be implemented to predict the daily water demand time series in Zhuzhou, China. CDBNN should initially be used to predict the urban water demand time series. First, the power spectrum and the largest Lyapunov exponent is used to determine the chaotic characteristic of the daily water demand time series. Second, C–C method is utilized to reconstruct the water demand time series’ phase space. Lastly, the forecasting model should be produced with the continuous deep belief network and neural network algorithms implemented for feature learning and regression, respectively, and the CDBNN input established by the best embedding dimension of the reconstructed phase space. The proposed method is contrasted with the support vector regression, generalized regression neural networks and feed forward neural networks, and they are accepted with the identical dataset. The predictive performance of the models is examined using normalized root-mean-square error (NRMSE), correlation coefficient (COR), and mean absolute percentage error (MAPE). The results suggest that the hybrid model has the smallest NRMSE and MAPE values, and the largest COR.     

静态图像中采用混合卷积结构进行人群密度估计

提出了一种混合卷积神经网络用于人群数量的感知计算,在高度密集的场景中可以准确地预测人群密度图。模型仅由两个部分组成：前端为扩张卷积神经网络提取二维特征;后端采用分数步长卷积神经网络降低下采样中的信息损失。为了验证和分析算法性能,模型设计基于当前较为流行的Shanghai Tech数据集,使用回归问题的评价指标,即平均绝对误差（MAE）和均方误差（MSE）作为评估算法性能的标准。在Shanghai Tech（MAE=100.8）、UCF_CC_50（MAE=305.3）与WorldExpo’10数据集上进行测试,实验表明模型在密集场景下较以往的方法有效降低了MAE和MSE,提高了密集人群计数的准确率。     

A novel data preprocessing method to estimate the air pollution (SO2): neighbor-based feature scaling (NBFS)

The forecasting of air pollution is important for living environment and public health. The prediction of SO₂ (sulfur dioxide), which is one of the indicators of air pollution, is a significant part of steps to be done in order to decrease the air pollution. In this study, a novel feature scaling method called neighbor-based feature scaling (NBFS) has been proposed and combined with artificial neural network (ANN) and adaptive network–based fuzzy inference system (ANFIS) prediction algorithms in order to predict the SO₂ concentration value is from air quality metrics belonging to Konya province in Turkey. This work consists of two stages. In the first stage, SO₂ concentration dataset has been scaled using neighbor-based feature scaling. In the second stage, ANN and ANFIS prediction algorithms have been used to forecast the SO₂ value of scaled SO₂ concentration dataset. SO₂ concentration dataset was obtained from Air Quality Statistics database of Turkish Statistical Institute. To constitute dataset, the mean values belonging to seasons of winter period have been used with the aim of watching the air pollution changes between dates of December, 1, 2003 and December, 30, 2005. In order to evaluate the performance of the proposed method, the performance measures including mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and IA (Index of Agreement) values have been used. After NBFS method applied to SO₂ concentration dataset, the obtained RMSE and IA values are 83.87–0.27 (IA) and 93–0.33 (IA) using ANN and ANFIS, respectively. Without NBFS, the obtained RMSE and IA values are 85.31–0.25 (IA) and 117.71–0.29 (IA) using ANN and ANFIS, respectively. The obtained results have demonstrated that the proposed feature scaling method has been obtained very promising results in the prediction of SO₂ concentration values.     

Nonlinear Short-term Prediction of Aluminum Foil Thickness via Global Regressor Combination

In this study, short-term prediction of aluminum foil thickness time-series data recorded during cold-rolling process was investigated. The locally projective nonlinear noise reduction was applied in order to improve the predictability of the time series. The higher-order statistics methods (bispectrum and bicoherence) were used to detect the nonlinearity. The embedding vectors with appropriate embedding dimension and time delay were obtained via the false nearest neighbors and mutual information methods, respectively. The maximum prediction horizon was determined depending on the maximal Lyapunov exponent. For various prediction horizons, the embedding vector and corresponding thickness value pairs were used as the dataset to assess the prediction performance of various machine learning algorithms (i.e., multilayer perceptron neural network, support vector machines with Pearson VII function-based kernel, and radial basis function network). The n-step ahead prediction outputs of the machine learning algorithms were globally combined with simple voting in favor of the one having minimum absolute error. The accuracy of our proposed method was compared with nonlinear autoregressive exogenous model for various thickness time-series data using mean absolute percentage error measure.     

基于深度残差网络的人脸表情识别

针对深度卷积神经网络随着卷积层数增加而导致网络模型难以训练和性能退化等问题,提出了一种基于深度残差网络的人脸表情识别方法。该方法利用残差学习单元来改善深度卷积神经网络模型训练寻优的过程,减少模型收敛的时间开销。此外,为了提高网络模型的泛化能力,从KDEF和CK+两种表情数据集上选取表情图像样本组成混合数据集用以训练网络。在混合数据集上采用十折（10-fold）交叉验证方法进行了实验,比较了不同深度的带有残差学习单元的残差网络与不带残差学习单元的常规卷积神经网络的表情识别准确率。当采用74层的深度残差网络时,可以获得90.79%的平均识别准确率。实验结果表明采用残差学习单元构建的深度残差网络可以解决网络深度和模型收敛性之间的矛盾,并能提升表情识别的准确率。     

ORESP：基于有序回归的软件缺陷严重程度预测方法

为提高软件缺陷严重程度的预测性能,通过充分考虑软件缺陷严重程度标签间的次序性,提出一种基于有序回归的软件缺陷严重程度预测方法ORESP.该方法首先使用基于Spearman的特征选择方法来识别并移除数据集内的冗余特征,随后使用基于比例优势模型的神经网络来构建预测模型.通过与五种经典分类方法的比较,所提的ORESP方法在四种不同类型的度量下均可取得更高的预测性能,其中基于平均0-1误差(MZE)评测指标,预测模型性能最大可提升10.3％;基于平均绝对误差(MAE)评测指标,预测模型性能最大可提升12.3％.除此之外,发现使用基于Spearman的特征选择方法可以有效提升ORESP方法的预测性能.     

基于DWT-GCN的短时交通流预测

交通流预测是智慧交通领域的研究热点之一, 为了深层次地挖掘交通流序列的时空特征, 提高预测精度, 提出了一种基于离散小波变换(discrete wavelet transformation, DWT)和图卷积网络(graph convolutional network, GCN)短时交通流预测模型. 首先, 利用DWT算法将原始交通序列分解为细节分量与近似分量, 降低交通流数据的非平稳性; 其次, 引入距离因子项优化GCN模型中的邻接矩阵, 进一步提取路网的空间特征; 最后, 将DWT分解的各组分量数据分别作为GCN模型的输入进行预测, 并对各组预测结果进行重构, 得到最终预测值. 利用美国加利福尼亚州交通局PeMS数据库实测交通数据对模型进行测试, 结果表明, 该模型相比于ARIMA、WNN、GCN, 平均绝对误差平均降低57%, 平均绝对百分比误差平均降低59%, 是一种有效的短时交通流预测方法.     

A new technique to predict fly-rock in bench blasting based on an ensemble of support vector regression and GLMNET

Fly-rock caused by blasting is one of the dangerous side effects that need to be accurately predicted in open-pit mines. This study proposed a new technique to predict the distance of fly-rock based on an ensemble of support vector regression models (SVRs) and Lasso and elastic-net regularized generalized linear model (GLMNET), called SVRs–GLMNET. It was developed based on a combination of six SVR models and a GLMNET model. Accordingly, the dataset including 210 experimental data was divided into three parts, i.e., training, validating, and testing. Of the whole dataset, 70% was used for the development of the six SVR models first as the sub-models. Subsequently, 20% of the entire dataset (the validating dataset) was used to predict fly-rock based on the six developed SVR models. The predicted results from the six developed SVR models were used as the input variables to establish the GLMNET model (i.e., SVRs–GLMNET model). Finally, the remaining 10% of the dataset was used for testing the performance of the proposed SVRs–GLMNET model. A comparison and evaluation of the six developed SVR models and the proposed SVRs–GLMNET model were implemented based on five statistical criteria, such as mean absolute error (MAE), mean absolute percentage error (MAPE), root-mean-square error (RMSE), variance account for (VAF), and determination of correlation (R²). The results indicated that the proposed SVRs–GLMNET model provided the most dominant performance in predicting the distance of fly-rock caused by bench blasting in this study with an RMSE of 3.737, R² of 0.993, MAE of 3.214, MAPE of 0.018, and VAF of 99.207. Whereas, the other models yielded poorer accuracy with RMSE of 7.058–12.779, R² of 0.920–0.972, MAE of 3.438–7.848, MAPE of 0.021–0.055, and VAF of 90.538–97.003.