Machine Learning Techniques for Software Maintainability Prediction: Accuracy Analysis

Maintaining software once implemented on the end-user side is laborious and, over its lifetime, is most often considerably more expensive than the initial software development. The prediction of software maintainability has emerged as an important research topic to address industry expectations for reducing costs, in particular, maintenance costs. Researchers and practitioners have been working on proposing and identifying a variety of techniques ranging from statistical to machine learning (ML) for better prediction of software maintainability. This review has been carried out to analyze the empirical evidence on the accuracy of software product maintainability prediction (SPMP) using ML techniques. This paper analyzes and discusses the findings of 77 selected studies published from 2000 to 2018 according to the following criteria: maintainability prediction techniques, validation methods, accuracy criteria, overall accuracy of ML techniques, and the techniques offering the best performance. The review process followed the well-known systematic review process. The results show that ML techniques are frequently used in predicting maintainability. In particular, artificial neural network (ANN), support vector machine/regression (SVM/R), regression &; decision trees (DT), and fuzzy &; neuro fuzzy (FNF) techniques are more accurate in terms of PRED and MMRE. The N-fold and leave-one-out cross-validation methods, and the MMRE and PRED accuracy criteria are frequently used in empirical studies. In general, ML techniques outperformed non-machine learning techniques, e.g., regression analysis (RA) techniques, while FNF outperformed SVM/R, DT, and ANN in most experiments. However, while many techniques were reported superior, no specific one can be identified as the best.

     

Modelling of diesel engine performance using advanced machine learning methods under scarce and exponential data set

Traditional methods on creating diesel engine models include the analytical methods like multi-zone models and the intelligent based models like artificial neural network (ANN) based models. However, those analytical models require excessive assumptions while those ANN models have many drawbacks such as the tendency to overfitting and the difficulties to determine the optimal network structure. In this paper, several emerging advanced machine learning techniques, including least squares support vector machine (LS-SVM), relevance vector machine (RVM), basic extreme learning machine (ELM) and kernel based ELM, are newly applied to the modelling of diesel engine performance. Experiments were carried out to collect sample data for model training and verification. Limited by the experiment conditions, only 24 sample data sets were acquired, resulting in data scarcity. Six-fold cross-validation is therefore adopted to address this issue. Some of the sample data are also found to suffer from the problem of data exponentiality, where the engine performance output grows up exponentially along the engine speed and engine torque. This seriously deteriorates the prediction accuracy. Thus, logarithmic transformation of dependent variables is utilized to pre-process the data. Besides, a hybrid of leave-one-out cross-validation and Bayesian inference is, for the first time, proposed for the selection of hyperparameters of kernel based ELM. A comparison among the advanced machine learning techniques, along with two traditional types of ANN models, namely back propagation neural network (BPNN) and radial basis function neural network (RBFNN), is conducted. The model evaluation is made based on the time complexity, space complexity, and prediction accuracy. The evaluation results show that kernel based ELM with the logarithmic transformation and hybrid inference is far better than basic ELM, LS-SVM, RVM, BPNN and RBFNN, in terms of prediction accuracy and training time.     

Sentiment analysis of tweets using a unified convolutional neural network‐long short‐term memory network model

Sentiment analysis focuses on identifying and classifying the sentiments expressed in text messages and reviews. Social networks like Twitter, Facebook, and Instagram generate heaps of data filled with sentiments, and the analysis of such data is very fruitful when trying to improve the quality of both products and services alike. Classic machine learning techniques have a limited capability to efficiently analyze such large amounts of data and produce precise results; they are thus supported by deep learning models to achieve higher accuracy. This study proposes a combination of convolutional neural network and long short‐term memory (CNN‐LSTM) deep network for performing sentiment analysis on Twitter datasets. The performance of the proposed model is analyzed with machine learning classifiers, including the support vector classifier, random forest (RF), stochastic gradient descent (SGD), logistic regression, a voting classifier (VC) of RF and SGD, and state‐of‐the‐art classifier models. Furthermore, two feature extraction methods (term frequency‐inverse document frequency and word2vec) are also investigated to determine their impact on prediction accuracy. Three datasets (US airline sentiments, women's e‐commerce clothing reviews, and hate speech) are utilized to evaluate the performance of the proposed model. Experiment results demonstrate that the CNN‐LSTM achieves higher accuracy than those of other classifiers.     

Learning to Rank with Extreme Learning Machine

Relevance ranking has been a popular and interesting topic over the years, which has a large variety of applications. A number of machine learning techniques were successfully applied as the learning algorithms for relevance ranking, including neural network, regularized least square, support vector machine and so on. From machine learning point of view, extreme learning machine actually provides a unified framework where the aforementioned algorithms can be considered as special cases. In this paper, pointwise ELM and pairwise ELM are proposed to learn relevance ranking problems for the first time. In particular, ELM type of linear random node is newly proposed together with kernel version of ELM to be linear as well. The famous publicly available dataset collection LETOR is tested to compare ELM-based ranking algorithms with state-of-art linear ranking algorithms.     

Learning to rank by using multivariate adaptive regression splines and conic multivariate adaptive regression splines

Learning to rank is a supervised learning problem that aims to construct a ranking model for the given data. The most common application of learning to rank is to rank a set of documents against a query. In this work, we focus on point‐wise learning to rank, where the model learns the ranking values. Multivariate adaptive regression splines (MARS) and conic multivariate adaptive regression splines (CMARS) are supervised learning techniques that have been proven to provide successful results on various prediction problems. In this article, we investigate the effectiveness of MARS and CMARS for point‐wise learning to rank problem. The prediction performance is analyzed in comparison to three well‐known supervised learning methods, artificial neural network (ANN), support vector machine, and random forest for two datasets under a variety of metrics including accuracy, stability, and robustness. The experimental results show that MARS and ANN are effective methods for learning to rank problem and provide promising results.     

Credit score classification using spiking extreme learning machine

Credit score classification is a prominent research problem in the banking or financial industry, and its predictive performance is responsible for the profitability of financial industry. This paper addresses how Spiking Extreme Learning Machine (SELM) can be effectively used for credit score classification. A novel spike-generating function is proposed in Leaky Nonlinear Integrate and Fire Model (LNIF). Its interspike period is computed and utilized in the extreme learning machine (ELM) for credit score classification. The proposed model is named as SELM and is validated on five real-world credit scoring datasets namely: Australian, German-categorical, German-numerical, Japanese, and Bankruptcy. Further, results obtained by SELM are compared with back propagation, probabilistic neural network, ELM, voting-based Q-generalized extreme learning machine, Radial basis neural network and ELM with some existing spiking neuron models in terms of classification accuracy, Area under curve (AUC), H-measure and computational time. From the experimental results, it has been noticed that improvement in accuracy and execution time for the proposed SELM is highly statistically important for all aforementioned credit scoring datasets. Thus, integrating a biological spiking function with ELM makes it more efficient for categorization.     

一种基于超限学习机的电子磁罗盘非线性误差补偿方法

针对磁罗盘传感器非线性校正中现有方法的不足,提出采用小波函数和双曲正弦函数作为超限学习机(ELM)的激活函数,并将此改进超限学习机用于磁罗盘的校正.同时,阐述了传感器的非线性校正原理,磁罗盘航向误差模型及改进超限学习机的实现过程,并分别采用BP神经网络法和传统ELM对磁罗盘进行非线性校正.实验结果表明,改进ELM算法补偿后最大误差为0.103°,均方根误差为0.0596°,优于BP神经网络算法(补偿后最大误差为0.5°,均方根误差为0.1805°)和传统ELM神经网络(补偿后最大误差为0.21°,均方根误差为0.1056°).     

A study on effectiveness of extreme learning machine

Extreme learning machine (ELM), proposed by Huang et al., has been shown a promising learning algorithm for single-hidden layer feedforward neural networks (SLFNs). Nevertheless, because of the random choice of input weights and biases, the ELM algorithm sometimes makes the hidden layer output matrix H of SLFN not full column rank, which lowers the effectiveness of ELM. This paper discusses the effectiveness of ELM and proposes an improved algorithm called EELM that makes a proper selection of the input weights and bias before calculating the output weights, which ensures the full column rank of H in theory. This improves to some extend the learning rate (testing accuracy, prediction accuracy, learning time) and the robustness property of the networks. The experimental results based on both the benchmark function approximation and real-world problems including classification and regression applications show the good performances of EELM.     

Extreme‐learning‐machine‐based land cover classification

The extreme learning machine (ELM), a single hidden layer neural network based supervised classifier is used for remote sensing classifications. In comparison to the backpropagation neural network, which requires the setting of several user‐defined parameters and may produce local minima, the ELM requires setting of one parameter, and produces a unique solution for a set of randomly assigned weights. Two datasets, one multispectral and another hyperspectral, were used for classification. Accuracies of 89.0% and 91.1% are achieved with this classifier using multispectral and hyperspectral data, respectively. Results suggest that the ELM provides a classification accuracy comparable to a backpropagation neural network with both datasets. The computational cost using the ELM classifier (1.25 s with Enhanced Thematic Mapper (ETM+) and 0.675 s with Digital Airborne Imaging Spectrometer (DAIS) data) is very small in comparison to the backpropagation neural network.     

Self-adaptive extreme learning machine

In order to overcome the disadvantage of the traditional algorithm for SLFN (single-hidden layer feedforward neural network), an improved algorithm for SLFN, called extreme learning machine (ELM), is proposed by Huang et al. However, ELM is sensitive to the neuron number in hidden layer and its selection is a difficult-to-solve problem. In this paper, a self-adaptive mechanism is introduced into the ELM. Herein, a new variant of ELM, called self-adaptive extreme learning machine (SaELM), is proposed. SaELM is a self-adaptive learning algorithm that can always select the best neuron number in hidden layer to form the neural networks. There is no need to adjust any parameters in the training process. In order to prove the performance of the SaELM, it is used to solve the Italian wine and iris classification problems. Through the comparisons between SaELM and the traditional back propagation, basic ELM and general regression neural network, the results have proven that SaELM has a faster learning speed and better generalization performance when solving the classification problem.     

Multistrategy learning approaches to generate and tune fuzzy control structures and their application in manufacturing

Intelligence is strongly connected with learning adapting abilities, therefore such capabilities are considered as indispensable features of intelligent manufacturing systems (IMSs). A number of approaches have been described to apply different machine learning (ML) techniques for manufacturing problems, starting with rule induction in symbolic domains and pattern recognition techniques in numerical, subsymbolic domains. In recent years, artificial neural network (ANN) based learning is the dominant ML technique in manufacturing. However, mainly because of the black box nature of ANNs, these solutions have limited industrial acceptance. In the paper, the integration of neural and fuzzy techniques is treated and former solutions are analysed. A genetic algorithm (GA) based approach is introduced to overcome problems that are experienced during manufacturing applications with other algorithms.     

Multi layer ELM-RBF for multi-label learning

Many neural network methods such as ML-RBF and BP-MLL have been used for multi-label classification. Recently, extreme learning machine (ELM) is used as the basic elements to handle multi-label classification problem because of its fast training time. Extreme learning machine based auto encoder (ELM-AE) is a novel method of neural network which can reproduce the input signal as well as auto encoder, but it can not solve the over-fitting problem in neural networks elegantly. Introducing weight uncertainty into ELM-AE, we can treat the input weights as random variables following Gaussian distribution and propose weight uncertainty ELM-AE (WuELM-AE). In this paper, a neural network named multi layer ELM-RBF for multi-label learning (ML-ELM-RBF) is proposed. It is derived from radial basis function for multi-label learning (ML-RBF) and WuELM-AE. ML-ELM-RBF firstly stacks WuELM-AE to create a deep network, and then it conducts clustering analysis on samples features of each possible class to compose the last hidden layer. ML-ELM-RBF has achieved satisfactory results on single-label and multi-label data sets. Experimental results show that WuELM-AE and ML-ELM-RBF are effective learning algorithms.     

基于可拓聚类的极限学习机神经网络

针对极限学习机(ELM)算法随机选择输入层权值的问题,借鉴第2类型可拓神经网络(ENN-2)聚类的思想,提出了一种基于可拓聚类的ELM(EC-ELM)神经网络。该神经网络是以隐含层神经元的径向基中心向量作为输入层权值,采用可拓聚类算法动态调整隐含层节点数目和径向基中心,并根据所确定的输入层权值,利用Moore-Penrose广义逆快速完成输出层权值的求解。同时,对标准的Friedman#1回归数据集和Wine分类数据集进行测试,结果表明,EC-ELM提供了一种简便的神经网络结构和参数学习方法,并且比基于可拓理论的径向基函数(ERBF)、ELM神经网络具有更高的建模精度和更快的学习速度,为复杂过程的建模提供了新思路。     

基于支持向量机的蒙脱土悬浮液电动电位预测

应用多元线性回归、人工神经网络、支持向量机3种方法,对加入聚乙二醇、十二烷基苯磺酸钠、石油磺酸盐和部分水解聚丙烯酰胺四种处理剂的蒙脱土悬浮液的电动电位进行预测。在模型训练中,分别采用了神经网络集成和非启发式参数优化来提高人工神经网络和支持向量机模型的泛化能力。检验结果表明,参数优化的支持向量机模型预测精度最高,其平均误差率为3.88%,最大误差率为7.55%。     

Comparison of modelling ANN and ELM to estimate solar radiation over Turkey using NOAA satellite data

In this study, solar radiation (SR) is estimated at 61 locations with varying climatic conditions using the artificial neural network (ANN) and extreme learning machine (ELM). While the ANN and ELM methods are trained with data for the years 2002 and 2003, the accuracy of these methods was tested with data for 2004. The values for month, altitude, latitude, longitude, and land-surface temperature (LST) obtained from the data of the National Oceanic and Atmospheric Administration Advanced Very High Resolution Radiometer (NOAA-AVHRR) satellite are chosen as input in developing the ANN and ELM models. SR is found to be the output in modelling of the methods. Results are then compared with meteorological values by statistical methods. Using ANN, the determination coefficient (R²), mean bias error (MBE), root mean square error (RMSE), and Willmott’s index (WI) values were calculated as 0.943, ?0.148 MJ m^?2, 1.604 MJ m^?2, and 0.996, respectively. While R² was 0.961, MBE, RMSE, and WI were found to be in the order 0.045 MJ m^?2, 0.672 MJ m^?2, and 0.997 by ELM. As can be understood from the statistics, ELM is clearly more successful than ANN in SR estimation.     

Learning from demonstration in robots: Experimental comparison of neural architectures

Robots have played an important role in the automation of computer aided manufacturing. The classical robot control implementation involves an expensive key step of model-based programming. An intuitive way to reduce this expensive exercise is to replace programming with machine learning of robot actions from demonstration where a (learner) robot learns an action by observing a demonstrator robot performing the same. To achieve this learning from demonstration (LFD) different machine learning techniques such as Artificial Neural Networks (ANN), Genetic Algorithms, Hidden Markov Models, Support Vector Machines, etc. can be used. This piece of work focuses exclusively on ANNs. Since ANNs have many standard architectural variations divided into two basic computational categories namely the recurrent networks and feed-forward networks, representative networks from each have been selected for study, i.e. Feed Forward Multilayer Perceptron (FF) network for feed-forward networks category and Elman (EL), and Nonlinear Autoregressive Exogenous Model (NARX) networks for the recurrent networks category. The main objective of this work is to identify the most suitable neural architecture for application of LFD in learning different robot actions. The sensor and actuator streams of demonstrated action are used as training data for ANN learning. Consequently, the learning capability is measured by comparing the error between demonstrator and corresponding learner streams. To achieve fairness in comparison three steps have been taken. First, Dynamic Time Warping is used to measure the error between demonstrator and learner streams, which gives resilience against translation in time. Second, comparison statistics are drawn between the best, instead of weight-equal, configurations of competing architectures so that learning capability of any architecture is not forced handicap. Third, each configuration's error is calculated as the average of ten trials of all possible learning sequences with random weight initialization so that the error value is independent of a particular sequence of learning or a particular set of initial weights. Six experiments are conducted to get a performance pattern of each architecture. In each experiment, a total of nine different robot actions were tested. Error statistics thus obtained have shown that NARX architecture is most suitable for this learning problem whereas Elman architecture has shown the worst suitability. Interestingly the computationally lesser MLP gives much lower and slightly higher error statistics compared to the computationally superior Elman and NARX neural architectures, respectively.     

Direct adaptive neural control of nonlinear systems with extreme learning machine

A direct adaptive neural control scheme for a class of nonlinear systems is presented in the paper. The proposed control scheme incorporates a neural controller and a sliding mode controller. The neural controller is constructed based on the approximation capability of the single-hidden layer feedforward network (SLFN). The sliding mode controller is built to compensate for the modeling error of SLFN and system uncertainties. In the designed neural controller, its hidden node parameters are modified using the recently proposed neural algorithm named extreme learning machine (ELM), where they are assigned random values. However, different from the original ELM algorithm, the output weight is updated based on the Lyapunov synthesis approach to guarantee the stability of the overall control system. The proposed adaptive neural controller is finally applied to control the inverted pendulum system with two different reference trajectories. The simulation results demonstrate good tracking performance of the proposed control scheme.     

Non-tuned machine learning approach for hydrological time series forecasting

Stream-flow forecasting is a crucial task for hydrological science. Throughout the literature, traditional and artificial intelligence models have been applied to this task. An attempt to explore and develop better expert models is an ongoing endeavor for this hydrological application. In addition, the accuracy of modeling, confidence and practicality of the model are the other significant problems that need to be considered. Accordingly, this study investigates modern non-tuned machine learning data-driven approach, namely extreme learning machine (ELM). This data-driven approach is containing single layer feedforward neural network that selects the input variables randomly and determine the output weights systematically. To demonstrate the reliability and the effectiveness, one-step-ahead stream-flow forecasting based on three time-scale pattern (daily, mean weekly and mean monthly) for Johor river, Malaysia, were implemented. Artificial neural network (ANN) model is used for comparison and evaluation. The results indicated ELM approach superior the ANN model level accuracies and time consuming in addition to precision forecasting in tropical zone. In measureable terms, the dominance of ELM model over ANN model was indicated in accordance with coefficient determination (R ²) root-mean-square error (RMSE) and mean absolute error (MAE). The results were obtained for example the daily time scale R ² = 0.94 and 0.90, RMSE = 2.78 and 11.63, and MAE = 0.10 and 0.43, for ELM and ANN models respectively.

     

MR-ELM: a MapReduce-based framework for large-scale ELM training in big data era

In the big data era, extreme learning machine (ELM) can be a good solution for the learning of large sample data as it has high generalization performance and fast training speed. However, the emerging big and distributed data blocks may still challenge the method as they may cause large-scale training which is hard to be finished by a common commodity machine in a limited time. In this paper, we propose a MapReduce-based distributed framework named MR-ELM to enable large-scale ELM training. Under the framework, ELM submodels are trained parallelly with the distributed data blocks on the cluster and then combined as a complete single-hidden layer feedforward neural network. Both classification and regression capabilities of MR-ELM have been theoretically proven, and its generalization performance is shown to be as high as that of the original ELM and some common ELM ensemble methods through many typical benchmarks. Compared with the original ELM and the other parallel ELM algorithms, MR-ELM is a general and scalable ELM training framework for both classification and regression and is suitable for big data learning under the cloud environment where the data are usually distributed instead of being located in one machine.     

Evolutionary extreme learning machine

Extreme learning machine (ELM) [G.-B. Huang, Q.-Y. Zhu, C.-K. Siew, Extreme learning machine: a new learning scheme of feedforward neural networks, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN2004), Budapest, Hungary, 25-29 July 2004], a novel learning algorithm much faster than the traditional gradient-based learning algorithms, was proposed recently for single-hidden-layer feedforward neural networks (SLFNs). However, ELM may need higher number of hidden neurons due to the random determination of the input weights and hidden biases. In this paper, a hybrid learning algorithm is proposed which uses the differential evolutionary algorithm to select the input weights and Moore-Penrose (MP) generalized inverse to analytically determine the output weights. Experimental results show that this approach is able to achieve good generalization performance with much more compact networks.