Comparing cost prediction models by resampling techniques

The accurate software cost prediction is a research topic that has attracted much of the interest of the software engineering community during the latest decades. A large part of the research efforts involves the development of statistical models based on historical data. Since there are a lot of models that can be fitted to certain data, a crucial issue is the selection of the most efficient prediction model. Most often this selection is based on comparisons of various accuracy measures that are functions of the model’s relative errors. However, the usual practice is to consider as the most accurate prediction model the one providing the best accuracy measure without testing if this superiority is in fact statistically significant. This policy can lead to unstable and erroneous conclusions since a small change in the data is able to turn over the best model selection. On the other hand, the accuracy measures used in practice are statistics with unknown probability distributions, making the testing of any hypothesis, by the traditional parametric methods, problematic. In this paper, the use of statistical simulation tools is proposed in order to test the significance of the difference between the accuracy of two prediction methods: regression and estimation by analogy. The statistical simulation procedures involve permutation tests and bootstrap techniques for the construction of confidence intervals for the difference of measures. Four known datasets are used for experimentation in order to validate the results and make comparisons between the simulation methods and the traditional parametric and non-parametric procedures.     

Fast robust estimation of prediction error based on resampling

Robust estimators of the prediction error of a linear model are proposed. The estimators are based on the resampling techniques cross-validation and bootstrap. The robustness of the prediction error estimators is obtained by robustly estimating the regression parameters of the linear model and by trimming the largest prediction errors. To avoid the recalculation of time-consuming robust regression estimates, fast approximations for the robust estimates of the resampled data are used. This leads to time-efficient and robust estimators of prediction error.     

Multi-step prediction error approach for controller performance monitoring

Performance monitoring of model predictive control (MPC) systems has received a great interest from both academia and industry. In recent years some novel approaches for multivariate control performance monitoring have been developed without the requirement of process models or interactor matrices. Among them the prediction error approach has been shown promising, but it is based on single-step prediction and may not be compatible with the MPC objective that is based on multi-step prediction. This paper develops a multi-step prediction error approach for performance monitoring of model predictive control systems, and demonstrates its application in a real industrial MPC performance monitoring and diagnosis problem.     

Restricted regression estimation in measurement error models

The problem of consistent estimation of the regression coefficients when some prior information about the regression coefficients is available is considered. Such prior information is expressed in the form of exact linear restrictions. The knowledge of covariance matrix of measurement errors that is associated with explanatory variables is used to construct the consistent estimators. Some consistent estimators are suggested which satisfy the exact linear restrictions also. Their asymptotic properties are derived and analytically analyzed under a multivariate ultrastructural model with not necessarily normally distributed measurement errors. The finite sample properties of the estimators are studied through a Monte-Carlo simulation experiment.     

Automatic covariate selection in logistic models for chest pain diagnosis: a new approach

A newly established method for optimizing logistic models via a minorization-majorization procedure is applied to the problem of diagnosing acute coronary syndromes (ACS). The method provides a principled approach to the selection of covariates which would otherwise require the use of a suboptimal method owing to the size of the covariate set. A strategy for building models is proposed and two models optimized for performance and for simplicity are derived via 10-fold cross-validation. These models confirm that a relatively small set of covariates including clinical and electrocardiographic features can be used successfully in this task. The performance of the models is comparable with previously published models using less principled selection methods. The models prove to be portable when tested on data gathered from three other sites. Whilst diagnostic accuracy and calibration diminishes slightly for these new settings, it remains satisfactory overall. The prospect of building predictive models that are as simple as possible for a required level of performance is valuable if data-driven decision aids are to gain wide acceptance in the clinical situation owing to the need to minimize the time taken to gather and enter data at the bedside.     

Local influence assessment in heteroscedastic measurement error models

Functional heteroscedastic measurement error models are investigated aiming to assess the effects of perturbations of data on some inferential procedures. This goal is accomplished by resorting to methods of local influence. The techniques provide to the practitioner a valuable tool that enables to identify potential influential elements and to quantify the effects of perturbations in these elements on results of interest. An illustrative example with a real data set is also reported.     

Improving the robustness of fractional polynomial models by preliminary covariate transformation: A pragmatic approach

Fractional polynomials have been found useful in univariate and multivariable non-linear regression analysis. As with many flexible regression models, they may be prone to distortion of the fitted function caused by values with high leverage at either extreme of the covariate distribution. Furthermore, fractional polynomial functions are not invariant to a change of origin of the covariate. A new approach, based on a preliminary, almost-linear transformation of a covariate, is proposed. The transformation is approximately linear within the bulk of the observations and tapers smoothly to a truncation of the extremes. It incorporates a predefined shift of the origin away from zero. Empirical studies show that this transformation is effective in reducing extreme leverages. In two real datasets, it is shown its use can result in more sensible final models.     

An error analysis approach for laminated composite plate finite element models

Errors in laminated composite plate finite element models occur at both the individual element level and at the discretization level. This paper shows that parasitic shear causes individual element errors and that its sources must be eliminated if numerically and physically correct results are to be provided by the finite element analysis. In addition, discretization errors occur when the behavior of the continuum is represented by a finite number of degrees of freedom. A procedure to estimate discretization errors in laminated composite plate finite element models and guide refinement, in order to achieve an acceptable level of accuracy, is developed. The error estimator built is based on the energy norm of the error in stress resultants.     

Prediction measurement with mean acceptable error for proper inconsistency in noisy weldability prediction data

Due to the complex nature of the welding process, the data used to construct prediction models often contain a significant amount of inconsistency. In general, this type of inconsistent data is treated as noise in the literature. However, for the weldability prediction, the inconsistency, which we describe as proper-inconsistency, may not be eliminated since the inconsistent data can help extract additional information about the process. This paper discusses that, in the presence of proper-inconsistency, it is inappropriate to perform the same approach generally employed with machine learning algorithms, in terms of the model construction and prediction measurement. Due to the numerical characteristics of proper-inconsistency, it is likely to achieve vague prediction results from the prediction model with the traditional prediction performance measures. In this paper, we propose a new prediction performance measure called mean acceptable error (MACE), which measures the performance of prediction models constructed with the presence of proper-inconsistency. This paper presents experimental results with real weldability prediction data, and we examine the prediction performance of k-nearest neighbor (kNN) and generalized regression neural network (GRNN) measured by MACE and the different characteristics of data in relation to MACE, kNN, and GRNN. The results indicate that using a smaller k on properly-inconsistent data increases the prediction performance measured by MACE. Also, the prediction performance on the correct data increases, while the effect of properly-inconsistent data decreases with the measurement of MACE.     

Algorithms for roundoff error analysis —A relative error approach

Methods are presented for performing various error analyses of numerical algorithms. These analyses include forward, backward, and B-analysis (a combination of forward and backward). These analyses additionally provide alternative criteria by which different algorithms that solve the same problem may be compared. The conclusions of various comparison criteria are related to the correlation of errors in each algorithm. Finally, the analysis of a composite algorithm, which is made up of concatenated sub-algorithms, is given in terms of analyses done on its parts.

---

Zusammenfassung In dieser Arbeit werden Methoden vorgestellt, die es gestatten, verschiedene Fehleranalysen numerischer Algorithmen zu vollziehen. Darunter befinden sich Vorwärts- und Rückwärtsanalyse (forward and backward analysis) sowie beidseitige Analyse (B-analysis, eine Kombination von forward and backward) ein. Diese Analysen liefern zusätzlich weitere Kriterien, durch welche verschiedene Algorithmen, die dasselbe Problem lösen, verglichen werden können. Die Aussagen der verschiedenen Vergleichskriterien beziehen sich auf die Fehlerkorelation in jedem Algorithmus. Schließlich wird die Analyse zusammengesetzter Algorithmen, welche aus verketteten Subalgorithmen bestehen, mit Hilfe der Analysen, die an den Teilen vollzogen wurden, dargestellt.

---

This work was supported in part by the National Science Foundation under NSF Grant MCS 75-21758.     

Uniqueness of prediction error estimates of multivariable moving average models

In prediction error (PE) identification of the parameter estimates is given by the global minimum of a scalar-valued function of the innovation sample covariance matrix. It may happen that the loss function has multiple local minimum points so that a numerical search routine can fail to find the global minimum. Such a situation, usually referred to as lack of uniqueness of the estimates, was experienced in practice and also theoretically examined for various model structures. A unique minimum of the criterion is also crucial for convergence of recursive PE algorithms. In this paper multivariable moving average (MA) models are considered. It is proved that for such models any reasonable PE criterion has asymptotically a unique stationary point. Furthermore it is shown that this stationary point is a (global) minimum which corresponds to the true parameter vector. This extends the result known for univariate MA models to the multivariate case.     

On the uniqueness of prediction error models for systems with noisy input-output data

This paper addresses the uniqueness problem of the prediction error (PE) identification for a class of linear systems with noisy input and output data. Necessary and sufficient conditions are derived for the corresponding PE loss function to have (asymptotically) a unique global minimum. The results indicate that a PE algorithm may give very bad parameter estimates for systems not satisfying these conditions. Such a possibility is illustrated by a numerical example. While the PE method is used as a vehicle for illustration, the derived conditions for global uniqueness (or identifiability) apply to any consistent estimation method based on second-order data.     

Rank tests and regression rank score tests in measurement error models

The rank and regression rank score tests of linear hypothesis in the linear regression model are modified for measurement error models. The modified tests are still distribution free. Some tests of linear subhypotheses are invariant to the nuisance parameter, others are based on the aligned ranks using the R-estimators. The asymptotic relative efficiencies of tests with respect to tests in models without measurement errors are evaluated. The simulation study illustrates the powers of the tests.     

The application of ANFIS prediction models for thermal error compensation on CNC machine tools

Thermal errors can have significant effects on CNC machine tool accuracy. The errors come from thermal deformations of the machine elements caused by heat sources within the machine structure or from ambient temperature change. The effect of temperature can be reduced by error avoidance or numerical compensation. The performance of a thermal error compensation system essentially depends upon the accuracy and robustness of the thermal error model and its input measurements. This paper first reviews different methods of designing thermal error models, before concentrating on employing an adaptive neuro fuzzy inference system (ANFIS) to design two thermal prediction models: ANFIS by dividing the data space into rectangular sub-spaces (ANFIS-Grid model) and ANFIS by using the fuzzy c-means clustering method (ANFIS-FCM model). Grey system theory is used to obtain the influence ranking of all possible temperature sensors on the thermal response of the machine structure. All the influence weightings of the thermal sensors are clustered into groups using the fuzzy c-means (FCM) clustering method, the groups then being further reduced by correlation analysis.A study of a small CNC milling machine is used to provide training data for the proposed models and then to provide independent testing data sets. The results of the study show that the ANFIS-FCM model is superior in terms of the accuracy of its predictive ability with the benefit of fewer rules. The residual value of the proposed model is smaller than ±4 μm. This combined methodology can provide improved accuracy and robustness of a thermal error compensation system.     

A self-adaptive system with auxiliary adjustment prediction

This paper presents a method for the improvement of a general class of two-adjustment, self-adaptive systems. The technique is based upon the use of existing, external signals. By the incorporation of an auxiliary adjustment prediction matrix, a hybrid scheduling and plant adaptive system is developed. This approach affords a relatively simple method for the possible improvement of a commonly discussed class of self-adaptive systems. It is shown that the over-all system's figure-of-merit can be improved with such a scheme.     

A fully Bayesian approach to inference for Coxian phase-type distributions with covariate dependent mean

Phase-type distributions represent the time to absorption for a finite state Markov chain in continuous time, generalising the exponential distribution and providing a flexible and useful modelling tool. We present a new reversible jump Markov chain Monte Carlo scheme for performing a fully Bayesian analysis of the popular Coxian subclass of phase-type models; the convenient Coxian representation involves fewer parameters than a more general phase-type model. The key novelty of our approach is that we model covariate dependence in the mean whilst using the Coxian phase-type model as a very general residual distribution. Such incorporation of covariates into the model has not previously been attempted in the Bayesian literature. A further novelty is that we also propose a reversible jump scheme for investigating structural changes to the model brought about by the introduction of Erlang phases. Our approach addresses more questions of inference than previous Bayesian treatments of this model and is automatic in nature. We analyse an example dataset comprising lengths of hospital stays of a sample of patients collected from two Australian hospitals to produce a model for a patient’s expected length of stay which incorporates the effects of several covariates. This leads to interesting conclusions about what contributes to length of hospital stay with implications for hospital planning. We compare our results with an alternative classical analysis of these data.     

A robust stochastic stability analysis approach for power system considering wind speed prediction error based on Markov model

This paper proposes a robust stochastic stability analysis approach with partly unknown transition probability by considering the wind speed prediction error in power system. Firstly, taking this prediction error into account, based on Markov modeling theory, the stochastic dynamic model of wind power system with uncertain transition probability is developed. Secondly, according to the stochastic stability theory of Markov jump system, the transition probability of wind power system mode is divided into three cases: fully known, only known upper and lower bounds, and completely unknown. Then, by using linear matrix inequality (LMI) technology, a robust stochastic stability criterion with disturbance attenuation is obtained. Finally, test results show that the proposed analysis approach does not need to obtain the trajectory of the actual system operation parameters, and has the advantages of high computational efficiency.     

Spindle thermal error prediction approach based on thermal infrared images: A deep learning method

It is essential to precisely model the spindle thermal error due to its dramatic influence on the machining accuracy. In this paper, the deep learning convolutional neural network (CNN) is used to model the axial and radial thermal errors of horizontal and vertical spindles. Unlike the traditional CNN model that relies entirely on thermal images, this model combines the thermal image with the thermocouple data to fully reflect the temperature field of the spindle. After pre-processing and data enhancement of the thermal images, a multi-classification model based on CNN is built and verified for accuracy and robustness. The experimental results show that the model prediction accuracy is approximately 90 %–93 %, which is higher than the BP model. When the spindle rotation speed changes, the model also shows good robustness. Real cutting tests show that the deep learning model has good applicability to the spindle thermal error prediction and compensation.     

A boosting approach for corporate failure prediction

Predicting corporate failure is an important management science problem. This is a typical classification question where the objective is to determine which indicators are involved in the failure/success of a corporation. Despite the importance of this problem, until now only classical machine learning tools have been considered to tackle this classification task. The objective of this paper is twofold. On the one hand, we introduce novel discerning measures to rank independent variables in a generic classification task. On the other hand, we apply boosting techniques to improve the accuracy of a classification tree. We apply this methodology to a set of European firms, considering the usual predicting variables such as financial ratios, as well as including novel variables rarely used before in corporate failure prediction, such as firm size, activity and legal structure. We show that our approach decreases the generalization error about thirty percent with respect to the error produced with a classification tree. In addition, the most important ratios deal with profitability and indebtedness, as is usual in failure prediction studies. E. A. Cortés · M. G. Martínez · N. G. Rubio. The authors teach Statistics at the Faculty of Economic and Business Sciences in the University of Castilla-La Mancha. Esteban Alfaro completed his degree in Business in 1999 and got his Ph.D. in Economics in 2005, both in the University of Castilla-La Mancha. His thesis dealt with the application of ensemble classifiers to corporate failure prediction. Matías Gámez got his degree in Mathematics at the University of Granada in 1991 and finished a Master in Applied Statistics a year after. He completed his Ph.D. in Economics at the University of Castilla-La Mancha in 1998 on the application of geo-statistical techniques to the estimation of housing prices. Noelia García got her degree in Economics at the University of Madrid (UAM) in 1996 and completed her Ph.D. in Economics in 2004 on the construction of an intelligent and automated system for property valuation through the combination of neural nets and a geographic information system (GIS). Current research deals with spatial statistics and the combination of classifiers (decision trees and neural nets) for solving heated topics in the Economics.     

一种简单有效的误差预报方法

本文提出一种简单有效的误差预报方法,该方法与传统的近似计算方法有很大不同,它是以误差分析为切入点,再由此确立各误差之间的关系,建立数学模型,用误差来预报误差,从而达到减小误差范围,提高计算精度的目的。