Response modeling methodology (RMM)—maximum likelihood estimation procedures

Response modeling methodology (RMM) is a new approach for empirical modeling. ML estimation procedures for the RMM model are developed. For relational modeling, the RMM model is estimated in two phases. In the first phase, the structure of the linear predictor (LP) is determined and its parameters estimated. This is accomplished by combining canonical correlation analysis with linear regression analysis. The former procedure is used to estimate coefficients in a Taylor series approximation to an unspecified response transformation. Canonical scores are then used in the latter procedure as response values in order to estimate coefficients of the LP. In the second phase, the parameters of the RMM model are estimated via ML, given the LP estimated earlier. For modeling random variation, it is assumed that the LP is constant and a new simple percentile-based estimating procedure is developed. The new estimation procedures are demonstrated for some published data.     

非线性时间序列建模的混合自回归滑动平均模型

提出了一类用于非线性时间序列建模的混合自回归滑动平均模型(MARMA).该模型是由K个平稳或非平稳的ARMA分量经过混合得到的.讨论了MARMA模型的平稳性条件和自相关函数.给出了MARMA模型参数估计的期望极大化(expectation maximization)算法.运用贝叶斯信息准则(Bayes information criterion)来选择该模型.MARMA模型分布形式富于变化的特征使得它能够对具有多峰分布以及条件异方差的序列进行建模.通过两个实例验证了该模型,并和其他模型进行比较,结果表明MARMA模型能够更好地描述这些数据的特征.     

Learning and approximate inference in dynamic hierarchical models

A new variant of the dynamic hierarchical model (DHM) that describes a large number of parallel time series is presented. The separate series, which may be interdependent, are modeled through dynamic linear models (DLMs). This interdependence is included in the model through the definition of a ‘top-level’ or ‘average’ DLM. The model features explicit dependences between the latent states of the parallel DLMs and the states of the average model, and thus the many parallel time series are linked to each other. The combination of dependences within each time series and dependences between the different DLMs makes the computation time that is required for exact inference cubic in the number of parallel time series, however, which is unacceptable for practical tasks that involve large numbers of parallel time series. Therefore, two methods for fast, approximate inference are proposed: a variational approximation and a factorial approach. Under these approximations, inference can be performed in linear time, and it still features exact means. Learning is implemented through a maximum likelihood (ML) estimation of the model parameters. This estimation is realized through an expectation maximization (EM) algorithm with approximate inference in the E-step. Examples of learning and forecasting on two data sets show that the addition of direct dependences has a ‘smoothing’ effect on the evolution of the states of the individual time series, and leads to better prediction results. The use of approximate instead of exact inference is further shown not to lead to inferior results on either data set.     

基于GAMP-MAP估计器的非对齐空中计算

现有的空中计算依赖融合中心的最大似然(maximum likelihood, ML)估计恢复来自不同设备传输信号的算术和,但在实践中,实现准确的信道增益预编码和设备之间的严格同步很困难,ML估计在非对齐空中计算场景中会遭受严重的错误传播,而且计算复杂度高。针对非对齐空中计算问题,设计了一个基于广义近似消息传递(generalized approximate message passing, GAMP)计算框架的最大后验估计器,即GAMP-MAP。该估计器使用传输符号数据的均值和方差作为先验以提高估计准确性,利用GAMP算法标量估计特性降低计算复杂度。在不同信道相位偏移和时间偏移下进行仿真实验,实验结果表明GAMP-MAP估计器的准确性明显优于ML估计器,其运行时间相对ML估计器和LMMSE估计器有显著下降。     

A new maximum-likelihood phase estimation method for X-ray pulsar signals

X-ray pulsar navigation （XPNAV） is an attractive method for autonomous navigation of deep space in the future. Currently, techniques for estimating the phase of X-ray pulsar radiation involve the maximization of the general non-convex object functions based on the average profile fxom the epoch folding method. This results in the suppression of useful information and highly complex computation. In this paper, a new maximum likelihood （ML） phase estimation method that directly utilizes the measured time of arrivals （TOAs） is presented. The X-ray pulsar radiation will be treated as a cyclo-stationary process and the TOAs of the photons in a period will be redefined as a new process, whose probability distribution function is the normalized standard profile of the pulsar. We demonstrate that the new process is equivalent to the generally used Poisson model. Then, the phase estimation problem is recast as a cyclic shift parameter estimation under the ML estimation, and we also put forward a parallel ML estimation method to improve the ML solution. Numerical simulation results show that the estimator described here presents a higher precision and reduces the computational complexity compared with currently used estimators.     

Optimal Machine Learning Enabled Performance Monitoring for Learning Management Systems

Learning Management System (LMS) is an application software that is used in automation, delivery, administration, tracking, and reporting of courses and programs in educational sector. The LMS which exploits machine learning (ML) has the ability of accessing user data and exploit it for improving the learning experience. The recently developed artificial intelligence (AI) and ML models helps to accomplish effective performance monitoring for LMS. Among the different processes involved in ML based LMS, feature selection and classification processes find beneficial. In this motivation, this study introduces Glowworm-based Feature Selection with Machine Learning Enabled Performance Monitoring (GSO-MFWELM) technique for LMS. The key objective of the proposed GSO-MFWELM technique is to effectually monitor the performance in LMS. The proposed GSO-MFWELM technique involves GSO-based feature selection technique to select the optimal features. Besides, Weighted Extreme Learning Machine (WELM) model is applied for classification process whereas the parameters involved in WELM model are optimally fine-tuned with the help of Mayfly Optimization (MFO) algorithm. The design of GSO and MFO techniques result in reduced computation complexity and improved classification performance. The presented GSO-MFWELM technique was validated for its performance against benchmark dataset and the results were inspected under several aspects. The simulation results established the supremacy of GSO-MFWELM technique over recent approaches with the maximum classification accuracy of 0.9589.     

Analytical Score for Multivariate GARCH Models

Multivariate GARCH models constitute the workhorse of empiricalapplications in several fields, a notable example being financialeconometrics. Unfortunately, ML (or quasi-ML) estimation of such models,although relatively straightforward in theory, is often made difficult bythe fact that available software relies on numerical methods for computingthe first derivatives of the log-likelihood; the fact that these modelsoften include a large number of parameters makes it impractical toestimate even medium-sized models. In this paper, closed-form expressionsfor the score of the BEKK model of Engle and Kroner (1995) are obtained,and strategies for efficient computation are discussed.     

Systematic literature review of machine learning based software development effort estimation models

Context

Software development effort estimation (SDEE) is the process of predicting the effort required to develop a software system. In order to improve estimation accuracy, many researchers have proposed machine learning (ML) based SDEE models (ML models) since 1990s. However, there has been no attempt to analyze the empirical evidence on ML models in a systematic way.

Objective

This research aims to systematically analyze ML models from four aspects: type of ML technique, estimation accuracy, model comparison, and estimation context.

Method

We performed a systematic literature review of empirical studies on ML model published in the last two decades (1991-2010).

Results

We have identified 84 primary studies relevant to the objective of this research. After investigating these studies, we found that eight types of ML techniques have been employed in SDEE models. Overall speaking, the estimation accuracy of these ML models is close to the acceptable level and is better than that of non-ML models. Furthermore, different ML models have different strengths and weaknesses and thus favor different estimation contexts.

Conclusion

ML models are promising in the field of SDEE. However, the application of ML models in industry is still limited, so that more effort and incentives are needed to facilitate the application of ML models. To this end, based on the findings of this review, we provide recommendations for researchers as well as guidelines for practitioners.     

Analysis of dataset selection for multi-fidelity surrogates for a turbine problem

Multi-fidelity surrogates (MFS) have become a popular way to combine small number of expensive high-fidelity (HF) samples and many cheap low-fidelity (LF) samples. In some situations LF samples can come from multiple sources and sometimes the HF samples alone can obtain a more accurate surrogate than the combination (HF&LF). Therefore this paper considers using maximum likelihood (ML) and cross validation (CV) to select the dataset leading to best surrogate accuracy, when multiple sample sources are available. The kriging and co-kriging techniques were employed to build surrogates. Unlike conventional model selection, the multi-fidelity datasets selection by ML and CV has to compare the surrogate accuracy of different true functions. The effectiveness of ML and CV is examined through a two-variable turbine problem, where samples can come from one HF and two LF models. The indicators were used to select between using only HF samples or combining them with one set of LF samples or the other. The best selection proved to depend on the design of experiments (DOE), and so datasets were generated for a large number of DOEs. It was found the CV and ML worked relatively well in selection between two LF sample sources for MFS. When selecting between only HF and HF&LF, the ML, which is frequently used in co-kriging hyper-parameter estimation, failed in detecting when the surrogate accuracy of only HF was better than HF & LF. The CV was successful only part of the time. The reasons behind the poor performance are analyzed with the help of a 1D example.     

Deep convolutional learning for general early design stage prediction models

Designers rely on performance predictions to direct the design toward appropriate requirements. Machine learning (ML) models exhibit the potential for rapid and accurate predictions. Developing conventional ML models that can be generalized well in unseen design cases requires an effective feature engineering and selection. Identifying generalizable features calls for good domain knowledge by the ML model developer. Therefore, developing ML models for all design performance parameters with conventional ML will be a time-consuming and expensive process. Automation in terms of feature engineering and selection will accelerate the use of ML models in design.Deep learning models extract features from data, which aid in model generalization. In this study, we (1) evaluate the deep learning model’s capability to predict the heating and cooling demand on unseen design cases and (2) obtain an understanding of extracted features. Results indicate that deep learning model generalization is similar to or better than that of a simple neural network with appropriate features. The reason for the satisfactory generalization using the deep learning model is its ability to identify similar design options within the data distribution. The results also indicate that deep learning models can filter out irrelevant features, reducing the need for feature selection.     

Maximum likelihood estimation methods for multispectral randomfield image models

Considers the problem of estimating parameters of multispectral random field (RF) image models using maximum likelihood (ML) methods. For images with an assumed Gaussian distribution, analytical results are developed for multispectral simultaneous autoregressive (MSAR) and Markov random field (MMRF) models which lead to practical procedures for calculating ML estimates. Although previous work has provided least squares methods for parameter estimation, the superiority of the ML method is evidenced by experimental results provided in this work. The effectiveness of multispectral RF models using ML estimates in modeling color texture images is also demonstrated     

Approximate conditional least squares estimation of a nonlinear state-space model via an unscented Kalman filter

The problem of estimating a nonlinear state-space model whose state process is driven by an ordinary differential equation (ODE) or a stochastic differential equation (SDE), with discrete-time data is studied. A new estimation method is proposed based on minimizing the conditional least squares (CLS) with the conditional mean function computed approximately via the unscented Kalman filter (UKF). Conditions are derived for the UKF–CLS estimator to preserve the limiting properties of the exact CLS estimator, namely, consistency and asymptotic normality, under the framework of infill asymptotics, i.e. sampling is increasingly dense over a fixed domain. The efficacy of the proposed method is demonstrated by simulation and a real application.     

Large margin mixture of AR models for time series classification

In this paper, we propose the large margin autoregressive (LMAR) model for classification of time series patterns. The parameters of the generative AR models for different classes are estimated using the margin of the boundaries of AR models as the optimization criterion. Models that use a mixture of AR (MAR) models are considered for representing the data that cannot be adequately represented using a single AR model for a class. Based on a mixture model representing each class, we propose the large margin mixture of AR (LMMAR) models. The proposed methods are applied on the simulated time series data, electrocardiogram data, speech data for E-set in English alphabet and electroencephalogram time series data. Performance of the proposed methods is compared with that of support vector machine (SVM) based classifier that uses AR coefficients based features. The proposed methods give a better classification performance compared to the SVM based classifier. Being generative models, the LMAR and LMMAR models provide a generative interpretation that enables utilization of the rejection option in the high risk classification tasks. The proposed methods can also be used for detection of novel time series data.     

ML aided context feature extraction for cognitive radio

This paper addresses the estimation of different context features of a primary user network, such as transmitters’ positions, antenna patterns and directions, and propagation model characteristics. It is based on radio signal strength measurements obtained by a sensor network without any prior knowledge about the configuration of the primary transmitters in terms of antenna types or propagation model. A Maximum Likelihood Aided Context Feature Extraction (MLACFE) method is introduced based on applying image processing and a Maximum Likelihood estimation algorithm over the set of measurements to identify the existing transmitters in the scenario and their parameters. The proposed method can provide a quite similar performance than a classical ML method, in terms of average estimation errors while at the same time reducing the computation time in about three orders of magnitude, for the considered case study.     

Automatic spectral analysis with missing data

The current computational power and some recently developed algorithms allow a new automatic spectral analysis method for randomly missing data. Accurate spectra and autocorrelation functions are computed from the estimated parameters of time series models, without user interaction. If only a few data are missing, the accuracy is almost the same as when all observations were available. For larger missing fractions, low-order time series models can still be estimated with a good accuracy if the total observation time is long enough. Autoregressive models are best estimated with the maximum likelihood method if data are missing. Maximum likelihood estimates of moving average and of autoregressive moving average models are not very useful with missing data. Those models are found most accurately if they are derived from the estimated parameters of an intermediate autoregressive model. With statistical criteria for the selection of model order and model type, a completely automatic and numerically reliable algorithm is developed that estimates the spectrum and the autocorrelation function in randomly missing data problems. The accuracy was better than what can be obtained with other methods, including the famous expectation–maximization (EM) algorithm.     

A Multistrategy Approach to Classifier Learning from Time Series

We present an approach to inductive concept learning using multiple models for time series. Our objective is to improve the efficiency and accuracy of concept learning by decomposing learning tasks that admit multiple types of learning architectures and mixture estimation methods. The decomposition method adapts attribute subset selection and constructive induction (cluster definition) to define new subproblems. To these problem definitions, we can apply metric-based model selection to select from a database of learning components, thereby producing a specification for supervised learning using a mixture model. We report positive learning results using temporal artificial neural networks (ANNs), on a synthetic, multiattribute learning problem and on a real-world time series monitoring application.     

Application of neural networks in forecasting engine systems reliability

This paper presents a comparative study of the predictive performances of neural network time series models for forecasting failures and reliability in engine systems. Traditionally, failure data analysis requires specifications of parametric failure distributions and justifications of certain assumptions, which are at times difficult to validate. On the other hand, the time series modeling technique using neural networks provides a promising alternative. Neural network modeling via feed-forward multilayer perceptron (MLP) suffers from local minima problems and long computation time. The radial basis function (RBF) neural network architecture is found to be a viable alternative due to its shorter training time. Illustrative examples using reliability testing and field data showed that the proposed model results in comparable or better predictive performance than traditional MLP model and the linear benchmark based on Box–Jenkins autoregressive-integrated-moving average (ARIMA) models. The effects of input window size and hidden layer nodes are further investigated. Appropriate design topologies can be determined via sensitivity analysis.     

A UD factorization‐based nonlinear adaptive set‐membership filter for ellipsoidal estimation

The extended set‐membership filter (ESMF) for nonlinear ellipsoidal estimation suffers from numerical instability, computation complexity as well as the difficulty in filter parameter selection. In this paper, a UD factorization‐based adaptive set‐membership filter is developed and applied to nonlinear joint estimation of both time‐varying states and parameters. As a result of using the proposed UD factorization, combined with a new sequential and selective measurement update strategy, the numerical stability and real‐time applicability of conventional ESMF are substantially improved. Furthermore, an adaptive selection scheme of the filter parameters is derived to reduce the computation complexity and achieve sub‐optimal estimation. Simulation results have shown the efficiency and robustness of the proposed method. Copyright © 2007 John Wiley & Sons, Ltd.     

Preliminary simulations of the flow field over the Attic Peninsula

In the frame of the APSIS experiment, three mesoscale meteorological models have been used over the Attic Peninsula. TVM (IAG UCl) developed from the URBMET code (USA, 1985) is hydrostatic, boussinesq and written in vorticity. MAR (IAG-UCL) is a fully compressible hydrostatic primitive equations model, and RAMS (CSU, USA) is non hydrostatic and fully compressible. These three models have been tested on the Athens region where strong interactions exist between different sea-breeze cells and the complex topography. Suring unstable daytime conditions, all models are shown to produce similar wind fields and good agreement with measurements, while the stable night-time leads to leads to larger differences.