Penalized spline estimation for functional coefficient regression models

The functional coefficient regression models assume that the regression coefficients vary with some “threshold” variable, providing appreciable flexibility in capturing the underlying dynamics in data and avoiding the so-called “curse of dimensionality” in multivariate nonparametric estimation. We first investigate the estimation, inference, and forecasting for the functional coefficient regression models with dependent observations via penalized splines. The P-spline approach, as a direct ridge regression shrinkage type global smoothing method, is computationally efficient and stable. With established fixed-knot asymptotics, inference is readily available. Exact inference can be obtained for fixed smoothing parameter λ, which is most appealing for finite samples. Our penalized spline approach gives an explicit model expression, which also enables multi-step-ahead forecasting via simulations. Furthermore, we examine different methods of choosing the important smoothing parameter λ: modified multi-fold cross-validation (MCV), generalized cross-validation (GCV), and an extension of empirical bias bandwidth selection (EBBS) to P-splines. In addition, we implement smoothing parameter selection using mixed model framework through restricted maximum likelihood (REML) for P-spline functional coefficient regression models with independent observations. The P-spline approach also easily allows different smoothness for different functional coefficients, which is enabled by assigning different penalty λ accordingly. We demonstrate the proposed approach by both simulation examples and a real data application.     

A new independent component analysis for speech recognition and separation

This paper presents a novel nonparametric likelihood ratio (NLR) objective function for independent component analysis (ICA). This function is derived through the statistical hypothesis test of independence of random observations. A likelihood ratio function is developed to measure the confidence toward independence. We accordingly estimate the demixing matrix by maximizing the likelihood ratio function and apply it to transform data into independent component space. Conventionally, the test of independence was established assuming data distributions being Gaussian, which is improper to realize ICA. To avoid assuming Gaussianity in hypothesis testing, we propose a nonparametric approach where the distributions of random variables are calculated using kernel density functions. A new ICA is then fulfilled through the NLR objective function. Interestingly, we apply the proposed NLR-ICA algorithm for unsupervised learning of unknown pronunciation variations. The clusters of speech hidden Markov models are estimated to characterize multiple pronunciations of subword units for robust speech recognition. Also, the NLR-ICA is applied to separate the linear mixture of speech and audio signals. In the experiments, NLR-ICA achieves better speech recognition performance compared to parametric and nonparametric minimum mutual information ICA.     

Data-based mechanistic modelling of stochastic rainfall-flow processes by state dependent parameter estimation

Due to the inherent nonlinearity in the process of transformation of rainfall into river flow, a simple direct input-output transfer function (TF) model may not sufficiently capture the catchment's hydrological dynamics. This paper presents an application of state dependent parameter (SDP) models for nonlinear, stochastic dynamic system to identify the location and form of the nonlinearity in the rainfall-effective rainfall dynamics. The objective was to develop an effective rainfall input time series that was then used to improve the performance of an originally developed direct input-output TF model of daily rainfall-flow relationship. The CAPTAIN Toolbox in the MATLAB^® environment was used in the model identification in which the recursive filtering and smoothing procedures formulated within a stochastic state space setting were applied to the time series data in order to identify the location and form of nonlinearities within a generic TF model. The nonparametric estimation as well as the parametric optimisation of the resulting nonlinear models was done using the Curve Fitting Toolbox in MATLAB^®. The results showed an improved and more parsimonious TF model. The model improved from explaining only 13% of the data to 56% presenting an improvement of 43% in the model fit. The study demonstrates that simple stochastic but robust tools can be successfully applied to develop and improve applicable hydrological models.     

Stepwise variable selection and maximum likelihood estimation of smoothing factors of kernel functions for nonparametric discriminant functions evaluated by different criteria

Kernel functions are used to estimate the probability density functions of variables for nonparametric discriminant analysis. In connection with stepwise variable identification a stepwise maximum likelihood estimation procedure for the estimation of smoothing factors of the kernel functions is developed. This procedure allows a step-by-step estimation of smoothing factors for every variable which is considered to be added to the model or which is examined to substitute a variable in a model. Different criteria for model evaluation in stepwise discriminant analysis are discussed. Beside criteria, like distance and dependence functions and the error and nonerror rate, a criterion which considers the ratio of probability densities of different classes at point x is proposed for stepwise variable identification. An application of the procedures described in this study to a medical decision problem shows the importance of stepwise parameter estimation of kernel functions for nonparametric discriminant analysis and the role of different model evaluation criteria for the selection of the best subset of variables.     

Consistency of functional learning methods based on derivatives

In some real world applications, such as spectrometry, functional models achieve better predictive performances if they work on the derivatives of order m of their inputs rather than on the original functions. As a consequence, the use of derivatives is a common practice in Functional Data Analysis, despite a lack of theoretical guarantees on the asymptotically achievable performances of a derivative based model. In this paper, we show that a smoothing spline approach can be used to preprocess multivariate observations obtained by sampling functions on a discrete and finite sampling grid in a way that leads to a consistent scheme on the original infinite dimensional functional problem. This work extends (Mas and Pumo, 2009) to nonparametric approaches and incomplete knowledge. To be more precise, the paper tackles two difficulties in a nonparametric framework: the information loss due to the use of the derivatives instead of the original functions and the information loss due to the fact that the functions are observed through a discrete sampling and are thus also unperfectly known: the use of a smoothing spline based approach solves these two problems. Finally, the proposed approach is tested on two real world datasets and the approach is experimentaly proven to be a good solution in the case of noisy functional predictors.     

Hybrid Bayes factors for genome-wide association studies when a robust test is used

Bayes factor (BF) is often used to measure evidence against the null hypothesis in Bayesian hypothesis testing. In the analysis of genome-wide association (GWA) studies, extreme BF values support the associations detected based on significant p-values. Results from recent GWA studies are presented, which show that existing BFs may not be consistent withp-values when a robust test is used due to using different genetic models in the BF and p-value approaches and this may result in misleading conclusions. Two hybrid BFs, which combine the advantages of both the frequentist and Bayesian methods, are then proposed for the markers showing at least moderate associations (p-value <10⁻⁵) based on a robust test. One is Bayesian model averaging using a posterior weighted likelihood and the other is the maximum BF using a profile likelihood. The proposed hybrid BFs and p-values of robust tests do not depend on a single genetic model, but instead, consolidate information over a set of models. We compare the hybrid BFs with two existing BF approaches, including an existing Bayesian model averaging method, in terms of false and true positive rates by simulations. The results show that, for markers showing at least moderate associations, both the hybrid BFs have higher true positive rates than the two existing BFs, while all false positive rates are similar. Applications of the two hybrid BFs to the markers associated with bipolar disorder, type 2 diabetes and age-related macular degeneration are presented. Our hybrid BFs provide better and more robust measures to compare significantly associated markers within and across GWA studies.     

Size and power properties of some tests in the Birnbaum-Saunders regression model

The Birnbaum-Saunders distribution has been used quite effectively to model times to failure for materials subject to fatigue and for modeling lifetime data. In this paper we obtain asymptotic expansions, up to order n^−1/2 and under a sequence of Pitman alternatives, for the non-null distribution functions of the likelihood ratio, Wald, score and gradient test statistics in the Birnbaum-Saunders regression model. The asymptotic distributions of all four statistics are obtained for testing a subset of regression parameters and for testing the shape parameter. Monte Carlo simulation is presented in order to compare the finite-sample performance of these tests. We also present two empirical applications.     

Generalized Additive Models and Inflated Type I Error Rates of Smoother Significance Tests

Generalized additive models (GAMs) have distinct advantages over generalized linear models as they allow investigators to make inferences about associations between outcomes and predictors without placing parametric restrictions on the associations. The variable of interest is often smoothed using a locally weighted scatterplot smoothing (LOESS) and the optimal span (degree of smoothing) can be determined by minimizing the Akaike Information Criterion (AIC). A natural hypothesis when using GAMs is to test whether the smoothing term is necessary or if a simpler model would suffice. The statistic of interest is the difference in deviances between models including and excluding the smoothed term. As approximate chi-square tests of this hypothesis are known to be biased, permutation tests are a reasonable alternative. We compare the type I error rates of the chi-square test and of three permutation test methods using synthetic data generated under the null hypothesis. In each permutation method a distribution of differences in deviances is obtained from 999 permuted datasets and the null hypothesis is rejected if the observed statistic falls in the upper 5% of the distribution. One test is a conditional permutation test using the optimal span size for the observed data; this span size is held constant for all permutations. This test is shown to have an inflated type I error rate. Alternatively, the span size can be fixed a priori such that the span selection technique is not reliant on the observed data. This test is shown to be unbiased; however, the choice of span size is not clear. A third method is an unconditional permutation test where the optimal span size is selected for observed and permuted datasets. This test is unbiased though computationally intensive.     

A new approach to bootstrap inference in functional coefficient models

To infer on functional dependence of regression parameters, a new, factor based bootstrap approach is introduced, that is robust under various forms of heteroskedastic error terms. Modeling the functional coefficient parametrically, the bootstrap approximation of an F-statistic is shown to hold asymptotically. In simulation studies with both parametric and nonparametric functional coefficients, factor based bootstrap inference outperforms the wild bootstrap and pairs bootstrap approach, according to its rejection frequencies under the null hypothesis. Applying the functional coefficient model to a cross sectional investment regression on savings, the saving retention coefficient is found to depend on third variables as the population growth rate and the openness ratio.     

Bayesian analysis of semiparametric reproductive dispersion mixed-effects models

Semiparametric reproductive dispersion mixed-effects model (SPRDMM) is an extension of the reproductive dispersion model and the semiparametric mixed model, and it includes many commonly encountered models as its special cases. A Bayesian procedure is developed for analyzing SPRDMMs on the basis of P-spline estimates of nonparametric components. A hybrid algorithm combining the Gibbs sampler and the Metropolis-Hastings algorithm is used to simultaneously obtain the Bayesian estimates of unknown parameters, smoothing function and random effects, as well as their standard error estimates. The Bayes factor for model comparison is employed to select better approximation of the smoothing function via path sampling. Several simulation studies and a real example are used to illustrate the proposed methodologies.     

Improved likelihood inference in Birnbaum-Saunders regressions

The Birnbaum-Saunders regression model is commonly used in reliability studies. We address the issue of performing inference in this class of models when the number of observations is small. Our simulation results suggest that the likelihood ratio test tends to be liberal when the sample size is small. We obtain a correction factor which reduces the size distortion of the test. Also, we consider a parametric bootstrap scheme to obtain improved critical values and improved p-values for the likelihood ratio test. The numerical results show that the modified tests are more reliable in finite samples than the usual likelihood ratio test. We also present an empirical application.     

Goodness-of-fit tests for the error distribution in nonparametric regression

Suppose the random vector (X,Y) satisfies the regression model Y=m(X)+σ(X)ε, where m(⋅) is the conditional mean, σ²(⋅) is the conditional variance, and ε is independent of X. The covariate X is d-dimensional (d≥1), the response Y is one-dimensional, and m and σ are unknown but smooth functions. Goodness-of-fit tests for the parametric form of the error distribution are studied under this model, without assuming any parametric form for m or σ. The proposed tests are based on the difference between a nonparametric estimator of the error distribution and an estimator obtained under the null hypothesis of a parametric model. The large sample properties of the proposed test statistics are obtained, as well as those of the estimator of the parameter vector under the null hypothesis. Finally, the finite sample behavior of the proposed statistics, and the selection of the bandwidths for estimating m and σ are extensively studied via simulations.     

The marginalized likelihood ratio test for detecting abrupt changes

The generalized likelihood ratio (GLR) test is a widely used method for detecting abrupt changes in linear systems and signals. In this paper the marginalized likelihood ratio (MLR) test is introduced for eliminating three shortcomings of GLR while preserving its applicability and generality. First, the need for a user-chosen threshold is eliminated in MLR. Second, the noise levels need not be known exactly and may even change over time, which means that MLR is robust. Finally, a very efficient exact implementation with linear in time complexity for batch-wise data processing is developed. This should be compared to the quadratic in time complexity of the exact GLR     

A framework for modelling overdispersed count data, including the Poisson-shifted generalized inverse Gaussian distribution

A variety of methods of modelling overdispersed count data are compared. The methods are classified into three main categories. The first category are ad hoc methods (i.e. pseudo-likelihood, (extended) quasi-likelihood, double exponential family distributions). The second category are discretized continuous distributions and the third category are observational level random effects models (i.e. mixture models comprising explicit and non-explicit continuous mixture models and finite mixture models). The main focus of the paper is a family of mixed Poisson distributions defined so that its mean μ is an explicit parameter of the distribution. This allows easier interpretation when μ is modelled using explanatory variables and provides a more orthogonal parameterization to ease model fitting. Specific three parameter distributions considered are the Sichel and Delaporte distributions. A new four parameter distribution, the Poisson-shifted generalized inverse Gaussian distribution is introduced, which includes the Sichel and Delaporte distributions as a special and a limiting case respectively. A general formula for the derivative of the likelihood with respect to μ, applicable to the whole family of mixed Poisson distributions considered, is given. Within the framework introduced here all parameters of the distributions are modelled as parametric and/or nonparametric (smooth) functions of explanatory variables. This provides a very flexible way of modelling count data. Maximum (penalized) likelihood estimation is used to fit the (non)parametric models.     

The likelihood ratio test for hidden Markov models in two-sample problems

The asymptotic distribution of the likelihood ratio test statistic in two-sample testing problems for hidden Markov models is derived when allowing for unequal sample sizes as well as for different families of state-dependent distributions. In both cases under regularity conditions the limit distribution is a standard χ²-distribution, and in particular does not depend on the ratio of the distinct sample sizes. In a simulation study, the finite sample properties are investigated, and the methodology is illustrated in an application to modeling the movement of Drosophila larvae.     

On frequency estimation for a periodic ergodic diffusion process

We consider the problem of frequency estimation by observations for a periodic diffusion process possessing ergodic properties in two different situations. The first corresponds to a trend coefficient continuously differentiable with respect to parameter, and the second, to a discontinuous trend coefficient. It is shown that in the first case the maximum likelihood and Bayesian estimators are asymptotically normal with rate T ^3/2, and in the second case these estimators have different limit distributions with rate T ².     

Speaker Verification Using Adapted Gaussian Mixture Models

Reynolds, Douglas A., Quatieri, Thomas F., and Dunn, Robert B., Speaker Verification Using Adapted Gaussian Mixture Models, Digital Signal Processing10(2000), 19–41.In this paper we describe the major elements of MIT Lincoln Laboratory's Gaussian mixture model (GMM)-based speaker verification system used successfully in several NIST Speaker Recognition Evaluations (SREs). The system is built around the likelihood ratio test for verification, using simple but effective GMMs for likelihood functions, a universal background model (UBM) for alternative speaker representation, and a form of Bayesian adaptation to derive speaker models from the UBM. The development and use of a handset detector and score normalization to greatly improve verification performance is also described and discussed. Finally, representative performance benchmarks and system behavior experiments on NIST SRE corpora are presented.     

On the ability of adaptation of speech signals and data hiding

The efficient wavelet masking is a scheme of speech-in-speech hiding based on the ability of adaptation of speech signals under the hypothesis “any (speech) secret signal may seem similar to a (speech) host signal if its wavelet coefficients are sorted” (Ballesteros L & Moreno A, 2012). In this paper, we delimitate the conditions under which the above hypothesis is true, as follows: (i) the secret and host signals must belong to legible voice signals, (ii) both signals must have the same sampling frequency, (iii) both signals must have the same time-frame, and finally (iv) the ratio between the non-zero coefficients of them should be in the interval [0.8 1.2]. Experimental tests were conducted to demonstrate the hypothesis on different cases: vowel to vowel, message to message and vowel & message, in three languages: English, French, and German. The parameter used to measure the similarity between the adapted secret message and the host signal is the squared Pearson correlation coefficient, r². The results demonstrate that the hypothesis is true under the theoretical conditions because in all the test cases r² was closed to 1 and the p-value was lower than 0.05.     

Probabilistic neural network training procedure based on Q(0)-learning algorithm in medical data classification

In this article, an iterative procedure is proposed for the training process of the probabilistic neural network (PNN). In each stage of this procedure, the Q(0)-learning algorithm is utilized for the adaptation of PNN smoothing parameter (σ). Four classes of PNN models are regarded in this study. In the case of the first, simplest model, the smoothing parameter takes the form of a scalar; for the second model, σ is a vector whose elements are computed with respect to the class index; the third considered model has the smoothing parameter vector for which all components are determined depending on each input attribute; finally, the last and the most complex of the analyzed networks, uses the matrix of smoothing parameters where each element is dependent on both class and input feature index. The main idea of the presented approach is based on the appropriate update of the smoothing parameter values according to the Q(0)-learning algorithm. The proposed procedure is verified on six repository data sets. The prediction ability of the algorithm is assessed by computing the test accuracy on 10 %, 20 %, 30 %, and 40 % of examples drawn randomly from each input data set. The results are compared with the test accuracy obtained by PNN trained using the conjugate gradient procedure, support vector machine algorithm, gene expression programming classifier, k–Means method, multilayer perceptron, radial basis function neural network and learning vector quantization neural network. It is shown that the presented procedure can be applied to the automatic adaptation of the smoothing parameter of each of the considered PNN models and that this is an alternative training method. PNN trained by the Q(0)-learning based approach constitutes a classifier which can be treated as one of the top models in data classification problems.     

Using intelligent methods to predict air-demand ratio in venturi weirs

Artificial intelligent methods are today extensively used in many areas. They are known as powerful tools to solve engineering problems with uncertainties. The purpose of this study was to develop a model, using artificial intelligent methods, for estimating air-demand ratio in venturi weirs. For this aim, Adaptive Network based Fuzzy Inference Systems (ANFIS) and Artificial Neural Network (ANNs) methods were used. The test results revealed that ANFIS model predicted the measured values at higher accuracy than ANNs model. Average correlation coefficients (R²) in ANFIS models were achieved equal to 0.9623 for β = 0.75 and 0.9666 for β = 0.50. Extremely good agreement between the predicted and measured values confirms that ANFIS model can be successfully used to predict air-demand ratio in venturi weirs.