Bootstrap variants of the Akaike information criterion for mixed model selection

Two bootstrap-corrected variants of the Akaike information criterion are proposed for the purpose of small-sample mixed model selection. These two variants are asymptotically equivalent, and provide asymptotically unbiased estimators of the expected Kullback-Leibler discrepancy between the true model and a fitted candidate model. The performance of the criteria is investigated in a simulation study where the random effects and the errors for the true model are generated from a Gaussian distribution. The parametric bootstrap is employed. The simulation results suggest that both criteria provide effective tools for choosing a mixed model with an appropriate mean and covariance structure. A theoretical asymptotic justification for the variants is presented in the Appendix.     

Conditional Akaike information criterion for generalized linear mixed models

In this study, a model identification instrument to determine the variance component structure for generalized linear mixed models (glmms) is developed based on the conditional Akaike information (cai). In particular, an asymptotically unbiased estimator of the cai (denoted as caicc) is derived as the model selection criterion which takes the estimation uncertainty in the variance component parameters into consideration. The relationship between bias correction and generalized degree of freedom for glmms is also explored. Simulation results show that the estimator performs well. The proposed criterion demonstrates a high proportion of correct model identification for glmms. Two sets of real data (epilepsy seizure count data and polio incidence data) are used to illustrate the proposed model identification method.     

Automatic modeling algorithm of stochastic error for inertial sensors

This paper proposes an automatic algorithm to determine the properties of stochastic processes and their parameters for inertial error. The proposed approach is based on a recently developed method called the generalized method of wavelet moments (GMWM), whose estimator was proven to be consistent and asymptotically normally distributed. This algorithm is suitable mainly (but not only) for the combination of several stochastic processes, where the model identification and parameter estimation are quite difficult for the traditional methods, such as the Allan variance and the power spectral density analysis. This algorithm further explores the complete stochastic error models and the candidate model ranking criterion to realize automatic model identification and determination. The best model is selected by making the trade-off between the model accuracy and the model complexity. The validation of this approach is verified by practical examples of model selection for MEMS-IMUs (micro-electro-mechanical system inertial measurement units) in varying dynamic conditions.     

Model selection for zero-inflated regression with missing covariates

Count data are widely existed in the fields of medical trials, public health, surveys and environmental studies. In analyzing count data, it is important to find out whether the zero-inflation exists or not and how to select the most suitable model. However, the classic AIC criterion for model selection is invalid when the observations are missing. In this paper, we develop a new model selection criterion in line with AIC for the zero-inflated regression models with missing covariates. This method is a modified version of Monte Carlo EM algorithm which is based on the data augmentation scheme. One of the main attractions of this new method is that it is applicable for comparison of candidate models regardless of whether there are missing data or not. What is more, it is very simple to compute as it is just a by-product of Monte Carlo EM algorithm when the estimations of parameters are obtained. A simulation study and a real example are used to illustrate the proposed methodologies.     

Bayesian projection approaches to variable selection in generalized linear models

A Bayesian approach to variable selection which is based on the expected Kullback-Leibler divergence between the full model and its projection onto a submodel has recently been suggested in the literature. For generalized linear models an extension of this idea is proposed by considering projections onto subspaces defined via some form of L₁ constraint on the parameter in the full model. This leads to Bayesian model selection approaches related to the lasso. In the posterior distribution of the projection there is positive probability that some components are exactly zero and the posterior distribution on the model space induced by the projection allows exploration of model uncertainty. Use of the approach in structured variable selection problems such as ANOVA models is also considered, where it is desired to incorporate main effects in the presence of interactions. Projections related to the non-negative garotte are able to respect the hierarchical constraints. A consistency result is given concerning the posterior distribution on the model induced by the projection, showing that for some projections related to the adaptive lasso and non-negative garotte the posterior distribution concentrates on the true model asymptotically.     

Bayesian adaptive bandwidth kernel density estimation of irregular multivariate distributions

In this paper, we propose a new methodology for multivariate kernel density estimation in which data are categorized into low- and high-density regions as an underlying mechanism for assigning adaptive bandwidths. We derive the posterior density of the bandwidth parameters via the Kullback-Leibler divergence criterion and use a Markov chain Monte Carlo (MCMC) sampling algorithm to estimate the adaptive bandwidths. The resulting estimator is referred to as the tail-adaptive density estimator. Monte Carlo simulation results show that the tail-adaptive density estimator outperforms the global-bandwidth density estimators implemented using different global bandwidth selection rules. The inferential potential of the tail-adaptive density estimator is demonstrated by employing the estimator to estimate the bivariate density of daily index returns observed from the USA and Australian stock markets.     

Evaluate the number of clusters in finite mixture models with the penalized histogram difference criterion

Aimed at the determination of the number of mixtures for finite mixture models (FMMs), in this work, a new method called the penalized histogram difference criterion (PHDC) is proposed and evaluated with other criteria such as Akaike information criterion (AIC), the minimum message length (MML), the information complexity (ICOMP) and the evidence of data criterion (EDC). The new method, which calculates the penalized histogram difference between the data generated from estimated FMMs and those for modeling purpose, turns out to be better than others for data with complicate mixtures patterns. It is demonstrated in this work that the PHDC can determine the optimal number of clusters of the FMM. Furthermore, the estimated FMMs asymptotically approximate the true model. The utility of the new method is demonstrated through synthetic data sets analysis and the batch-wise comparison of citric acid fermentation processes.     

Bayesian optimum designs for discriminating between models with any distribution

The Bayesian KL-optimality criterion is useful for discriminating between any two statistical models in the presence of prior information. If the rival models are not nested then, depending on which model is true, two different Kullback-Leibler distances may be defined. The Bayesian KL-optimality criterion is a convex combination of the expected values of these two possible Kullback-Leibler distances between the competing models. These expectations are taken over the prior distributions of the parameters and the weights of the convex combination are given by the prior probabilities of the models. Concavity of the Bayesian KL-optimality criterion is proved, thus classical results of Optimal Design Theory can be applied. A standardized version of the proposed criterion is also given in order to take into account possible different magnitudes of the two Kullback-Leibler distances. Some illustrative examples are provided.     

Modeling strategies in longitudinal data analysis: Covariate, variance function and correlation structure selection

A modeling paradigm is proposed for covariate, variance and working correlation structure selection for longitudinal data analysis. Appropriate selection of covariates is pertinent to correct variance modeling and selecting the appropriate covariates and variance function is vital to correlation structure selection. This leads to a stepwise model selection procedure that deploys a combination of different model selection criteria. Although these criteria find a common theoretical root based on approximating the Kullback-Leibler distance, they are designed to address different aspects of model selection and have different merits and limitations. For example, the extended quasi-likelihood information criterion (EQIC) with a covariance penalty performs well for covariate selection even when the working variance function is misspecified, but EQIC contains little information on correlation structures. The proposed model selection strategies are outlined and a Monte Carlo assessment of their finite sample properties is reported. Two longitudinal studies are used for illustration.     

A block-diagonal growth curve model

We consider a variation of the growth curve (GC) model, referred to as the block-diagonal growth curve (BDGC) model, where the unknown regression coefficient matrix is constrained to be block-diagonal. A closed-form approximate maximum likelihood (AML) estimator for this model is derived based on the maximum likelihood principle. We analyze the statistical properties of this method theoretically and show that the AML estimate is unbiased and asymptotically statistically efficient for a large snapshot number. Via numerical examples in wireless communications, we also show that the proposed AML estimator can achieve excellent estimation accuracy.     

基于查询聚类的排序学习算法

排序学习算法作为信息检索与机器学习的一个交叉领域,越来越受到人们的重视。然而,几乎没有排序学习算法考虑到查询差异的存在。文中查询被建模为多元高斯分布,KL距离被用来度量查询之间的距离,利用谱聚类方法对查询进行聚类,为每个聚类类别训练一个排序函数。实验结果表明经过聚类得到的排序函数需要较少的训练样例,但是它的性能却和没有经过聚类得到的排序函数具有可比性,甚至优于后者。     

Model Selection for Unsupervised Learning of Visual Context

This study addresses the problem of choosing the most suitable probabilistic model selection criterion for unsupervised learning of visual context of a dynamic scene using mixture models. A rectified Bayesian Information Criterion (BICr) and a Completed Likelihood Akaike’s Information Criterion (CL-AIC) are formulated to estimate the optimal model order (complexity) for a given visual scene. Both criteria are designed to overcome poor model selection by existing popular criteria when the data sample size varies from small to large and the true mixture distribution kernel functions differ from the assumed ones. Extensive experiments on learning visual context for dynamic scene modelling are carried out to demonstrate the effectiveness of BICr and CL-AIC, compared to that of existing popular model selection criteria including BIC, AIC and Integrated Completed Likelihood (ICL). Our study suggests that for learning visual context using a mixture model, BICr is the most appropriate criterion given sparse data, while CL-AIC should be chosen given moderate or large data sample sizes.     

A New Distance Measure for Model-Based Sequence Clustering

We review the existing alternatives for defining model-based distances for clustering sequences and propose a new one based on the Kullback-Leibler divergence. This distance is shown to be especially useful in combination with spectral clustering. For improved performance in real-world scenarios, a model selection scheme is also proposed.     

Subspace information criterion for nonquadratic regularizers-Modelselection for sparse regressors

Nonquadratic regularizers, in particular the l(1) norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l(1) norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced,which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.     

嵌套删失数据期望最大化的高斯混合聚类算法

针对聚类问题中的非随机性缺失数据, 本文基于高斯混合聚类模型, 分析了删失型数据期望最大化算法的有效性, 并揭示了删失数据似然函数对模型算法的作用机制. 从赤池弘次信息准则、信息散度等指标, 比较了所提出方法与标准的期望最大化算法的优劣性. 通过删失数据划分及指示变量, 推导了聚类模型参数后验概率及似然函数, 调整了参数截尾正态函数的一阶和二阶估计量. 并根据估计算法的有效性理论, 通过关于得分向量期望的方程得出算法估计的最优参数. 对于同一删失数据集, 所提出的聚类算法对数据聚类中心估计更精准. 实验结果证实了所提出算法在高斯混合聚类的性能上优于标准的随机性缺失数据期望最大化算法.     

融合[L2]和KL保真项的图像恢复算法

为了有效抑制高斯-泊松混合噪声,针对调和模型不能有效保存图像的边缘细节信息和Kullback-Leibler散度作为保真项（KL保真项）的全变差图像恢复模型对光滑的区域部分去噪会产生“阶梯效应”的不足,提出一种针对高斯-泊松混合噪声去噪的图像恢复变分模型。该模型利用增广拉格朗日算法进行数值实现,将调和模型和全变分模型按照比例进行融合,结合两种模型的优点,增强模型的去噪性能;Kullback-Leibler散度作为保真项和[L2]保真项按照比例进行混合,能有效去除高斯-泊松混合噪声的同时,保护图像的边缘细节;使用多幅含不同混合噪声的图像进行对比实验,采用峰值信噪比、结构相似度指标评定图像的恢复效果。实验结果表明,该模型的峰值信噪比和结构相似度大于使用Kullback-Leibler散度作为保真项的全变差图像恢复（TV-KL）模型、改进MS模型（MRT）,以及保真项混合模型（MFT）这三个模型,并且计算的CPU时间更短,去噪效果得到明显改善。所提模型具有更好的去噪性能,有效地保持了图像细节和纹理特征方面的信息,获得了更理想的视觉效果,不仅能提高了图像质量,而且在客观上得到了有效的证实,可以应用于X射线图像去噪。     

Analytic performance investigation of signal level estimator based on empirical characteristic function in impulsive noise

In this paper, we evaluate the mean square error (MSE) performance of empirical characteristic function (ECF) based signal level estimator in a binary communication system. By calculating Cramér-Rao lower bound (CRLB) we investigate the performance of the ECF based estimator in the presence of Laplace and Gaussian mixture noises. We have derived an analytic expression for the variance of the ECF based estimator which shows that it is asymptotically unbiased and consistent. Simulation and analytic results indicate that the ECF based level estimator outperforms the previously proposed estimators in some signal to noise ratio (SNR) regions when the observation noise distribution is unknown.     

Network information criterion-determining the number of hiddenunits for an artificial neural network model

The problem of model selection, or determination of the number of hidden units, can be approached statistically, by generalizing Akaike's information criterion (AIC) to be applicable to unfaithful (i.e., unrealizable) models with general loss criteria including regularization terms. The relation between the training error and the generalization error is studied in terms of the number of the training examples and the complexity of a network which reduces to the number of parameters in the ordinary statistical theory of AIC. This relation leads to a new network information criterion which is useful for selecting the optimal network model based on a given training set.