Maximum likelihood computation based on the Fisher scoring and Gauss-Newton quadratic approximations

The Fisher scoring and Gauss-Newton methods are two known methods for maximum likelihood computation. This paper provides a generalization for each method in a unified manner so that they can be used for some difficult maximum likelihood computation, when, for example, there exist constraints on the parameters. A generalized method does not use directly the Newton-type iteration formulas of these methods, but, instead, uses the corresponding quadratic functions transformed from them. It proceeds by repeatedly approximating the log-likelihood function with the quadratic functions in the neighborhoods of the current iterates and optimizing each quadratic function within the parameter space. It is shown that each quadratic function has a weighted linear regression formulation, which can be conveniently solved. This generalization also extends the applicability of the Fisher scoring method to situations when the expected Fisher information matrices are unavailable in closed form. Fast computation can generally be anticipated, owing to their small rates of convergence and a rapid solution of each linear regression problem. While the generalized Gauss-Newton method may sometimes suffer for the so-called large residual problem, the generalized Fisher scoring method has performed consistently well in the numerical experiments we conducted.     

Type I multivariate zero-inflated Poisson distribution with applications

Motivated from the stochastic representation of the univariate zero-inflated Poisson (ZIP) random variable, the authors propose a multivariate ZIP distribution, called as Type I multivariate ZIP distribution, to model correlated multivariate count data with extra zeros. The distributional theory and associated properties are developed. Maximum likelihood estimates for parameters of interest are obtained by Fisher’s scoring algorithm and the expectation–maximization (EM) algorithm, respectively. Asymptotic and bootstrap confidence intervals of parameters are provided. Likelihood ratio test and score test are derived and are compared via simulation studies. Bayesian methods are also presented if prior information on parameters is available. Two real data sets are used to illustrate the proposed methods. Under both AIC and BIC, our analysis of the two data sets supports the Type I multivariate zero-inflated Poisson model as a much less complex alternative with feasibility to the existing multivariate ZIP models proposed by Li et al. (Technometrics, 29–38, Vol 41, 1999).     

基于高斯分布监督学习样本的Fisher核构造方法

结合实际应用背景, 针对各类样本服从高斯分布的监督学习情形, 提出了构造Fisher核的新方法. 由于利用了样本中的类别信息, 该方法用极大似然估计代替EM算法估计GMM参数, 有效降低了Fisher核构造的时间复杂度. 结合核Fisher分类法, 上述方法在标准人脸库上的仿真实验结果显示, 用所提方法所构造的Fisher核不仅时间复杂度低, 且识别率也优于传统的高斯核与多项式核. 本文的研究有利于将Fisher 核的应用从语音识别领域拓展到图像识别等领域.     

费歇算法在彩色图像分割中的应用

费歇算法是一种优良的线性判决方法，适用于很多分类场合。本文简要介绍了费歇算法的原理，结合实例说明了利用费歇算法分割彩色图像的方法和步骤。分割结果表明，费歇算法能有效地对彩色图像进行分割，取得比较好的效果。     

Iterative weighted least squares algorithms for neural networks classifiers

This paper discusses learning algorithms of layered neural networks from the standpoint of maximum likelihood estimation. At first we discuss learning algorithms for the most simple network with only one neuron. It is shown that Fisher information of the network, namely minus expected values of Hessian matrix, is given by a weighted covariance matrix of input vectors. A learning algorithm is presented on the basis of Fisher's scoring method which makes use of Fisher information instead of Hessian matrix in Newton's method. The algorithm can be interpreted as iterations of weighted least squares method. Then these results are extended to the layered network with one hidden layer. Fisher information for the layered network is given by a weighted covariance matrix of inputs of the network and outputs of hidden units. Since Newton's method for maximization problems has the difficulty when minus Hessian matrix is not positive definite, we propose a learning algorithm which makes use of Fisher information matrix, which is non-negative, instead of Hessian matrix. Moreover, to reduce the computation of full Fisher information matrix, we propose another algorithm which uses only block diagonal elements of Fisher information. The algorithm is reduced to an iterative weighted least squares algorithm in which each unit estimates its own weights by a weighted least squares method. It is experimentally shown that the proposed algorithms converge with fewer iterations than error back-propagation (BP) algorithm.     

Decision tree-based technology credit scoring for start-up firms: Korean case

Various types of Technology Credit Guarantees (TCGs) have been issued to support technology development of start-up firms. Technology evaluation has become a critical part of TCG system. However, general technology credit scoring models have not been applied reflecting the special phenomena of start-ups, which are distinguishable from those of established firms. Furthermore, somewhat complicated approaches have been applied to existing models. We propose a rather simple decision tree-based technology credit scoring for start-ups which can serve as a-replacement for the complicated models currently used for general purposes. Our result is expected to provide valuable information to evaluator for start-up firms.     

多级中文文本情感分类算法研究

针对文本情感分类准确率不高的问题,提出基于CCA-VSM分类器和KFD的多级文本情感分类方法。采用典型相关性分析对文档的权重特征向量和词性特征向量进行降维,在约简向量集上构建向量空间模型,根据模型之间的差异度设计VSM分类器,筛选出与测试文档差异度较小的R个模型作为核Fisher判别的输入,最终判别出文档的情感观点。实验结果表明:该方法比传统支持向量机有较高的分类准确率和较快的分类速度,权重特征和词性特征对分类准确率的影响较大。     

Penalized Fisher Discriminant Analysis and Its Application to Image-Based Morphometry

Image-based morphometry is an important area of pattern recognition research, with numerous applications in science and technology (including biology and medicine). Fisher linear discriminant analysis (FLDA) techniques are often employed to elucidate and visualize important information that discriminates between two or more populations. We demonstrate that the direct application of FLDA can lead to undesirable errors in characterizing such information and that the reason for such errors is not necessarily the ill conditioning in the resulting generalized eigenvalue problem, as usually assumed. We show that the regularized eigenvalue decomposition often used is related to solving a modified FLDA criterion that includes a least-squares-type representation penalty, and derive the relationship explicitly. We demonstrate the concepts by applying this modified technique to several problems in image-based morphometry, and build discriminant representative models for different data sets.     

基于改进随机森林算法的停电敏感用户分类

目前,我国电网企业对于识别停电投诉风险,开展用户停电敏感程度分析的研究工作还处在起步阶段.为了有效地分析停电用户的敏感程度,提出了一种基于改进随机森林算法的停电敏感用户分类算法.首先,对原始数据进行清洗、特征选择等预处理;接着,采用SMOTE算法增加少数敏感用户样本数据量,解决数据分布不均匀问题;然后,以Fisher比作为特征的重要性度量,按比例随机采样选取具有代表性的特征构成子特征空间;最后,利用随机森林算法识别停电敏感用户.通过在真实停电数据上的实验,验证了提出的方法不仅具有较好的准确性和时间性能,而且可以有效处理高维、冗余特征的数据.     

基于虹膜纹理深度特征和Fisher向量的人种分类

目的 虹膜是位于人眼表面黑色瞳孔和白色巩膜之间的圆环形区域,有着丰富的纹理信息。虹膜纹理具有高度的区分性和稳定性。人种分类是解决虹膜识别在大规模数据库上应用难题的主要方法之一。现有的虹膜图像人种分类方法主要采用手工设计的特征,而且针对亚洲人和非亚洲人的基本人种分类,无法很好地解决亚种族分类问题。为此提出一种基于虹膜纹理深度特征和Fisher向量的人种分类方法。方法 首先用CNN（convolutional neural network）对归一化后的虹膜纹理图像提取深度特征向量,作为底层特征;然后使用高斯混合模型提取Fisher向量作为最终的虹膜特征表达;最后用支持向量机分类得到最终结果。结果 本文方法在亚洲人和非亚洲人的数据集上采用non-person-disjoint的方式取得99.93%的准确率,采用person-disjoint的方式取得91.94%的准确率;在汉族人和藏族人的数据集上采用non-person-disjoint的方式取得99.69%的准确率,采用person-disjoint的方式取得82.25%的准确率。结论 本文通过数据驱动的方式从训练数据中学习到更适合人种分类的特征,可以很好地实现对基本人种以及亚种族人种的分类,提高了人种分类的精度。同时也首次证明了用虹膜图像进行亚种族分类的可行性,对人种分类理论进行了进一步地丰富和完善。     

Maximum likelihood estimation of an extended latent Markov model for clustered binary panel data

Computational aspects concerning a model for clustered binary panel data are analyzed. The model is based on the representation of the behavior of a subject (individual panel member) in a given cluster by means of a latent process. This latent process is decomposed into a cluster-specific component and an individual-specific component. The first component follows a first-order Markov chain, whereas the second is time-invariant and is represented by a discrete random variable. An algorithm for computing the joint distribution of the response variables is introduced. The algorithm may be used even in the presence of a large number of subjects in the same cluster. An Expectation-Maximization (EM) scheme for the maximum likelihood estimation of the model is also described together with the estimation of the Fisher information matrix on the basis of the numerical derivative of the score vector. The estimate of this matrix is used to obtain standard errors for the parameter estimates and to check the identifiability of the model and the convergence of the EM algorithm. The approach is illustrated by means of an application to a data set concerning Italian employees’ illness benefits.     

基于动态径向基神经网络的人脸识别算法研究

提出了一个通用而且有效的方法来设计RBF神经网络分类器用于人脸识别。为了避免过拟合和减少计算量,用主元分析法和Fisher线性判别技术来降低维数,以提取人脸特征;利用一个混合的学习算法来训练RBF神经网络,使梯度下降法的搜索空间大大减少;采用一种基于训练样本类别信息的新的聚类算法,所有同类的数据可被聚集在一起,尽量减少不同类数据混杂在一起,同时选取结构尽可能紧凑的RBF神经网络分类器。在ORL数据库上进行了仿真,实验结果表明,该算法具有高效性和有效性。     

Face recognition based on multi-class mapping of Fisher scores

A new hidden Markov model (HMM) based feature generation scheme is proposed for face recognition (FR) in this paper. In this scheme, HMM method is used to model classes of face images. A set of Fisher scores is calculated through partial derivative analysis of the parameters estimated in each HMM. These Fisher scores are further combined with some traditional features such as log-likelihood and appearance based features to form feature vectors that exploit the strengths of both local and holistic features of human face. Linear discriminant analysis (LDA) is then applied to analyze these feature vectors for FR. Performance improvements are observed over stand-alone HMM method and Fisher face method which uses appearance based feature vectors. A further study reveals that, by reducing the number of models involved in the training and testing stages of LDA, the proposed feature generation scheme can maintain very high discriminative power at much lower computational complexity comparing to the traditional HMM based FR system. Experimental results on a public available face database are provided to demonstrate the viability of this scheme.     

System learning approach to assess sustainability and forecast trends in regional dynamics: The San Luis Basin study,Colorado, U.S.A

This paper presents a methodology that combines the power of an Artificial Neural Network and Information Theory to forecast variables describing the condition of a regional system. The novelty and strength of this approach is in the application of Fisher information, a key method in Information Theory, to preserve trends in the historical data and prevent over fitting projections. The methodology was applied to demographic, environmental, food and energy consumption, and agricultural production in the San Luis Basin regional system in Colorado, U.S.A. These variables are important for tracking conditions in human and natural systems. However, available data are often so far out of date that they limit the ability to manage these systems. Results indicate that the approaches developed provide viable tools for forecasting outcomes with the aim of assisting management toward sustainable trends. This methodology is also applicable for modeling different scenarios in other dynamic systems.     

无线传感器网络中基于模型拟合的可信近似查询处理算法

无线传感器网络的一个重要应用是可信地查询网络中所有节点的监测数据.目前,多数研究主要集中在如何利用节点之间的时空相关性,节省能量地查询感知数据.但是这些方法的查询结果不能满足某些应用对数据的高可信要求,也不能适用于节点之间不存在空间相关性或空间相关性不稳定的情况.针对这一问题,提出了基于模型拟合的可信近似查询处理方法.该方法在感知数据集合上寻找具有最小数据传输比的拟合模型,通过传输模型及其参数来代替传输实际的监测数据.理论分析和实验结果证明,基于模型拟合的可信近似查询处理方法不仅能够节省大量能源而且能够返回满足用户精度要求的可信查询结果.     

Exact filters for Newton–Raphson parameter estimation algorithms for continuous-time partially observed stochastic systems

This paper presents explicit finite-dimensional filters for implementing Newton–Raphson (NR) parameter estimation algorithms. The models which exhibit nonlinear parameter dependence are stochastic, continuous-time and partially observed. The implementation of the NR algorithm requires evaluation of the log-likelihood gradient and the Fisher information matrix. Fisher information matrices are important in bounding the estimation error from below, via the Cramer–Rao bound. The derivations are based on relations between incomplete and complete data, likelihood, gradient and Hessian likelihood functions, which are derived using Girsanov's measure transformations.     

A regularization framework for multiclass classification: A deterministic annealing approach

In this paper, we propose a general regularization framework for multiclass classification based on discriminant functions. Since the objective function in the primal optimization problem of this framework is always not differentiable, the optimal solution cannot be obtained directly. With the aid of the deterministic annealing approach, a differentiable objective function is derived subject to a constraint on the randomness of the solution. The problem can be approximated by solving a sequence of differentiable optimization problems, and such approximation converges to the original problem asymptotically. Based on this approach, class-conditional posterior probabilities can be calculated directly without assuming the underlying probabilistic model. We also notice that there is a connection between our approach and some existing statistical models, such as Fisher discriminant analysis and logistic regression.     

On the Fernández-Steel distribution: Inference and application

In this article, we perform statistical inference on a skew model that belongs to a class of distributions proposed by Fernández and Steel (1998). Specifically, we introduce two ways to represent this model by means of which moments and generation of random numbers can be obtained. In addition, we carry out estimation of the model parameters by moment and maximum likelihood methods. Asymptotic inference based on both of these methods is also produced. We analyze the expected Fisher information matrix associated with the model and highlight the fact that this does not have the singularity problem, as occurs with the corresponding information matrix of the skew-normal model introduced by Azzalini (1985). Furthermore, we conduct a simulation study to compare the performance of the moment and maximum likelihood estimators. Finally, an application based on real data is carried out.     

Choice of generalized linear mixed models using predictive crossvalidation

The choice of generalized linear mixed models is difficult, because it involves the selection of both fixed and random effects. Classical criteria like Akaike’s information criterion (AIC) are often not suitable for the latter task, and others which are useful in linear mixed models are difficult to extend to the generalized case, especially for overdispersed data. A predictive leave-one-out crossvalidation approach is suggested that can be applied for choosing both fixed and random effects, even in models with overdispersion, and is based on proper scoring rules. An attractive feature of this approach is the fact that the model has to be fitted just once to the data set, which makes computations fast and convenient. As the calculation of the leave-one-out predictive distribution is not possible analytically, it is shown how an iteratively weighted least squares algorithm combined with some analytic approximations can be used for this task. A simulation study and two applications of the methodology to binary and count data are provided, as well as comparisons with two other methods.     

融合Fisher准则和势函数的多阈值图像分割

针对传统多阈值图像分割算法复杂度高、分割效果欠佳等问题,提出了一种基于Fisher准则和势函数相结合的多阈值图像分割方法。首先对Fisher准则函数进行简化,再对简化后的Fisher准则采用递推算法降低计算复杂度,然后由直方图势函数方法确定图像的分割类数,最后将改进的Fisher准则用于多阈值图像分割,并对最终分割结果进行后续处理。实验结果表明,融合Fisher准则和势函数的多阈值分割方法不仅分割效果好,而且分割时间短,能够运用到实时应用的场合。