Using continuous features in the maximum entropy model

We investigate the problem of using continuous features in the maximum entropy (MaxEnt) model. We explain why the MaxEnt model with the moment constraint (MaxEnt-MC) works well with binary features but not with the continuous features. We describe how to enhance constraints on the continuous features and show that the weights associated with the continuous features should be continuous functions instead of single values. We propose a spline-based solution to the MaxEnt model with non-linear continuous weighting functions and illustrate that the optimization problem can be converted into a standard log-linear model at a higher-dimensional space. The empirical results on two classification tasks that contain continuous features are reported. The results confirm our insight and show that our proposed solution consistently outperforms the MaxEnt-MC model and the bucketing approach with significant margins.     

On estimation of a heteroscedastic measurement error model under heavy-tailed distributions

It is common in epidemiology and other fields that the analyzing data is collected with error-prone observations and the variances of the measurement errors change across observations. Heteroscedastic measurement error (HME) models have been developed for such data. This paper extends the structural HME model to situations in which the observations jointly follow scale mixtures of normal (SMN) distribution. We develop the EM algorithm to compute the maximum likelihood estimates for the model with and without equation error respectively, and derive closed forms of asymptotic variances. We also conduct simulations to verify the effective of the EM estimates and confirm their robust behaviors based on heavy-tailed SMN distributions. A practical application is reported for the data from the WHO MONICA Project on cardiovascular disease.     

Faster ARMA maximum likelihood estimation

A new likelihood based AR approximation is given for ARMA models. The usual algorithms for the computation of the likelihood of an ARMA model require O(n) flops per function evaluation. Using our new approximation, an algorithm is developed which requires only O(1) flops in repeated likelihood evaluations. In most cases, the new algorithm gives results identical to or very close to the exact maximum likelihood estimate (MLE). This algorithm is easily implemented in high level quantitative programming environments (QPEs) such as Mathematica, MatLab and R. In order to obtain reasonable speed, previous ARMA maximum likelihood algorithms are usually implemented in C or some other machine efficient language. With our algorithm it is easy to do maximum likelihood estimation for long time series directly in the QPE of your choice. The new algorithm is extended to obtain the MLE for the mean parameter. Simulation experiments which illustrate the effectiveness of the new algorithm are discussed. Mathematica and R packages which implement the algorithm discussed in this paper are available [McLeod, A.I., Zhang, Y., 2007. Online supplements to “Faster ARMA Maximum Likelihood Estimation”, 〈http://www.stats.uwo.ca/faculty/aim/2007/faster/〉]. Based on these package implementations, it is expected that the interested researcher would be able to implement this algorithm in other QPEs.     

New measures of weighted fuzzy entropy and their applications for the study of maximum weighted fuzzy entropy principle

Keeping in view the non-probabilistic nature of experiments, two new measures of weighted fuzzy entropy have been introduced and to check their authenticity, the essential properties of these measures have been studied. Under the fact that measures of entropy can be used for the study of optimization principles when certain partial information is available, we have applied the existing as well as the newly introduced weighted measures of fuzzy entropy to study the maximum entropy principle.     

极大熵自适应微粒群混合算法求解绝对值方程*

绝对值方程Ax-|x|=b是一个不可微的NP-hard问题。在假设矩阵A的奇异值大于1（这里矩阵A的奇异值定义为矩阵ATA特征值的非负平方根）时,给出了求解绝对值方程的一个新算法。通过引进一种极大熵函数把绝对值方程进行光滑化处理,再引入适当的目标函数,从而把绝对值方程问题转换为无约束优化问题,然后利用自适应微粒群算法对其进行求解。数值实验结果表明了该方法的正确性和有效性。     

Application of maximum entropy principle for reliability-based design optimization

The maximum entropy principle (MEP) is used to generate a natural probability distribution among the many possible that have the same moment conditions. The MEP can accommodate higher order moment information and therefore facilitate a higher quality PDF model. The performance of the MEP for PDF estimation is studied by using more than four moments. For the case with four moments, the results are compared with those by the Pearson system. It is observed that as accommodating higher order moment, the estimated PDF converges to the original one. A sensitivity analysis formulation of the failure probability based on the MEP is derived for reliability-based design optimization (RBDO) and the accuracy is compared with that by finite difference method (FDM). Two RBDO examples including a realistic three-dimensional wing design are solved by using the derived sensitivity formula and the MEP-based moment method. The results are compared with other methods such as TR-SQP, FAMM + Pearson system, FFMM + Pearson system in terms of accuracy and efficiency. It is also shown that an improvement in the accuracy by including more moment terms can increase numerical efficiency of optimization for the three-dimensional wing design. The moment method equipped with the MEP is found flexible and well adoptable for reliability analysis and design.     

A maximum entropy type test of fit

In this paper, we propose a test of fit based on maximum entropy. The asymptotic distribution of the proposed test statistic is established and a corrected form for small and medium sample sizes is furnished. The performance of the test is investigated through extensive Monte Carlo simulations. Real examples are also presented and analyzed.     

On the estimation of the linear relation when the error variances are known

The problem of consistent estimation in measurement error models in a linear relation with not necessarily normally distributed measurement errors is considered. Three possible estimators which are constructed as different combinations of the estimators arising from direct and inverse regression are considered. The efficiency properties of these three estimators are derived and the effect of non-normally distributed measurement errors is analyzed. A Monte-Carlo experiment is conducted to study the performance of these estimators in finite samples.     

Exact maximum likelihood estimation of partially nonstationary vector ARMA models

A useful class of partially nonstationary vector autoregressive moving average (VARMA) models is considered with regard to parameter estimation. An exact maximum likelihood (EML) approach is developed on the basis of a simple transformation applied to the error-correction representation of the models considered. The employed transformation is shown to provide a standard VARMA model with the important property that it is stationary. Parameter estimation can thus be carried out by applying standard EML methods to the stationary VARMA model obtained from the error-correction representation. This approach resolves at least two problems related to the current limited availability of EML estimation methods for partially nonstationary VARMA models. Firstly, it resolves the apparent impossibility of computing the exact log-likelihood for such models using currently available methods. And secondly, it resolves the inadequacy of considering lagged endogenous variables as exogenous variables in the error-correction representation. Theoretical discussion is followed by an example using a popular data set. The example illustrates the feasibility of the EML estimation approach as well as some of its potential benefits in cases of practical interest which are easy to come across. As in the case of stationary models, the proposed EML method provides estimated model structures that are more reliable and accurate than results produced by conditional methods.     

Intelligent-guided adaptive search for the maximum covering location problem

Computational intelligence techniques are part of the search process in several recent heuristics. One of their main benefits is the use of an adaptive memory to guide the search towards regions with promising solutions. This paper follows this approach proposing a variation of a well-known iteration independent metaheuristic. This variation adds a learning stage to the search process, which can improve the quality of the solutions found. The proposed metaheuristic, named Intelligent-Guided Adaptive Search (IGAS), provides an efficient solution to the maximum covering facility location problem. Computational experiments conducted by the authors showed that the solutions found by IGAS were better than the solutions obtained by popular methods found in the literature.     

高维数据固有维数的自适应极大似然估计

如何确定高维数据的固有维数是降维成功与否的关键。基于极大似然估计（MLE）的维数估计方法是一种新近出现的方法,实现简单,选择合适的近邻能取得不错的结果。但当近邻数过小或过大时,均有比较明显的偏差。其根本原因是没有考虑每个点对固有维数的不同贡献。在充分考虑数据集的分布信息之后,提出了一种改进的MLE——自适应极大似然估计(AMLE)。实验表明,无论在合成数据集还是真实数据集上,AMLE较MLE在估计准确度上均有很大的提高,对近邻数的变化也不甚敏感。     

An approach for constructing complex discriminating surfaces based on Bayesian interference of the maximum entropy

In this paper we present a comprehensive Maximum Entropy (MaxEnt) procedure for the classification tasks. This MaxEnt is applied successfully to the problem of estimating the probability distribution function (pdf) of a class with a specific pattern, which is viewed as a probabilistic model handling the classification task. We propose an efficient algorithm allowing to construct a non-linear discriminating surfaces using the MaxEnt procedure. The experiments that we carried out shows the performance and the various advantages of our approach.     

Estimating the width of a uniform distribution when data are measured with additive normal errors with known variance

The problem of estimating the width of the symmetric uniform distribution on the line when data are measured with normal additive error is considered. The main purpose is to discuss the efficiency of the maximum likelihood estimator and the moment method estimator. It is shown that the model is regular and that the maximum likelihood estimator is more efficient than the moment method estimator. A sufficient condition is also given for the existence of both estimators.     

On the equivalence of time and frequency domain maximum likelihood estimation

Maximum likelihood estimation has a rich history. It has been successfully applied to many problems including dynamical system identification. Different approaches have been proposed in the time and frequency domains. In this paper we discuss the relationship between these approaches and we establish conditions under which the different formulations are equivalent for finite length data. A key point in this context is how initial (and final) conditions are considered and how they are introduced in the likelihood function.     

自适应模糊神经网络预测阳离子标准熵的研究

依据独立共同可别粒子体系的熵与配分函数的关系,采用自适应模糊神经网络的方法,以元素原子量和其电子层数为参数,关联阳离子标准熵。利用减法聚类算法确定模糊神经网络的结构,并结合模糊推理系统调整网络参数,仿真的结果令人满意。成功地关联了固体化合物中70种阳离子的标准熵。在此基础上,预报目前尚缺的17种阳离子的标准熵。自适应模糊神经网络可望成为研究元素和化合物构效关系的辅助手段。     

基于极大后验估计的自适应容积卡尔曼滤波器

针对标准的容积卡尔曼滤波器(CKF) 设计需要精确已知噪声先验统计知识的问题, 提出一种自适应CKF 算法. 该算法在滤波过程中, 利用Sage-Husa 极大后验估值器对噪声的统计特性进行在线估计和修正, 有效地提高了CKF 的估计精度和数值稳定性. 在某些情况下, 噪声协方差估计会出现异常现象使得滤波发散, 进而提出了相应的改进方法. 仿真结果表明了自适应CKF 算法的可行性和有效性, 且明显改善了标准CKF 算法的滤波效果.     

H-R自适应边界元的模糊误差估计方法

为克服在利用传统自适应边界元法求解弹性问题时需对不同问题设计不同误差估计公式的缺点,以专家经验为基础,利用模糊逻辑理论,提出一种新的误差分析方法.基于H-R自适应边界元法,用FORTRAN编写求解2个经典平面弹性静力学问题的程序.分析表明该误差分析方法能较好地估计边界元解的误差.     

Design of a complex system based on the maximum entropy principle

A complex system is a system composed of many dynamic elements with mutual interactions. This paper proposes a unified approach for the design of an information processing system using a complex system. The method of design is based on the maximum entropy principle. After a detailed explanation, the proposed method is applied to the design of a spatial filter using a complex system. This work was presented, in part, at the International Symposium on Artificial Life and Robotics, Oita, Japan, February 18–20, 1996.     

On the expressive power of the relational algebra with partially ordered domains

Assuming data domains are partially ordered, we define the partially ordered relational algebra (PORA) by allowing the ordering predicate ? to be used in formulae of the selection operator σ. We apply Paredaens and Bancilhon's Theorem to examine the expressiveness of the PORA, and show that the PORA expresses exactly the set of all possible relations which are invariant under order-preserving automorphisms of databases. The extension is consistent with the two important extreme cases of unordered and linearly ordered domains. We also investigate the three hierarchies of: (1) computable queries, (2) query languages and (3) partially ordered domains, and show that there is a one-to-one correspondence between them.     

On transient performance improvement of adaptive control architectures

While adaptive control theory has been used in numerous applications to achieve given system stabilisation or command following criteria without excessive reliance on mathematical models, the ability to obtain a predictable transient performance is still an important problem – especially for applications to safety-critical systems and when there is no a-priori knowledge on upper bounds of existing system uncertainties. To address this problem, we present a new approach to improve the transient performance of adaptive control architectures. In particular, our approach is predicated on a novel controller architecture, which involves added terms in the update law entitled artificial basis functions. These terms are constructed through a gradient optimisation procedure to minimise the system error between an uncertain dynamical system and a given reference model during the learning phase of an adaptive controller. We provide a detailed stability analysis of the proposed approach, discuss the practical aspects of its implementation, and illustrate its efficacy on a numerical example.