Parameter estimation of Poisson mixture with automated model selection through BYY harmony learning

Finite mixture is widely used in the fields of information processing and data analysis. However, its model selection, i.e., the selection of components in the mixture for a given sample data set, has been still a rather difficult task. Recently, the Bayesian Ying-Yang (BYY) harmony learning has provided a new approach to the Gaussian mixture modeling with a favorite feature that model selection can be made automatically during parameter learning. In this paper, based on the same BYY harmony learning framework for finite mixture, we propose an adaptive gradient BYY learning algorithm for Poisson mixture with automated model selection. It is demonstrated well by the simulation experiments that this adaptive gradient BYY learning algorithm can automatically determine the number of actual Poisson components for a sample data set, with a good estimation of the parameters in the original or true mixture where the components are separated in a certain degree. Moreover, the adaptive gradient BYY learning algorithm is successfully applied to texture classification.     

The BYY annealing learning algorithm for Gaussian mixture with automated model selection

Bayesian Ying-Yang (BYY) learning has provided a new mechanism that makes parameter learning with automated model selection via maximizing a harmony function on a backward architecture of the BYY system for the Gaussian mixture. However, since there are a large number of local maxima for the harmony function, any local searching algorithm, such as the hard-cut EM algorithm, does not work well. In order to overcome this difficulty, we propose a simulated annealing learning algorithm to search the global maximum of the harmony function, being expressed as a kind of deterministic annealing EM procedure. It is demonstrated by the simulation experiments that this BYY annealing learning algorithm can efficiently and automatically determine the number of clusters or Gaussians during the learning process. Moreover, the BYY annealing learning algorithm is successfully applied to two real-life data sets, including Iris data classification and unsupervised color image segmentation.     

Entropy Regularized Likelihood Learning on Gaussian Mixture: Two Gradient Implementations for Automatic Model Selection

In Gaussian mixture modeling, it is crucial to select the number of Gaussians or mixture model for a sample data set. Under regularization theory, we aim to solve this kind of model selection problem through implementing entropy regularized likelihood (ERL) learning on Gaussian mixture via a batch gradient learning algorithm. It is demonstrated by the simulation experiments that this gradient ERL learning algorithm can select an appropriate number of Gaussians automatically during the parameter learning on a sample data set and lead to a good estimation of the parameters in the actual Gaussian mixture, even in the cases of two or more actual Gaussians overlapped strongly. We further give an adaptive gradient implementation of the ERL learning on Gaussian mixture followed with theoretic analysis, and find a mechanism of generalized competitive learning implied in the ERL learning.     

BYY harmony learning, independent state space, and generalized APTfinancial analyses

First, the relationship between factor analysis (FA) and the well-known arbitrage pricing theory (APT) for financial market is discussed comparatively, with a number of to-be-improved problems listed. An overview is made from a unified perspective on the related studies in the literatures of statistics, control theory, signal processing, and neural networks. Next, we introduce the fundamentals of the Bayesian Ying Yang (BYY) system and the harmony learning principle. We further show that a specific case of the framework, called BYY independent state space (ISS) system, provides a general guide for systematically tackling various FA related learning tasks and the above to-be-improved problems for the APT analyses. Third, on various specific cases of the BYY ISS system in three typical architectures, adaptive algorithms, regularization methods and model selection criteria are provided for either or both of parameter learning with automated model selection and parameter learning followed by model selection. Finally, we introduce some other financial applications that are based on the underlying independent factors via the APT analyses.     

基于马尔可夫混合模型的电子商务搜索引擎用户行为聚类

对搜索引擎用户行为进行聚类分析有利于为用户提供个性化的服务。为了能准确地刻画用户行为的动态性,提出利用马尔可夫混合模型,对电子商务搜索引擎的用户行为模式聚类。模型假设每一类用户行为可表示为一个马尔可夫模型,当用户使用搜索引擎时,每个用户以一定的概率属于某一聚类;该用户的行为序列,由对应的马尔可夫模型产生。同时,为了解决参数估计和模型自动选择的问题,将贝叶斯阴阳和谐学习理论应用于该混合模型,提出针对该模型的和谐度函数及自适应梯度算法。仿真实验结果表明,与传统的最大期望（EM）算法相比,基于贝叶斯阴阳机的自适应梯度算法能更高效和准确地同时进行参数学习和模型选择。最后,将所提出的聚类方法应用于真实的电子商务搜索引擎点击日志,初步验证了本模型的有效性。     

Temporal BYY encoding, Markovian state spaces, and space dimension determination.

As a complementary to those temporal coding approaches of the current major stream, this paper aims at the Markovian state space temporal models from the perspective of the temporal Bayesian Ying-Yang (BYY) learning with both new insights and new results on not only the discrete state featured Hidden Markov model and extensions but also the continuous state featured linear state spaces and extensions, especially with a new learning mechanism that makes selection of the state number or the dimension of state space either automatically during adaptive learning or subsequently after learning via model selection criteria obtained from this mechanism. Experiments are demonstrated to show how the proposed approach works.     

An Overview and Perspectives On Bidirectional Intelligence: Lmser Duality,Double IA Harmony,and Causal Computation

Advances on bidirectional intelligence are overviewed along three threads, with extensions and new perspectives. The first thread is about bidirectional learning architecture, exploring five dualities that enable Lmser six cognitive functions and provide new perspectives on which a lot of extensions and particularlly flexible Lmser are proposed. Interestingly, either or two of these dualities actually takes an important role in recent models such as U-net, ResNet, and DenseNet. The second thread is about bidirectional learning principles unified by best yIng-yAng (IA) harmony in BYY system. After getting insights on deep bidirectional learning from a bird-viewing on existing typical learning principles from one or both of the inward and outward directions, maximum likelihood, variational principle, and several other learning principles are summarised as exemplars of the BYY learning, with new perspectives on advanced topics. The third thread further proceeds to deep bidirectional intelligence, driven by long term dynamics (LTD) for parameter learning and short term dynamics (STD) for image thinking and rational thinking in harmony. Image thinking deals with information flow of continuously valued arrays and especially image sequence, as if thinking was displayed in the real world, exemplified by the flow from inward encoding/cognition to outward reconstruction/transformation performed in Lmser learning and BYY learning. In contrast, rational thinking handles symbolic strings or discretely valued vectors, performing uncertainty reasoning and problem solving. In particular, a general thesis is proposed for bidirectional intelligence, featured by BYY intelligence potential theory (BYY-IPT) and nine essential dualities in architecture, fundamentals, and implementation, respectively. Then, problems of combinatorial solving and uncertainty reasoning are investigated from this BYY IPT perspective. First, variants and extensions are suggested for AlphaGoZero like searching tasks, such as traveling salesman problem (TSP) and attributed graph matching (AGM) that are turned into Go like problems with help of a feature enrichment technique. Second, reasoning activities are summarized under guidance of BYY IPT from the aspects of constraint satisfaction, uncertainty propagation, and path or tree searching. Particularly, causal potential theory is proposed for discovering causal direction, with two roads developed for its implementation.      

A Semi-supervised Learning Algorithm on Gaussian Mixture with Automatic Model Selection

In Gaussian mixture modeling, it is crucial to select the number of Gaussians for a sample set, which becomes much more difficult when the overlap in the mixture is larger. Under regularization theory, we aim to solve this problem using a semi-supervised learning algorithm through incorporating pairwise constraints into entropy regularized likelihood (ERL) learning which can make automatic model selection for Gaussian mixture. The simulation experiments further demonstrate that the presented semi-supervised learning algorithm (i.e., the constrained ERL learning algorithm) can automatically detect the number of Gaussians with a good parameter estimation, even when two or more actual Gaussians in the mixture are overlapped at a high degree. Moreover, the constrained ERL learning algorithm leads to some promising results when applied to iris data classification and image database categorization.     

全局竞争和声搜索算法

提出一种全局竞争和声搜索(GCHS) 算法, 给出随机局部平均和声和全局平均和声的概念, 建立竞争搜索机制, 实现每次迭代产生两个和声向量并进行竞争选择. 设计自适应全局调整和局部学习策略, 平衡算法的局部搜索和全局搜索, 详细分析参数HMS、HMCR和PAR对算法优化性能的影响. 数值结果表明, GCHS 算法在精度、收敛速度和鲁棒性方面比和声搜索算法及最近文献中提出的7 种优秀改进和声搜索算法要好.

     

Gaussian mixture model learning based image denoising method with adaptive regularization parameters

Gaussian mixture model learning based image denoising as a kind of structured sparse representation method has received much attention in recent years. In this paper, for further enhancing the denoised performance, we attempt to incorporate the gradient fidelity term with the Gaussian mixture model learning based image denoising method to preserve more fine structures of images. Moreover, we construct an adaptive regularization parameter selection scheme by combing the image gradient with the local entropy of the image. Experiment results show that our proposed method performs an improvement both in visual effects and peak signal to noise values.     

Cluster number selection for a small set of samples using theBayesian Ying-Yang model

One major problem in cluster analysis is the determination of the number of clusters. In this paper, we describe both theoretical and experimental results in determining the cluster number for a small set of samples using the Bayesian-Kullback Ying-Yang (BYY) model selection criterion. Under the second-order approximation, we derive a new equation for estimating the smoothing parameter in the cost function. Finally, we propose a gradient descent smoothing parameter estimation approach that avoids complicated integration procedure and gives the same optimal result.     

基于和声搜索算法的无线传感器网络多重连通覆盖

连通与覆盖是传感器网络的重要问题,对传感器网络的服务质量有重要影响。对给定候选位置中选择最少数量的位置放置传感器节点来保证监测目标的多重覆盖和传感器节点之间的多重连通问题进行研究,提出一种基于改进和声搜索算法的节点部署策略。算法以放置节点的数量、监测目标的多重覆盖和节点的多重连通为优化目标,在和声搜索算法中一方面加入学习自动机增强算法参数的自适应性,另一方面通过对算法求解过程中优秀解的再利用,增强了算法的优化效率。为了对比算法性能,提出了一种基于贪婪算法的节点部署策略。仿真结果显示,提出的改进和声搜索算法优于提出的贪婪算法和原始和声搜索算法,证明了算法的有效性。     

改进CNN的多通道语义合成情感分类模型研究

为了解决传统神经网络由于上下文语序变化而导致的情感歧义问题,提出一种多通道语义合成卷积神经网络（SFCNN）。使用改进的情感倾向注意力机制对词向量进行情感加权操作;将情感倾向词向量进行多通道语义合成,生成带有文本上下文语义信息的深度语义向量,构建情感分类模型;使用自适应学习率的梯度下降算法对模型参数进行优化,完成行情感分类任务。为了验证改进算法的有效性,使用多种微博数据样本集在提出的模型上进行对比实验。实验结果表明,改进的情感倾向注意力机制结合多通道语义合成卷积神经网络具有较好的情感分类能力,并且自适应学习率的梯度下降算法可以更快地完成模型收敛工作。     

Improved Harmony Search Algorithm: LHS

In this paper, we propose an improved harmony search algorithm named LHS with three key features: (i) adaptive global pitch adjustment is designed to enhance the exploitation ability of solution space; (ii) opposition-based learning technique is blended to increase the diversity of solution; (iii) competition selection mechanism is established to improve solution precision and enhance the ability of escaping local optima. The performance of the LHS algorithm with respect to harmony memory size (HMS) and harmony memory considering rate (HMCR) are also analyzed in detail. To further evaluate the performance of the proposed LHS algorithm, comparison with ten state-of-the-art harmony search variants over a large number of benchmark functions with different characteristics is carried out. The numerical results confirm the superiority of the proposed LHS algorithm in terms of accuracy, convergence speed and robustness.     

策略梯度的超启发算法求解带容量约束车辆路径问题

有容量车辆路径问题是组合优化问题中比较热门的问题, 它属于经典的NP-hard问题并且时间复杂度高.本文提出了一种基于策略梯度的超启发算法, 将强化学习中的确定性策略梯度算法引入到超启发算法的高层策略中的底层算法选择策略, 确定性策略梯度算法采用Actor-Critic框架, 另外为了能够在后续计算和神经网络参数更新中引用历史经验数据, 在确定性策略梯度算法中设计了经验池用于存储状态转移数据. 在超启发算法解的接受准则方面, 文中通过实验对比了3种接受准则的效果, 最终选择了自适应接受准则作为高层策略中解的接受准则. 通过对有容量车辆路径问题标准算例的计算, 并将求解结果与其他算法对比, 验证了所提算法在该问题求解上的有效性和稳定性.     

极限学习机在岩性识别中的应用

基于传统支持向量机(SVM)训练速度慢、参数选择难等问题,提出了基于极限学习机(ELM)的岩性识别.该算法是一种新的单隐层前馈神经网络(SLFNs)学习算法,不但可以简化参数选择过程,而且可以提高网络的训练速度.在确定了最优参数的基础上,建立了ELM的岩性分类模型,并且将ELM的分类结果与SVM进行对比.实验结果表明,ELM以较少的神经元个数获得与SVM相当的分类正确率,并且ELM参数选择比SVM简便,有效降低了训练速度,表明了ELM应用于岩性识别的可行性和算法的有效性.     

Competitive EM algorithm for finite mixture models

In this paper, we present a novel competitive EM (CEM) algorithm for finite mixture models to overcome the two main drawbacks of the EM algorithm: often getting trapped at local maxima and sometimes converging to the boundary of the parameter space. The proposed algorithm is capable of automatically choosing the clustering number and selecting the “split” or “merge” operations efficiently based on the new competitive mechanism we propose. It is insensitive to the initial configuration of the mixture component number and model parameters.Experiments on synthetic data show that our algorithm has very promising performance for the parameter estimation of mixture models. The algorithm is also applied to the structure analysis of complicated Chinese characters. The results show that the proposed algorithm performs much better than previous methods with slightly heavier computation burden.     

基于自适应动量优化算法的正则化极限学习机

针对极限学习机(extreme learning machine,ELM)隐节点不确定性导致的系统不稳定,以及对大型数据计算负担过重的问题,提出了基于自适应动量优化算法(adaptive and momentum method,AdaMom)的正则化极限学习机.算法主要思想是构造连续可微的目标函数,在梯度下降过程中计算自适应学习率,求自适应学习率与梯度乘积的指数加权平均值,通过迭代得到损失函数最小值对应的隐层输出权重矩阵.实验结果表明,在相同基准数据集的训练中,AdaMom-ELM算法具有非常良好的泛化性能和鲁棒性,提高了计算效率.     

An efficient compensatory neuro-fuzzy system and its applications

In this study, a compensatory neuro-fuzzy system (CNFS) is proposed. The compensatory fuzzy reasoning method uses adaptive fuzzy operations of a neuro-fuzzy system to make the fuzzy logic system more adaptive and effective. Furthermore, an online learning algorithm that consists of structure learning and parameter learning is proposed to automatically construct the CNFS. The structure learning is based on the fuzzy similarity measure to determine the number of fuzzy rules, and the parameter learning is based on backpropagation algorithm to adjust the parameters. The simulation results have shown that (1) the CNFS model converges quickly and (2) the CNFS model has a lower root mean square (RMS) error than other models.     

一种基于参数辨识的迭代学习控制算法及应用

针对间歇过程中模型参数变化的问题,提出了一种基于遗忘因子最小二乘法辨识的迭代学习控制算法。迭代学习律的参数随模型参数变化而更新,利用遗忘因子大大减小参数变化时"错误"数据对算法的影响,使算法具有更强的自适应性。把这一算法应用于黄酒发酵过程,提高了发酵过程的优化控制效果。仿真结果表明当模型参数随着批次变化时,系统的跟踪性能得到了改进。