Nonlinear system identification using memetic differential evolution trained neural networks

Several gradient-based approaches such as back propagation (BP) and Levenberg Marquardt (LM) methods have been developed for training the neural network (NN) based systems. But, for multimodal cost functions these procedures may lead to local minima, therefore, the evolutionary algorithms (EAs) based procedures are considered as promising alternatives. In this paper we focus on a memetic algorithm based approach for training the multilayer perceptron NN applied to nonlinear system identification. The proposed memetic algorithm is an alternative to gradient search methods, such as back-propagation and back-propagation with momentum which has inherent limitations of many local optima. Here we have proposed the identification of a nonlinear system using memetic differential evolution (DE) algorithm and compared the results with other six algorithms such as Back-propagation (BP), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Differential Evolution (DE), Genetic Algorithm Back-propagation (GABP), Particle Swarm Optimization combined with Back-propagation (PSOBP). In the proposed system identification scheme, we have exploited DE to be hybridized with the back propagation algorithm, i.e. differential evolution back-propagation (DEBP) where the local search BP algorithm is used as an operator to DE. These algorithms have been tested on a standard benchmark problem for nonlinear system identification to prove their efficacy. First examples shows the comparison of different algorithms which proves that the proposed DEBP is having better identification capability in comparison to other. In example 2 good behavior of the identification method is tested on an one degree of freedom (1DOF) experimental aerodynamic test rig, a twin rotor multi-input-multi-output system (TRMS), finally it is applied to Box and Jenkins Gas furnace benchmark identification problem and its efficacy has been tested through correlation analysis.     

The crowding approach to niching in genetic algorithms

A wide range of niching techniques have been investigated in evolutionary and genetic algorithms. In this article, we focus on niching using crowding techniques in the context of what we call local tournament algorithms. In addition to deterministic and probabilistic crowding, the family of local tournament algorithms includes the Metropolis algorithm, simulated annealing, restricted tournament selection, and parallel recombinative simulated annealing. We describe an algorithmic and analytical framework which is applicable to a wide range of crowding algorithms. As an example of utilizing this framework, we present and analyze the probabilistic crowding niching algorithm. Like the closely related deterministic crowding approach, probabilistic crowding is fast, simple, and requires no parameters beyond those of classical genetic algorithms. In probabilistic crowding, subpopulations are maintained reliably, and we show that it is possible to analyze and predict how this maintenance takes place. We also provide novel results for deterministic crowding, show how different crowding replacement rules can be combined in portfolios, and discuss population sizing. Our analysis is backed up by experiments that further increase the understanding of probabilistic crowding.     

基于概率无向图模型的近邻传播聚类算法

针对近邻传播聚类算法偏向参数难选定、生成的簇数目偏多等问题,提出一种概率无向图模型的近邻传播聚类算法.首先为样本数据构建概率无向图模型,利用极大团和势函数计算无向图中数据样本的概率密度,将此概率密度作为一种聚类先验知识注入近邻传播算法的偏向参数中,提高算法的聚类效率;并用高斯降噪和簇归并方法进一步提升算法的聚类精度.在UCI数据集上的实验结果表明,所提出算法的聚类效率和精度均优于相比较的同类算法.     

Variations over the message computation algorithm of lazy propagation.

Improving the performance of belief updating becomes increasingly important as real-world Bayesian networks continue to grow larger and more complex. In this paper, an investigation is done on how variations over the message-computation algorithm of lazy propagation may impact its performance. Lazy propagation is a junction-tree-based inference algorithm for belief updating in Bayesian networks. Lazy propagation combines variable elimination (VE) with a Shenoy-Shafer message-passing scheme in an attempt to exploit the independence properties induced by evidence in a junction-tree-based algorithm. The authors investigate, the use of arc reversal (AR) and symbolic probabilistic inference (SPI) as alternative algorithms for computing clique-to-clique messages in lazy propagation. The paper presents the results of an empirical evaluation of the performance of lazy propagation using AR, SPI, and VE as the message-computation algorithm. The results of the empirical evaluation show that no single algorithm outperforms or is outperformed by the other two alternatives. In many cases, there is no significant difference in the performance of the three algorithms.     

A novel approach in parameter adaptation and diversity maintenance for genetic algorithms

In this paper, we propose a probabilistic rule-driven adaptive model (PRAM) for parameter adaptation and a repelling approach for diversity maintenance in genetic algorithms. PRAM uses three parameter values and a set of greedy rules to adapt the value of the control parameters automatically. The repelling algorithm is proposed to maintain the population diversity. It modifies the fitness value to increase the survival opportunity of chromosomes with rare alleles. The computation overheads of repelling are reduced by the lazy repelling algorithm, which decreases the frequency of the diversity fitness evaluations. From experiments with commonly used benchmark functions, it is found that the PRAM and repelling techniques outperform other approaches on both solution quality and efficiency.     

Application placement in computer clustering in software as a service (SaaS) networks

One major service provided by cloud computing is Software as a Service (SaaS). As competition in the SaaS market intensifies, it becomes imperative for a SaaS provider to design and configure its computing system properly. This paper studies the application placement problem encountered in computer clustering in SaaS networks. This problem involves deciding which software applications to install on each computer cluster of the provider and how to assign customers to the clusters in order to minimize total cost. Given the complexity of the problem, we propose two algorithms to solve it. The first one is a probabilistic greedy algorithm which includes randomization and perturbation features to avoid getting trapped in a local optimum. The second algorithm is based on a reformulation of the problem where each cluster is to be assigned an application configuration from a properly generated subset of configurations. We conducted an extensive computational study using large data sets with up to 300 customers and 50 applications. The results show that both algorithms outperform a standard branch-and-bound procedure for problem instances with large sizes. The probabilistic greedy algorithm is shown to be the most efficient in solving the problem.     

Polynomial-Time Probabilistic Controllability Analysis of Discrete-Time Piecewise Affine Systems

This paper proposes a probabilistic approach to the controllability analysis for discrete-time piecewise affine (PWA) systems. Three kinds of randomized algorithms, which are based on random sampling of the mode sequence and/or the initial state, for determining with a probabilistic accuracy if the system is controllable are presented: a positive one-sided error algorithm, a negative one-sided error algorithm, and a two-sided error algorithm. It is proven that these are polynomial-time algorithms with respect to several variables of the problem. It is also shown with some examples, for which it is hopeless to check the controllability in a deterministic way, that these algorithms are efficient.     

Energy optimal data propagation in wireless sensor networks

We propose an algorithm to compute the optimal parameters of a probabilistic data propagation algorithm for wireless sensor networks (WSN). The probabilistic data propagation algorithm we consider was introduced in previous work, and it is known that this algorithm, when used with adequate parameters, balances the energy consumption and increases the lifespan of the WSN. However, we show that in the general case achieving energy balance may not be possible. We propose a centralized algorithm to compute the optimal parameters of the probabilistic data propagation algorithm, and prove that these parameters maximize the lifespan of the network even when it is not possible to achieve energy balance. Compared to previous work, our contribution is the following: (a) we give a formal definition of an optimal data propagation algorithm: an algorithm maximizing the lifespan of the network. (b) We find a simple necessary and sufficient condition for the data propagation algorithm to be optimal. (c) We constructively prove that there exists a choice of parameters optimizing the probabilistic data propagation algorithm. (d) We provide a centralized algorithm to compute these optimal parameters, thus enabling their use in a WSN. (e) We extend previous work by considering the energy consumption per sensor, instead of the consumption per slice, and propose a spreading technique to balance the energy among sensors of a same slice. The technique is numerically validated by simulating a WSN accomplishing a data monitoring task and propagating data using the probabilistic data propagation algorithm with optimal parameters.     

A class of iterative refined Max-sum algorithms via non-consecutive value propagation strategies

As an important technique to solve distributed constraint optimization problems, Max-sum has drawn a lot of attention and successfully been deployed in real applications. Unfortunately, Max-sum fails to converge in cyclic problems and usually traverses states with low quality. Max-sum_AD and Max-sum_ADVP were proposed to guarantee the single phase convergence and the cross phase convergence respectively, and greatly improve the solution quality of Max-sum. However, the solution quality is closely related to the timing for starting value propagation in Max-sum_ADVP. In other words, low-quality initial assignments will lead to a poor result. In this paper, we prove that value propagation could restrict the exploration ability brought by Max-sum and eventually makes Max-sum_ADVP equivalent to a sequential greedy local search algorithm. For getting a balance between exploration and exploitation, several non-consecutive value propagation strategies are proposed to relax the restriction caused by value propagation: single-side value propagation which executes value propagation and Max-sum_AD in an interleaved way, probabilistic value propagation which performs value propagation stochastically and hybrid belief/value propagation where agents perform Max-sum_AD and value propagation in one round. We illustrate that agents in our algorithms can make decisions beyond local functions. Our empirical evaluations demonstrate the superiority of our methods over Max-sum and its variants. It also can be found that our methods are independent of the value propagation timing which is a major concern in Max-sum_ADVP.     

基于混合概率模型的无监督离散化算法

现实应用中常常涉及许多连续的数值属性，而且前许多机器学习算法则要求所处理的属性取离散值，根据在对数值属性的离散化过程中，是否考虑相关类别属性的值，离散化算法可分为有监督算法和无监督算法两类。基于混合概率模型，该文提出了一种理论严格的无监督离散化算法，它能够在无先验知识，无类别是属性的前提下，将数值属性的值域划分为若干子区间，再通过贝叶斯信息准则自动地寻求最佳的子区间数目和区间划分方法。     

Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling

This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg–Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.      

基于多隐层Gibbs采样的深度信念网络训练方法

深度信念网络（Deep belief network，DBN）作为一类非常重要的概率生成模型，在多个领域都有着广泛的用途.现有深度信念网的训练分为两个阶段，首先是对受限玻尔兹曼机（Restricted Boltzmann machine，RBM）层自底向上逐层进行的贪婪预训练，使得每层的重构误差最小，这个阶段是无监督的；随后再对整体的权值使用有监督的反向传播方法进行精调.本文提出了一种新的DBN训练方法，通过多隐层的Gibbs采样，将局部RBM层组合，并在原有的逐层预训练和整体精调之间进行额外的预训练，有效地提高了DBN的精度.本文同时比较了多种隐层的组合方式，在MNIST和ShapeSet以及Cifar10数据集上的实验表明，使用两两嵌套组合方式比传统的方法错误率更低.新的训练方法可以在更少的神经元上获得比以往的训练方法更好的准确度，有着更高的算法效率.     

Latent Variable Model for Estimation of Distribution Algorithm Based on a Probabilistic Context-Free Grammar

Estimation of distribution algorithms are evolutionary algorithms using probabilistic techniques instead of traditional genetic operators. Recently, the application of probabilistic techniques to program and function evolution has received increasing attention, and this approach promises to provide a strong alternative to the traditional genetic programming techniques. Although a probabilistic context-free grammar (PCFG) is a widely used model for probabilistic program evolution, a conventional PCFG is not suitable for estimating interactions among nodes because of the context freedom assumption. In this paper, we have proposed a new evolutionary algorithm named programming with annotated grammar estimation based on a PCFG with latent annotations, which allows this context freedom assumption to be weakened. By applying the proposed algorithm to several computational problems, it is demonstrated that our approach is markedly more effective at estimating building blocks than prior approaches.      

A constrained, globalized, and bounded Nelder–Mead method for engineering optimization

One of the fundamental difficulties in engineering design is the multiplicity of local solutions. This has triggered much effort in the development of global search algorithms. Globality, however, often has a prohibitively high numerical cost for real problems. A fixed cost local search, which sequentially becomes global, is developed in this work. Globalization is achieved by probabilistic restarts. A spacial probability of starting a local search is built based on past searches. An improved Nelder–Mead algorithm is the local optimizer. It accounts for variable bounds and nonlinear inequality constraints. It is additionally made more robust by reinitializing degenerated simplexes. The resulting method, called the Globalized Bounded Nelder–Mead (GBNM) algorithm, is particularly adapted to tackling multimodal, discontinuous, constrained optimization problems, for which it is uncertain that a global optimization can be afforded. Numerical experiments are given on two analytical test functions and two composite laminate design problems. The GBNM method compares favorably with an evolutionary algorithm, both in terms of numerical cost and accuracy.     

最大受限路径相容约束传播算法的研究进展

约束传播技术对于约束满足问题的求解性能至关重要。约束传播技术在一个预处理过程中能彻底地移除一些局部不相容值,或者在搜索期间高效地剪枝搜索树。最大受限路径相容算法(max Restricted Path Consistency,maxRPC)是最近提出的一种强相容性约束传播算法,它能够删除更多不相容值,在解决复杂问题中取得了很好的效果。文中对弧相容算法AC和最大受限路径相容算法maxRPC的相关算法AC3,AC3rm,maxRPC1,maxRPC2,maxRPCrm,maxRPC3等及其相关变体分别进行介绍和比较。在Mistral求解器上的实验测试结果验证了各种算法的性能。     

An unequal packet loss resilience scheme for video over the Internet

We present an unequal packet loss resilience scheme for robust transmission of video over the Internet. By jointly exploiting the unequal importance existing in different levels of syntax hierarchy in video coding schemes, GOP-level and Resynchronization-packet-level Integrated Protection (GRIP) is designed for joint unequal loss protection (ULP) in these two levels using forward error correction (FEC) across packets. Two algorithms are developed to achieve efficient FEC assignment for the proposed GRIP framework: a model-based FEC assignment algorithm and a heuristic FEC assignment algorithm. The model-based FEC assignment algorithm is to achieve optimal allocation of FEC codes based on a simple but effective performance metric, namely distortion-weighted expected length of error propagation, which is adopted to quantify the temporal propagation effect of packet loss on video quality degradation. The heuristic FEC assignment algorithm aims at providing a much simpler yet effective FEC assignment with little computational complexity. The proposed GRIP together with any of the two developed FEC assignment algorithms demonstrates strong robustness against burst packet losses with adaptation to different channel status.     

Estimation-based metaheuristics for the probabilistic traveling salesman problem

The probabilistic traveling salesman problem (PTSP) is a central problem in stochastic routing. Recently, we have shown that empirical estimation is a promising approach to devise highly effective local search algorithms for the PTSP. In this paper, we customize two metaheuristics, an iterated local search algorithm and a memetic algorithm, to solve the PTSP. This customization consists in adopting the estimation approach to evaluate the solution cost, exploiting a recently developed estimation-based local search algorithm, and tuning the metaheuristics parameters. We present an experimental study of the estimation-based metaheuristic algorithms on a number of instance classes. The results show that the proposed algorithms are highly effective and that they define a new state-of-the-art for the PTSP.     

神经纤维跟踪算法研究进展

神经纤维跟踪通过整合纤维局部结构方向信息,可以描绘出具有解剖学意义的空间纤维结构,是扩散磁共振成像的关键步骤,对临床医学与神经科学等有着重大意义。然而,大量的研究和临床应用表明,目前的神经纤维跟踪算法重构出了大量虚假纤维而备受质疑。为了给研究者和临床医生选择神经纤维跟踪算法提供依据,本文深入分析了当前的主要跟踪算法并进行定量评估与定性比较。从确定型、概率型和全局优化等方法详细介绍各典型跟踪算法;利用Fibercup和国际医学磁共振学会（International Society for Magnetic Resonance in Medicine,ISMRM）2015挑战数据进行实验,定量对比9种常用算法的优缺点,并分析了这些算法在实际临床数据的成像结果及其面临的挑战;结合实验结果与算法理论分析各算法的内在联系与区别。不同跟踪算法在效果上有着较大的差异,确定型算法在描绘主要纤维结构上更为明显,概率型算法描绘的纤维分布更为全面,全局优化算法的纤维轨迹更符合全局数据而避免了误差累积问题。纤维跟踪对于分析人脑神经纤维连接具有很高的研究价值和应用价值。不同类型的算法有着各自的优缺点,目前并没有一种跟踪算法可以摒弃其他算法缺点而结合所有优点。另外目前纤维跟踪算法的结果与实际情况均有着一定差距,如何描绘出更为精确的纤维轨迹仍是一个具有挑战性的问题。     

融合分区和局部搜索的多模态多目标优化

为解决多模态多目标优化中种群多样性维持难和所得等价解数量不足问题,基于分区搜索和局部搜索,本研究提出一种融合分区和局部搜索的多模态多目标粒子群算法(multimodal multi-objective particle swarm optimization combing zoning search and local search,ZLS-SMPSO-MM)。在所提算法中,整个搜索空间被分割成多个子空间以维持种群多样性和降低搜索难度;然后,使用已有的自组织多模态多目标粒子群算法在每个子空间搜索等价解和挖掘邻域信息,并利用局部搜索能力较强的协方差矩阵自适应算法对有潜力的区域进行精细搜索。通过14个多模态多目标优化问题测试,并与其他5种知名算法进行比较;实验结果表明ZLS-SMPSO-MM在决策空间能够找到更多的等价解,且整体性能要好于所比较算法。