Exploring the state sequence space for hidden Markov and semi-Markov chains

The knowledge of the state sequences that explain a given observed sequence for a known hidden Markovian model is the basis of various methods that may be divided into three categories: (i) enumeration of state sequences; (ii) summary of the possible state sequences in state profiles; (iii) computation of a global measure of the state sequence uncertainty. Concerning the first category, the generalized Viterbi algorithm for computing the top L most probable state sequences and the forward-backward algorithm for sampling state sequences are derived for hidden semi-Markov chains and hidden hybrid models combining Markovian and semi-Markovian states. Concerning the second category, a new type of state (and state change) profiles is proposed. The Viterbi forward-backward algorithm for computing these state profiles is derived for hidden semi-Markov chains and hidden hybrid models combining Markovian and semi-Markovian states. Concerning the third category, an algorithm for computing the entropy of the state sequence that explains an observed sequence is proposed. The complementarity and properties of these methods for exploring the state sequence space (including the classical state profiles computed by the forward-backward algorithm) are investigated and illustrated with examples.     

Finite dimensional predictors for hidden Markov chains

A continuous time Markov chain is observed in Gaussian noise. Finite dimensional normalized and unnormalized (Zakai) predictors are obtained for the state of the chain, for the number of jumps from one state to another and for the occupation time in any state.     

Multi-GPU accelerated multi-spin Monte Carlo simulations of the 2D Ising model

A Modern Graphics Processing unit (GPU) is able to perform massively parallel scientific computations at low cost. We extend our implementation of the checkerboard algorithm for the two-dimensional Ising model [T. Preis et al., Journal of Chemical Physics 228 (2009) 4468-4477] in order to overcome the memory limitations of a single GPU which enables us to simulate significantly larger systems. Using multi-spin coding techniques, we are able to accelerate simulations on a single GPU by factors up to 35 compared to an optimized single Central Processor Unit (CPU) core implementation which employs multi-spin coding. By combining the Compute Unified Device Architecture (CUDA) with the Message Parsing Interface (MPI) on the CPU level, a single Ising lattice can be updated by a cluster of GPUs in parallel. For large systems, the computation time scales nearly linearly with the number of GPUs used. As proof of concept we reproduce the critical temperature of the 2D Ising model using finite size scaling techniques.     

Continuous-time hidden Markov models for network performance evaluation

In this paper, we study the use of continuous-time hidden Markov models (CT-HMMs) for network protocol and application performance evaluation. We develop an algorithm to infer the CT-HMM from a series of end-to-end delay and loss observations of probe packets. This model can then be used to simulate network environments for network performance evaluation. We validate the simulation method through a series of experiments both in ns and over the Internet. Our experimental results show that this simulation method can represent a wide range of real network scenarios. It is easy to use, accurate and time efficient.     

Markov chain representations of discrete distributions applied to queueing models

We present applications of Markov chain based representations of discrete renewal distributions to queueing models, and extend the notion of that representation to some non-renewal discrete distributions. Two representations are considered; one based on remaining time, the other on elapsed time. These representations make it easier to use matrix-analytic methods for several stochastic models, especially queueing models, thereby allowing us to develop better algorithmically tractable procedures for their analysis. Specifically, they allow us to capitalize on the resulting special structures. We first discuss some key measures of these distributions using phase type distribution results, including some time reversibility relations between the elapsed and remaining time representations. We then show applications to the MAP/G/1, the GI/MSP/1 and the GI/G/1 systems, and briefly explain how the representations of the non-renewal types of discrete distributions can be used for the MRP/SMP/1 system. The emphasis of this paper is about efficient procedures for the R and G matrices associated with these queueing models.     

Reachability problems for Markov chains

We consider the following decision problem: given a finite Markov chain with distinguished source and target states, and given a rational number r, does there exist an integer n such that the probability to reach the target from the source in n steps is r? This problem, which is not known to be decidable, lies at the heart of many model checking questions on Markov chains. We provide evidence of the hardness of the problem by giving a reduction from the Skolem Problem: a number-theoretic decision problem whose decidability has been open for many decades.     

Hidden Markov random field models for TCA image analysis

Tooth Cementum Annulation (TCA) is an age estimation method carried out on thin cross sections of the root of mammalian teeth. Age is computed by adding the tooth eruption age to the count of annual incremental lines which are called tooth rings and appear in the cementum band. The number of rings is computed from an intensity (gray scale) image of the cementum band, by estimating the average ring width and then dividing the area of the cementum band by this estimate. The ring width is estimated by modelling the image by a hidden Markov random field, where intensities are assumed to be pixelwise conditionally independent and normally distributed, given a Markov random field of hidden binary labels, representing the“true scene”. To incorporate image macro-features (the long-range dependence among intensities and the quasi-periodicity in the placement of tooth rings), the label random field is defined by an energy function that depends on a parametric Gabor filter, convolved with the true scene. The filter parameter represents the unknown of main interest, i.e. the average width of the rings. The model is estimated through an EM algorithm, relying on the mean field approximation of the hidden label distribution and allows to predict the locations of the rings in the image.     

Quasi-Monte Carlo methods for Markov chains with continuous multi-dimensional state space

We describe a quasi-Monte Carlo method for the simulation of discrete time Markov chains with continuous multi-dimensional state space. The method simulates copies of the chain in parallel. At each step the copies are reordered according to their successive coordinates. We prove the convergence of the method when the number of copies increases. We illustrate the method with numerical examples where the simulation accuracy is improved by large factors compared with Monte Carlo simulation.     

Loss rates for stochastic fluid models

We introduce loss rates, a novel class of performance measures for Markovian stochastic fluid models and discuss their applications potential. We derive analytical expressions for loss rates and describe efficient methods for their evaluation. Further, we study interesting asymptotic properties of loss rates for large size of the buffer, which are crucial for identifying the Quality of Service requirements guaranteed for each user. We illustrate the theory with a numerical example.     

Almost sure parameter estimation and convergence rates for hidden Markov models

A continuous-time version of Kronecker's Lemma is established and used to give rates of convergence for parameter estimates in hidden Markov models.     

A note on exponential stability of the nonlinear filter for denumerable Markov chains

Asymptotic stability of the optimal filter with respect to its initial conditions is investigated in this paper. Under the assumption that the observation function is one to one and the observation noise is sufficiently small, it is shown that exponential stability of the nonlinear filter holds for a large class of denumerable Markov chains, including all finite Markov chains. Throughout this paper, ergodicity of the signal process is not assumed.     

差分进化算法马尔可夫链模型及收敛性分析

差分进化算法是一种基于种群差异的优化算法,主要应用于解决连续空间的优化问题。目前,研究人员主要在算法的改进和应用方面研究差分进化算法,很少从理论角度对其进行研究。为了分析差分进化算法的收敛性,定义优化个体、种群的状态转移,并提出种群的最优状态集合。根据差分进化算法的操作算子计算出个体的状态迁移概率,并证明种群状态序列是有限齐次马尔可夫链,进而建立差分进化算法的马尔可夫链模型;最后,证明差分进化算法无法保证全局收敛。理论研究结果表明,适当保证种群的多样性能够提高差分进化算法的性能。     

Analysis of migration models of biogeography-based optimization using Markov theory

Biogeography-based optimization (BBO) is a new evolutionary algorithm inspired by biogeography, which involves the study of the migration of biological species between habitats. Previous work has shown that various migration models of BBO result in significant changes in performance. Sinusoidal migration models have been shown to provide the best performance so far. Motivated by biogeography theory and previous results, in this paper a generalized sinusoidal migration model curve is proposed. A previously derived BBO Markov model is used to analyze the effect of migration models on optimization performance, and new theoretical results which are confirmed with simulation results are obtained. The results show that the generalized sinusoidal migration model is significantly better than other models for simple but representative problems, including a unimodal one-max problem, a multimodal problem, and a deceptive problem. In addition, performance comparison is further investigated through 23 benchmark functions with a wide range of dimensions and diverse complexities, to verify the superiority of the generalized sinusoidal migration model.     

Risk-sensitive filtering and smoothing for hidden Markov models

In this paper, we address the problem of risk-sensitive filtering and smoothing for discrete-time Hidden Markov Models (HMM) with finite-discrete states. The objective of risk-sensitive filtering is to minimise the expectation of the exponential of the squared estimation error weighted by a risk-sensitive parameter. We use the so-called Reference Probability Method in solving this problem. We achieve finite-dimensional linear recursions in the information state, and thereby the state estimate that minimises the risk-sensitive cost index. Also, fixed-interval smoothing results are derived. We show that L₂ or risk-neutral filtering for HMMs can be extracted as a limiting case of the risk-sensitive filtering problem when the risk-sensitive parameter approaches zero.     

Measure change estimates for hidden Markov models

Using a change of measure, the finite state observation process of a Markov chain is transformed into a sequence of independent random variables. By computing unnormalized conditional estimates under the new measure, simple, recursive formulate are obtained.     

Quotients of Markov chains and asymptotic properties of the stationary distribution of the Markov chain associated to an evolutionary algorithm

In this work, a method is presented for analysis of Markov chains modeling evolutionary algorithms through use of a suitable quotient construction. Such a notion of quotient of a Markov chain is frequently referred to as “coarse graining” in the evolutionary computation literature. We shall discuss the construction of a quotient of an irreducible Markov chain with respect to an arbitrary equivalence relation on the state space. The stationary distribution of the quotient chain is “coherent” with the stationary distribution of the original chain. Although the transition probabilities of the quotient chain depend on the stationary distribution of the original chain, we can still exploit the quotient construction to deduce some relevant properties of the stationary distribution of the original chain. As one application, we shall establish inequalities that describe how fast the stationary distribution of Markov chains modeling evolutionary algorithms concentrates on the uniform populations as the mutation rate converges to 0. Further applications are discussed. One of the results related to the quotient construction method is a significant improvement of the corresponding result of the authors’ previous conference paper [Mitavskiy et al. (2006) In: Simulated Evolution and Learning, Proceedings of SEAL 2006, Lecture Notes in Computer Science v. 4247, Springer Verlag, pp 726–733]. This papers implications are all strengthened accordingly.

Causal coding and control for Markov chains

A Markov chain is controlled by a decision maker receiving his observations of the state via a noisy memoriless channel. That information is encoded causally. The encoder is assumed to have perfect channel feedback information.Separation results are derived and used to prove that encoding is useless for a class of symmetric channels.This paper extends the results of the authors (1983) by using methods similar to those of that paper.     

A tool for model-checking Markov chains

Markov chains are widely used in the context of the performance and reliability modeling of various systems. Model checking of such chains with respect to a given (branching) temporal logic formula has been proposed for both discrete [34, 10] and continuous time settings [7, 12]. In this paper, we describe a prototype model checker for discrete and continuous-time Markov chains, the Erlangen–Twente Markov Chain Checker E⊢MC², where properties are expressed in appropriate extensions of CTL. We illustrate the general benefits of this approach and discuss the structure of the tool. Furthermore, we report on successful applications of the tool to some examples, highlighting lessons learned during the development and application of E⊢MC². Published online: 19 November 2002 Correspondence to: Holger Hermanns     

Parallel and interacting Markov chain Monte Carlo algorithm

In many situations it is important to be able to propose N independent realizations of a given distribution law. We propose a strategy for making N parallel Monte Carlo Markov chains (MCMC) interact in order to get an approximation of an independent N-sample of a given target law. In this method each individual chain proposes candidates for all other chains. We prove that the set of interacting chains is itself a MCMC method for the product of N target measures. Compared to independent parallel chains this method is more time consuming, but we show through examples that it possesses many advantages. This approach is applied to a biomass evolution model.