Wavelet-based statistical signal processing using hidden Markovmodels

Wavelet-based statistical signal processing techniques such as denoising and detection typically model the wavelet coefficients as independent or jointly Gaussian. These models are unrealistic for many real-world signals. We develop a new framework for statistical signal processing based on wavelet-domain hidden Markov models (HMMs) that concisely models the statistical dependencies and non-Gaussian statistics encountered in real-world signals. Wavelet-domain HMMs are designed with the intrinsic properties of the wavelet transform in mind and provide powerful, yet tractable, probabilistic signal models. Efficient expectation maximization algorithms are developed for fitting the HMMs to observational signal data. The new framework is suitable for a wide range of applications, including signal estimation, detection, classification, prediction, and even synthesis. To demonstrate the utility of wavelet-domain HMMs, we develop novel algorithms for signal denoising, classification, and detection     

Upper Bound Kullback–Leibler Divergence for Transient Hidden Markov Models

This paper reports an upper bound for the Kullback–Leibler divergence (KLD) for a general family of transient hidden Markov models (HMMs). An upper bound KLD (UBKLD) expression for Gaussian mixtures models (GMMs) is presented which is generalized for the case of HMMs. Moreover, this formulation is extended to the case of HMMs with nonemitting states, where under some general assumptions, the UBKLD is proved to be well defined for a general family of transient models. In particular, the UBKLD has a computationally efficient closed-form for HMMs with left-to-right topology and a final nonemitting state, that we refer to as left-to-right transient HMMs. Finally, the usefulness of the closed-form expression is experimentally evaluated for automatic speech recognition (ASR) applications, where left-to-right transient HMMs are used to model basic acoustic-phonetic units. Results show that the UBKLD is an accurate discrimination indicator for comparing acoustic HMMs used for ASR.      

Unsupervised texture segmentation using multichannel decompositionand hidden Markov models

In this paper, we describe an automatic unsupervised texture segmentation scheme using hidden Markov models (HMMs). First, the feature map of the image is formed using Laws' micromasks and directional macromasks. Each pixel in the feature map is represented by a sequence of 4-D feature vectors. The feature sequences belonging to the same texture are modeled as an HMM. Thus, if there are M different textures present in an image, there are M distinct HMMs to be found and trained. Consequently, the unsupervised texture segmentation problem becomes an HMM-based problem, where the appropriate number of HMMs, the associated model parameters, and the discrimination among the HMMs become the foci of our scheme. A two-stage segmentation procedure is used. First, coarse segmentation is used to obtain the approximate number of HMMs and their associated model parameters. Then, fine segmentation is used to accurately estimate the number of HMMs and the model parameters. In these two stages, the critical task of merging the similar HMMs is accomplished by comparing the discrimination information (DI) between the two HMMs against a threshold computed from the distribution of all DI's. A postprocessing stage of multiscale majority filtering is used to further enhance the segmented result. The proposed scheme is highly suitable for pipeline/parallel implementation. Detailed experimental results are reported. These results indicate that the present scheme compares favorably with respect to other successful schemes reported in the literature.     

A computationally efficient approach to the estimation of two- and three-dimensional hidden Markov models.

Statistical modeling methods are becoming indispensable in today's large-scale image analysis. In this paper, we explore a computationally efficient parameter estimation algorithm for two-dimensional (2-D) and three-dimensional (3-D) hidden Markov models (HMMs) and show applications to satellite image segmentation. The proposed parameter estimation algorithm is compared with the first proposed algorithm for 2-D HMMs based on variable state Viterbi. We also propose a 3-D HMM for volume image modeling and apply it to volume image segmentation using a large number of synthetic images with ground truth. Experiments have demonstrated the computational efficiency of the proposed parameter estimation technique for 2-D HMMs and a potential of 3-D HMM as a stochastic modeling tool for volume images.     

A Posteriori Probability Distances Between Finite-Alphabet Hidden Markov Models

In this correspondence, we consider a probability distance problem for a class of hidden Markov models (HMMs). The notion of conditional relative entropy between conditional probability measures is introduced as an a posteriori probability distance which can be used to measure the discrepancy between hidden Markov models when a realized observation sequence is observed. Using a measure change technique, we derive a representation for conditional relative entropy in terms of the parameters of the HMMs and conditional expectations given measurements. With this representation, we show that this distance can be calculated using an information state approach     

An MCMC sampling approach to estimation of nonstationary hiddenMarkov models

Hidden Markov models (HMMs) represent a very important tool for analysis of signals and systems. In the past two decades, HMMs have attracted the attention of various research communities, including the ones in statistics, engineering, and mathematics. Their extensive use in signal processing and, in particular, speech processing is well documented. A major weakness of conventional HMMs is their inflexibility in modeling state durations. This weakness can be avoided by adopting a more complicated class of HMMs known as nonstationary HMMs. We analyze nonstationary HMMs whose state transition probabilities are functions of time that indirectly model state durations by a given probability mass function and whose observation spaces are discrete. The objective of our work is to estimate all the unknowns of a nonstationary HMM, which include its parameters and the state sequence. To that end, we construct a Markov chain Monte Carlo (MCMC) sampling scheme, where sampling from all the posterior probability distributions is very easy. The proposed MCMC sampling scheme has been tested in extensive computer simulations on finite discrete-valued observed data, and some of the simulation results are presented     

Aggregated Renewal Markov Processes With Applications in Simulating Mobile Broadcast Systems

For purposes of simulating contemporary communication systems, it is, in many cases, useful to apply error models for specific levels of abstraction. Such models should approximate the packet error behavior of a given system at a specific protocol layer, thus incorporating the possible detrimental effects of lower protocol layers. Packet error models can efficiently be realized using finite-state models; for example, there exists a wide range of studies on using Markov models to simulate communication channels. In this paper, we consider aggregated Markov processes, which are a subclass of hidden Markov models (HMMs). Artificial limitations are set on the state transition probabilities of the models to find efficient methods of parameter estimation. We apply these models to the simulation of the performance of digital video broadcasting-handheld (DVB-H). The parameters of the packet error models are approximated as functions of the time-variant received signal strength and speed of a mobile vehicular DVB-H receiver, and it is shown that useful results may be achieved with the described packet error models, particularly when simulating mobile reception in field conditions.     

Phonemic hidden Markov models with continuous mixture outputdensities for large vocabulary word recognition

The authors demonstrate the effectiveness of phonemic hidden Markov models with Gaussian mixture output densities (mixture HMMs) for speaker-dependent large-vocabulary word recognition. Speech recognition experiments show that for almost any reasonable amount of training data, recognizers using mixture HMMs consistently outperform those employing unimodal Gaussian HMMs. With a sufficiently large training set (e.g. more than 2500 words), use of HMMs with 25-component mixture distributions typically reduces recognition errors by about 40%. It is also found that the mixture HMMs outperform a set of unimodal generalized triphone models having the same number of parameters. Previous attempts to employ mixture HMMs for speech recognition proved discouraging because of the high complexity and computational cost in implementing the Baum-Welch training algorithm. It is shown how mixture HMMs can be implemented very simply in unimodal transition-based frameworks by allowing multiple transitions from one state to another     

Maximum Likelihood Training and Adaptation of Embedded Speech Recognizers for Mobile Environments

For the acoustic models of embedded speech recognition systems, hidden Markov models (HMMs) are usually quantized and the original full space distributions are represented by combinations of a few quantized distribution prototypes. We propose a maximum likelihood objective function to train the quantized distribution prototypes. The experimental results show that the new training algorithm and the link structure adaptation scheme for the quantized HMMs reduce the word recognition error rate by 20.0%.     

Application of the Gibbs distribution to hidden Markov modeling inspeaker independent isolated word recognition

A method of integrating the Gibbs distributions (GDs) into hidden Markov models (HMMs) is presented. The probabilities of the hidden state sequences of HMMs are modeled by GDs in place of the transition probabilities. The GDs offer a general way in modeling neighbor interactions of Markov random fields where the Markov chains in HMMs are special cases. An algorithm for estimating the model parameters is developed based on Baum reestimation, and an algorithm for computing the probability terms is developed using a lattice structure. The GD models were used for experiments in speech recognition on the TI speaker-independent, isolated digit database. The observation sequences of the speech signals were modeled by mixture Gaussian autoregressive densities. The energy functions of the GDs were developed using very few parameters and proved adequate in hidden layer modeling. The results of the experiments showed that the GD models performed at least as well as the HMM models     

Support Vector Machine Training for Improved Hidden Markov Modeling

We present a discriminative training algorithm, that uses support vector machines (SVMs), to improve the classification of discrete and continuous output probability hidden Markov models (HMMs). The algorithm uses a set of maximum-likelihood (ML) trained HMM models as a baseline system, and an SVM training scheme to rescore the results of the baseline HMMs. It turns out that the rescoring model can be represented as an unnormalized HMM. We describe two algorithms for training the unnormalized HMM models for both the discrete and continuous cases. One of the algorithms results in a single set of unnormalized HMMs that can be used in the standard recognition procedure (the Viterbi recognizer), as if they were plain HMMs. We use a toy problem and an isolated noisy digit recognition task to compare our new method to standard ML training. Our experiments show that SVM rescoring of hidden Markov models typically reduces the error rate significantly compared to standard ML training.     

Isolated-utterance speech recognition using hidden Markov modelswith bounded state durations

Hidden Markov models (HMMs) with bounded state durations (HMM/BSD) are proposed to explicitly model the state durations of HMMs and more accurately consider the temporal structures existing in speech signals in a simple, direct, but effective way. A series of experiments have been conducted for speaker dependent applications using 408 highly confusing first-tone Mandarin syllables as the example vocabulary. It was found that in the discrete case the recognition rate of HMM/BSD (78.5%) is 9.0%, 6.3%, and 1.9% higher than the conventional HMMs and HMMs with Poisson and gamma distribution state durations, respectively. In the continuous case (partitioned Gaussian mixture modeling), the recognition rates of HMM/BSD (88.3% with 1 mixture, 88.8% with 3 mixtures, and 89.4% with 5 mixtures) are 6.3%, 5.0%, and 5.5% higher than those of the conventional HMMs, and 5.9% (with 1 mixture), 3.9% (with 3 mixtures) and 3.1% (with 1 mixture), 1.8% (with 3 mixtures) higher than HMMs with Poisson and gamma distributed state durations, respectively     

Spoken language understanding

This article is intended to serve as an introduction to the field of statistical SLU, based on the mainstream statistical modeling approach that shares a similar mathematical framework with many other statistical pattern recognition applications such as speech recognition. In particular, we formulated a number of statistical models for SLU in the literature as extensions to HMMs as segment models, where a multiple-word block (segment) with word dependency is generated from each underlying Markov state corresponding to each individual semantic slot defined from the application domain. In the past, due partly to its nature of symbolic rather than numeric processing, the important field of SLU in human language technology has not been widely exposed to the signal processing research community. However, many key techniques in SLU originated from statistical signal processing. And because SLU is becoming increasingly important, as one major target application area of ASR that has been dear to many signal processing researchers, we contribute this article to provide a natural bridge between ASR and SLU in methodological and mathematical foundation. It is our hope that when the mathematical basis of SLU becomes well known through this introductory article, more powerful techniques established by signal processing researchers may further advance SLU to form a solid application area, making speech technology a successful component for intelligent human-machine communication.     

基于隐马尔可夫模型的中文文本事件信息抽取

提出了一种基于隐马尔可夫模型的中文文本事件抽取方法,该方法首先通过触发词探测从文本中发现特定的候选事件语句,然后利用隐马尔可夫模型从这些语句中抽取每个候选事件的事件要素,为每一类事件要素构建一个独立的隐马尔可夫模型用于该类事件要素的抽取,构建模型的关键是模型结构的学习和参数估计。实验结果表明,该方法能较好地实现中文文本事件抽取,较其他方法有更好的抽取性能。     

Use of multiple vector quantisation for semicontinuous-HMM speechrecognition

Although the continuous hidden Markov model (CHMM) technique seems to be the most flexible and complete tool for speech modelling. It is not always used for the implementation of speech recognition systems because of several problems related to training and computational complexity. Thus, other simpler types of HMMs, such as discrete (DHMM) or semicontinuous (SCHMM) models, are commonly utilised with very acceptable results. Also, the superiority of continuous models over these types of HMMs is not clear. The authors' group has previously introduced the multiple vector quantisation (MVQ) technique, the main feature of which is the use of one separated VQ codebook for each recognition unit. The MVQ technique applied to DHMM models generates a new HMM modelling (basic MVQ models) that allows incorporation into the recognition dynamics of the input sequence information wasted by the discrete models in the VQ process. The authors propose a new variant of HMM models that arises from the idea of applying MVQ to SCHMM models. These are SCMVQ-HMM (semicontinuous multiple vector quantisation HMM) models that use one VQ codebook per recognition unit and several quantisation candidates for each input vector. It is shown that SCMVQ modelling is formally the closest one to CHMM, although requiring even less computation than SCHMMs. After studying several implementation issues of the MVQ technique. Such as which type of probability density function should be used, the authors show the superiority of SCMVQ models over other types of HMM models such as DHMMs, SCHMMs or the basic MVQs     

Complexity reduction in fixed-lag smoothing for hidden Markovmodels

We investigate approximate smoothing schemes for a class of hidden Markov models (HMMs), namely, HMMs with underlying Markov chains that are nearly completely decomposable. The objective is to obtain substantial computational savings. Our algorithm can not only be used to obtain aggregate smoothed estimates but can be used also to obtain systematically approximate full-order smoothed estimates with computational savings and rigorous performance guarantees, unlike many of the aggregation methods proposed earlier     

Hidden Markov Models with Factored Gaussian Mixtures Densities

1　Introduction　Manyrealobserveddataarecharacterizedbymultiplecoupledcausesorfactors.Forinstance ,faceimagesmaybegeneratedbycombiningeyebrows,eyes ,noseandmouth .Similarly ,speechsignalsmayresultfromanin teractionofmotionsoffactorssuchasthejaw ,tongue ,velum ,lipandmouth .RecentlyZemelandHintonpro posedafactoriallearningarchitecture[1～ 2 ] todealwithfactorialdata .Thegoaloffactoriallearningistodiscov erthemultipleunderlyingcausesorfactorsfromtheob serveddataandfindarepresentationthatwillbo…     

Automatic identification of audio recordings based on statistical modeling

This paper describes a methodology for the automatic identification of audio recordings of ethnic music. The identification is based on an application of hidden Markov models (HMMs), which are automatically built from a representation of the music pieces to be identified. States of the HMMs are labeled with music events, and the transition and observation probabilities are directly computed from the information on the music piece. The recordings are modeled by a set of acoustic features that are computed according with the characteristics of the music events. Three alternative approaches, based on typical applications of HMMs, are proposed to perform the identification. Tests carried out on collections of recordings showed that the methodology can achieve good results, and the identification rate is high enough to suggest applications for automatic retrieval of metadata and for the identification of alternative recordings of the same piece.     

Asymptotic smoothing errors for hidden Markov models

In this paper, the asymptotic smoothing error for hidden Markov models (HMMs) is investigated using hypothesis testing ideas. A family of HMMs is studied parametrised by a positive constant /spl epsiv/, which is a measure of the frequency of change. Thus, when /spl epsiv//spl rarr/0, the HMM becomes increasingly slower moving. We show that the smoothing error is O(/spl epsiv/). These theoretical predictions are confirmed by a series of simulations.