Learning the dynamics of two-dimensional stochastic Markov processes

We show that it is possible to learn the forces causing an observed two-dimensional stochastic Markov process. Hereby, we extend the ideas presented in our earlier work [1–3], where we discussed one-dimensional processes. Appropriate short-time correlation function measurements are used as constraints in the maximum information principle of Jaynes, allowing us to formulate the joint probability distribution function of the process. This is done using the method of Lagrange multipliers, which we determine by means of a dynamical learning method. Next, we derive explicit formulas expressing the drift- and diffusion coefficients of the Ito-Langevin equation corresponding to the process in terms of the Lagrange multipliers. This provides us with the sought for underlying deterministic and stochastic dynamics. The method was tested on a simulated Ornstein-Uhlenbeck process, showing good confirmation of the theory.     

Robustness of linearly solvable Markov games employing inaccurate dynamics model

As a model-based reinforcement learning technique, linearly solvable Markov decision process (LMDP) gives an efficient way to find an optimal policy by making the Bellman equation linear under some assumptions. Since LMDP is regarded as model-based reinforcement learning, the performance of LMDP is sensitive to the accuracy of the environmental model. To overcome the problem of the sensitivity, linearly solvable Markov game (LMG) has been proposed, which is an extension of LMDP based on the game theory. This paper investigates the robustness of LMDP- and LMG-based controllers against modeling errors in both discrete and continuous state-action problems. When there is a discrepancy between the model used for building the control policy and dynamics of the tested environment, the LMG-based control policy maintained good performance while that of the LMDP-based control policy deteriorated drastically. Experimental results support the usefulness of LMG framework when acquiring an accurate model of the environment is difficult.     

Iterative analysis of Markov regenerative models

Conventional algorithms for the steady-state analysis of Markov regenerative models suffer from high computational costs which are caused by densely populated matrices. In this paper, a new algorithm is suggested which avoids computing these matrices explicitly. Instead, a two-stage iteration scheme is used. An extended version of uniformization is applied as a subalgorithm to compute the required transient quantities “on-the fly”. The algorithm is formulated in terms of stochastic Petri nets. A detailed example illustrates the proposed concepts.     

On-line sensitivity analysis of Markov chains

Discrete-event systems modeled as continuous-time Markov processes and characterized by some integer-valued parameter are considered. The problem addressed is that of estimating performance sensitivities with respect to this parameter by directly observing a single sample path of the system. The approach is based on transforming the nominal Markov chain into a reduced augmented chain, the stationary-state probabilities which can be easily combined to obtain stationary-state probability sensitivities with respect to the given parameter. Under certain conditions, the reduced augmented chain state transitions are observable with respect to the state transitions of the system itself, and no knowledge of the nominal Markov-chain state of the transition rates is required. Applications for some queueing systems are included. The approach incorporates estimation of unknown transition rates when needed and is extended to real-valued parameters     

Availability analysis of a four-state Markov system

This paper presents a mathematical model of a system with four modes : normal, partial failure, down and failed. The problem is formulated on the assumption that all the failure and repair time distributions are arbitrary. The case when all the rates are negative exponential has been worked out. Laplace transforms of transient probabilities are employed to obtain the pointwise availability of the system and earlier results are verified.     

隐马尔可夫模型的多序列比对研究

研究一种关于隐马尔可夫模型的多序列比对,利用值和特征序列的保守性,通过增加频率因子,改进传统隐马尔可夫模型算法的不足。实验表明,新算法不但提高了模型的稳定性,而且应用于蛋白质家族识别,平均识别率比传统隐马尔可夫算法提高了3.3个百分点。     

Statics and dynamics of cognitive and qualitative matchmaking in task fulfillment

Two main ingredients related to successful task performance are cognition and quality. Supply and demand of these concepts for knowledge intensive tasks are studied in this paper to fuel successful task fulfillment. Cognitive characteristics are supplied by actors performing tasks. Organizational developments such as growing complexity and increasing customer orientation may increase cognitive load. Stakeholders of tasks have quality requirements. These requirements may be affected if actors experience an increase in cognitive load. It is observed that knowledge intensive tasks demand cognitive characteristics and supply quality factors. Actors supply cognition and stakeholders demand quality. The gap between supply and demand can be bridged by introducing several models. These models consist of a matchmaking framework, conceptual models, and dynamic models. The matchmaking framework shows how supply and demand of cognitive characteristics or quality factors can be matched. Relations and roles of the concepts involved in task fulfillment are mapped out by the conceptual models. The dynamic models show causes that have effects on the supply of cognitive characteristics and the level of quality. These insights in the relations and dependencies between cognition and quality increase our understanding of the key concepts for successful task fulfillment.     

Algorithms for qualitative analysis of complex systems

Algorithms are proposed that construct first integrals and Lyapunov functions for stability analysis of stationary solutions of differential equations describing mechanical systems of linked bodies.Translated from Kibernetika i Sistemnyi Analiz, No. 1, pp. 138–148, January–February, 1992.     

Computational Bayesian analysis of hidden Markov mesh models

Versions of the Gibbs sampler are derived for the analysis of data from the hidden Markov mesh random fields sometimes used in image analysis. This provides a numerical approach to the otherwise intractable Bayesian analysis of these problems. Detailed formulation is provided for particular examples based on Devijver's Markov mesh model (1988), and the BUGS package is used to do the computations. Theoretical aspects are discussed and a numerical study, based on image analysis, is reported     

Stochastic analysis of OneMax problem using Markov chain

Experimental and analytical investigations are performed for OneMax problem using Wright–Fisher model. This study investigates the distribution of the first order schema frequency in the evolution process of Genetic Algorithm (GA). Effects of mutation in GA are analyzed for the standard mutation and asymmetric mutation models. If a population is in linkage equilibrium, it can be shown that OneMax problem is equivalent to the asymmetric mutation model. Thus, we can apply theoretical results obtained in the asymmetric mutation model to OneMax problem and investigate the convergence time of GA calculation within the framework of Wright–Fisher model.     

Numerical analysis of rational processes beyond Markov chains

Matrix exponential distributions and rational arrival processes have been proposed as an extension to pure Markov models. The paper presents an approach where these process types are used to describe the timing behavior in quantitative models like queueing networks, stochastic Petri nets or stochastic automata networks. The resulting stochastic process, which is called a rational process, is defined and it is shown that the matrix governing the behavior of the process has a structured representation which allows one to represent the matrix in a very compact form.     

Mining qualitative patterns in spatial cluster analysis

Clustering is an important concept formation process within AI. It detects a set of objects with similar characteristics. These similar aggregated objects represent interesting concepts and categories. As clustering becomes more mature, post-clustering activities that reason about clusters need a great attention. Numerical quantitative information about clusters is not as intuitive as qualitative one for human analysis, and there is a great demand for an intelligent qualitative cluster reasoning technique in data-rich environments. This article introduces a qualitative cluster reasoning framework that reasons about clusters. Experimental results demonstrate that our proposed qualitative cluster reasoning reveals interesting cluster structures and rich cluster relations.     

Quantitative and qualitative analysis of SysML activity diagrams

Model-driven engineering refers to a range of approaches that uses models throughout systems and software development life cycle. Towards sustaining such a successful approach in practice, we present a model-based verification framework that supports the quantitative and qualitative analysis of SysML activity diagrams. To this end, we propose an algorithm that maps SysML activity diagrams into Markov decision processes expressed using the language of the probabilistic symbolic model checker PRISM. Furthermore, we elaborate on the correctness of our translation algorithm by proving its soundness with respect to a SysML activity diagrams operational semantics that we also present in this work. The generated models can be verified against a set of properties expressed in the probabilistic computation tree logic. To automate our approach, we developed a prototype tool that interfaces a modeling environment and the probabilistic model checker. We also show how to leverage adversary generation to provide the developer with a useful counterexample/witness as a feedback on the verified properties. Finally, the established theoretical foundations are complemented with an illustrative case study that demonstrates the usability and benefit of such a framework.     

Simulation-based sequential analysis of Markov switching stochastic volatility models

We propose a simulation-based algorithm for inference in stochastic volatility models with possible regime switching in which the regime state is governed by a first-order Markov process. Using auxiliary particle filters we developed a strategy to sequentially learn about states and parameters of the model. The methodology is tested against a synthetic time series and validated with a real financial time series: the IBOVESPA stock index (São Paulo Stock Exchange).     

Hidden Markov chains and the analysis of genome structure

In this paper, statistical methods based on a hidden Markov chain model are used to study the structure of some small complete genomes and a human genome segment. A variety of discrete compositional domains are discovered and their correlations with genome function are explored.     

基于马尔可夫链的舆情热度趋势分析

针对舆情热度的波动范围较大,并会呈现某种趋势的特点,提出了一种基于马尔可夫链的舆情热度趋势分析模型。该模型采集相关热点舆情的指标数据,得到热度的时间序列值;分析热度的趋势变化,划分状态空间,构建状态转移矩阵,预测热度的趋势变化区间。实验表明,该方法能有效地预测热点舆情的走势,进而辅助对舆情的引导和控制。