Perturbation realization, potentials, and sensitivity analysis ofMarkov processes

Two fundamental concepts and quantities, realization factors and performance potentials, are introduced for Markov processes. The relations among these two quantities and the group inverse of the infinitesimal generator are studied. It is shown that the sensitivity of the steady-state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a Markov process. Based on these results, algorithms for estimating performance sensitivities on a single sample path of a Markov process can be proposed. The potentials in this paper are defined through realization factors and are shown to be the same as those defined by Poisson equations. The results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis (IPA) and non-IPA approaches to the sensitivity analysis of steady-state performance; they also provide a theoretical background for the PA algorithms developed in recent years     

New performance sensitivity formulae for a class of product-form queueing networks

Perturbation analysis (PA) applies a dynamic point of view to the sample paths of stochastic systems; the realization factor, one of the main concepts of PA, measures the final effect of a perturbation on system performance and provides a novel approach in obtaining performance sensitivities. In this paper, we solve analytically the set of equations for realization factors of a two-server cyclic network. We prove an invariance property of the performance sensitivity for Norton's aggregation. Using the results, we derive closed-form formulae for the derivatives of performance measures in a closed queueing network with load-dependent exponential servers. The performance measures have two general forms: customer average and time average. In contrast with the usual approach based on product-form solutions, our results provide additional insights into the performance sensitivity of closed queueing networks and have immediate applications to problems of optimal control. The general formulae are expressed in terms of Buzen's algorithm with a computational complexity comparable to that of the formulae obtained by directly taking the derivatives of the product-form solutions.     

The potential structure of sample paths and performance sensitivities of Markov systems

We study the structure of sample paths of Markov systems by using performance potentials as the fundamental units. With a sample path-based approach, we show that performance sensitivity formulas (performance gradients and performance differences) of Markov systems can be constructed intuitively, by first principles, with performance potentials (or equivalently, perturbation realization factors) as building blocks. In particular, we derive sensitivity formulas for two Markov chains with possibly different state spaces. The proposed approach can be used to obtain flexibly the sensitivity formulas for a wide range of problems, including those with partial information. These formulas are the basis for performance optimization of discrete event dynamic systems, including perturbation analysis, Markov decision processes, and reinforcement learning. The approach thus provides insight on on-line learning and performance optimization and opens up new research directions. Sample path based algorithms can be developed.     

Policy iteration for customer-average performance optimization of closed queueing systems

We consider the optimization of queueing systems with service rates depending on system states. The optimization criterion is the long-run customer-average performance, which is an important performance metric, different from the traditional time-average performance. We first establish, with perturbation analysis, a difference equation of the customer-average performance in closed networks with exponentially distributed service times and state-dependent service rates. Then we propose a policy iteration optimization algorithm based on this difference equation. This algorithm can be implemented on-line with a single sample path and does not require knowing the routing probabilities of queueing systems. Finally, we give numerical experiments which demonstrate the efficiency of our algorithm. This paper gives a new direction to efficiently optimize the “customer-centric” performance in queueing systems.     

Comparative effectiveness of certain queueing systems with adaptive demand and service mechanisms

This paper sets out to evaluate simple queueing models embodying the concept of adaptation of service to demand and vice versa, systems which hold out the promise of improved operational characteristics by comparison with conventional non-adaptive systems. Over the many years during which research in queueing theory has been in progress there has been little concern with models of this kind, yet it is in the direction of environmental adaptation that one must look for operational improvements. The models studied are mainly of birth and death type. Some consideration is given to renewal models which, in a certain sense, are the equivalents of birth and death types. The emphasis in the paper is placed on the requirements of an operational assessment and on its realization. The use of digital computers in research of this kind is underlined. For illustration extracts from [9] are used to describe the performance of three fundamental adaptive systems.     

A note on the relation between weak derivatives and perturbation realization

Studies the relationship between two important approaches in perturbation analysis (PA)-perturbation realization (PR) and weak derivatives (WDs). Specifically, we study the relation between PR and WDs for estimating the gradient of stationary performance measures of a finite state-space Markov chain. We show that the WDs expression for the gradient of a stationary performance measure can be interpreted as the expected PR factor where the expectation is carried out with respect to a distribution that is given through the weak derivative of the transition kernel of the Markov chain. Moreover, we present unbiased gradient estimators.     

闭排队网络当性能函数与参数相关时的性能灵敏度分析

借助于无穷小矩阵摄动方法, 讨论了一类Markov过程, 其稳态性能关于参数摄动的灵敏度分析问题. 然后研究了闭排队网络的稳态性能灵敏度分析问题, 并在参数相关性能函数的情况下, 给出了网络的几种稳态性能的灵敏度公式. 这些公式表明稳态性能灵敏度很容易通过网络势能进行计算.     

Semi-Markov decision problems and performance sensitivity analysis

Recent research indicates that Markov decision processes (MDPs) can be viewed from a sensitivity point of view; and the perturbation analysis (PA), MDPs, and reinforcement learning (RL) are three closely related areas in optimization of discrete-event dynamic systems that can be modeled as Markov processes. The goal of this paper is two-fold. First, we develop the PA theory for semi-Markov processes (SMPs); and then we extend the aforementioned results about the relation among PA, MDP, and RL to SMPs. In particular, we show that performance sensitivity formulas and policy iteration algorithms of semi-Markov decision processes can be derived based on the performance potential and realization matrix. Both the long-run average and discounted-cost problems are considered. This approach provides a unified framework for both problems, and the long-run average problem corresponds to the discounted factor being zero. The results indicate that performance sensitivities and optimization depend only on first-order statistics. Single sample path-based implementations are discussed.     

Lead time control in multi-class multi-stage assembly systems with finite capacity

In this paper, we model multi-class multi-stage assembly systems with finite capacity as queueing networks. It is assumed that different classes (types) of products are produced by the production system and products’ orders for different classes are received according to independent Poisson processes. Each service station of the queueing network specifies a manufacturing or assembly operation, in that processing times for different types of products are independent and exponentially distributed random variables with service rates, which are controllable, and the queueing discipline is First Come First Served (FCFS). Different types of products may be different in their routing sequences of manufacturing and assembly operations. For modeling multi-class multi-stage assembly systems, we first consider every class separately and convert the queueing network of each class into an appropriate stochastic network. Then, by using the concept of continuous-time Markov processes, a system of differential equations is created to obtain the distribution function of manufacturing lead time for any type of product, which is actually the time between receiving the order and the delivery of finished product. Furthermore, we develop a multi-objective model with three conflicting objectives to optimally control the service rates, and use goal attainment method to solve a discrete-time approximation of the original multi-objective continuous-time problem.     

A single sample path-based performance sensitivity formula forMarkov chains

Using a sample path approach, we derive a new formula for performance sensitivities of discrete-time Markov chains. A distinguished feature of this formula is that the quantities involved can be estimated by analyzing a single sample path of a Markov chain. Thus, the formula provides a new direction for sensitivity analysis and can be viewed as an extension of the perturbation realization theory to problems where infinitesimal perturbation analysis does not work well     

Departure process of a single server queueing system with Markov renewal input and general service time distribution

The independence of processes in queueing systems is generally assumed when developing queueing models. However, real systems often involve several process dependencies, and failure to take these into consideration can lead to serious underestimation of the performance measures. We consider herein a single server queueing system with a Markov renewal process (MRP) for its arrival process and a general service time distribution, and derive the distribution function and correlation coefficient of the departure process. Since the departure process also often corresponds to an arrival process in downstream queues, the results obtained here can be used to derive a better approximation of the performance measures of a non-product form general queueing network.     

The Relations Among Potentials, Perturbation Analysis, and Markov Decision Processes

This paper provides an introductory discussion for an important concept, the performance potentials of Markov processes, and its relations with perturbation analysis (PA), average-cost Markov decision processes (MDP), Poisson equations, -potentials, the fundamental matrix, and the group inverse of the transition matrix (or the infinitesimal generators). Applications to single sample path-based performance sensitivity estimation and performance optimization are also discussed. On-line algorithms for performance sensitivity estimates and on-line schemes for policy iteration methods are presented. The approach is closely related to reinforcement learning algorithms.     

Smoothed (conditional) perturbation analysis of discrete event dynamical systems

A new sample path analysis approach based on the smoothing property of conditional expectation for estimating the performance sensitivity of discrete event dynamical systems is proposed. Several examples are presented to show how this approach overcomes a difficulty of the ordinary infinitesimal perturbation analysis. The basic message is that one can get more knowledge about the system performance by observing and analyzing the sample path than by using the conventional simulation approach. It is also pointed out that the classical queueing theory approach for getting the performance sensitivity and the sample path based infinitesimal perturbation analysis approach can be unified in the framework of the new approach, the smoothed (conditional) perturbation analysis.     

Integrated Performance Models for Distributed Processing in Computer Communication Networks

This paper deals with the analysis of large-scale closed queueing network (QN) models which are used for the performance analysis of computer communication networks (CCN's). The computer systems are interconnected by a wide-area network. Users accessing local/remote computers are affected by the contention (queueing delays) at the computer systems and the communication subnet. The computational cost of analyzing such models increases exponentially with the number of user classes (chains), even when the QN is tractable (product-form). In fact, the submodels of the integrated model are generally not product-form, e.g., due to blocking at computer systems (multiprogramming level constraints) and in the communication subnet (window flow control constraints). Two approximate solution methods are proposed in this paper to analyze the integrated QN model. Both methods use decomposition and iterative techniques to exploit the structure of the QN model such that computational cost is proportional to the number of chains. The accuracy of the solution methods is validated against each other and simulation. The model is used to study the effect that channel capacity assignments, window sizes for congestion control, and routing have on system performance.     

Combining Response Surface Methodology with Numerical Methods for Optimization of Markovian Models

In general, decision support is one of the main purposes of model-based analysis of systems. Response surface methodology (RSM) is an optimization technique that has been applied frequently in practice, but few automated variants are currently available. In this paper, we show how to combine RSM with numerical analysis methods to optimize continuous time Markov chain models. Among the many known numerical solution methods for large Markov chains, we consider a Gauss-Seidel solver with relaxation that relies on a hierarchical Kronecker representation as implemented in the APNN Toolbox. To effectively apply RSM for optimizing numerical models, we propose three strategies which are shown to reduce the required number of iterations of the numerical solver. With a set of experiments, we evaluate the proposed strategies with a model of a production line and apply them to optimize a class-based queueing system.     

Markov-chain model and algorithmic procedure for the performance analysis of closed cyclic queues

This paper presents a model and an algorithmic procedure to analyze closed cyclic queues that are subject to blocking. We consider the first two moments of the processing time and present the fitting of phase-type distributions such that the number of phases and transitions is minimal. Using phase-type distributions, we enable the analysis of queueing systems with processing times with any coefficient of variation. We model the closed cyclic queues subject to blocking as continuous-time Markov chains. The implementation procedure covers the state-space generation and the determination of the infinitesimal generator matrix. Apart from rounding errors, we obtain exact results for the queueing model this way. The results are useful as reference values for the output of approximate approaches. Further, the algorithmic procedure enables a repeated analysis of different configuration alternatives as needed in optimization procedures. Though the method is very fast for small cyclic queues, it takes a long computation time for larger systems. Furthermore, the size of the queueing model to be analyzed is restricted due to the limited working memory with its present-day capacity. In a numerical study, the computation times for different configurations are investigated, limits in the size of the applicable queueing model are given, and numerical results of the performance measures are provided.     

Modelling and performance analysis of inter-processor messaging in distributed systems

This paper addresses some performance modelling issues arising from the inter-processor messaging requirements of distributed real-time processing systems. We consider two basic classes of message transfer protocols, namely clocked schemes where message transfer is initiated as a periodic timed task, and event-driven schemes where the transfer mechanism is triggered by the messaging requests themselves. The aim is to maximize system efficiency by passing messages in batches. It is shown how the classes of protocols may be modelled by a two-stage queueing systems, which is analysed using the theory of imbedded Markov chains and semi-Markov processes. The results are used to show how the important performance measures are derived, and how the protocol parameters should be chosen to optimize the overall message system performance. The methods are illustrated by numerical examples.     

Performance sensitivities for parameterized Markov systems

It is known that the performance potentials (or equivalently, perturbation realization factors) can be used as building blocks for performance sensitivities of Markov systems. In parameterized systerns, the changes in parameters may only affect some states, and the explicit transition probability matrix may not be known. In this paper, we use an example to show that we can use potentials to construct performance sensitivities m a more flexible way; only the potentials at the affected states need to be estimated, and the transition probability matrix need not be known. Policy iteration algorithms, which are simpler than the standard one, can be established.     

A new approach to the queue length and waiting time of BMAP/G/1 queues

We apply the factorization principle to derive the generating function of the queue length and the vector Laplace–Stieltjes transform of the waiting time of a BMAP/G/1 queue. The mean performance measures are provided with a computational experience.Scope and purposeThe classical method of obtaining the queue length and waiting time distributions of BMAP/G/1 queues starts with the analysis of the imbedded Markov renewal process at departure epochs. This method is intricate and time consuming when the idle period process is complicated. In this paper, we demonstrate that the factorization property can be applied efficiently and effectively to derive the queue length distributions of BMAP/G/1 queueing systems by avoiding the conventional standard procedures. The approach demonstrated in this paper can be applied to the analysis of many other BMAP/G/1 queueing systems with higher behavioral complexities.