Perturbation realization, potentials, and sensitivity analysis ofMarkov processes

Two fundamental concepts and quantities, realization factors and performance potentials, are introduced for Markov processes. The relations among these two quantities and the group inverse of the infinitesimal generator are studied. It is shown that the sensitivity of the steady-state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a Markov process. Based on these results, algorithms for estimating performance sensitivities on a single sample path of a Markov process can be proposed. The potentials in this paper are defined through realization factors and are shown to be the same as those defined by Poisson equations. The results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis (IPA) and non-IPA approaches to the sensitivity analysis of steady-state performance; they also provide a theoretical background for the PA algorithms developed in recent years     

A time aggregation approach to Markov decision processes

We propose a time aggregation approach for the solution of infinite horizon average cost Markov decision processes via policy iteration. In this approach, policy update is only carried out when the process visits a subset of the state space. As in state aggregation, this approach leads to a reduced state space, which may lead to a substantial reduction in computational and storage requirements, especially for problems with certain structural properties. However, in contrast to state aggregation, which generally results in an approximate model due to the loss of Markov property, time aggregation suffers no loss of accuracy, because the Markov property is preserved. Single sample path-based estimation algorithms are developed that allow the time aggregation approach to be implemented on-line for practical systems. Some numerical and simulation examples are presented to illustrate the ideas and potential computational savings.     

Ill-conditioned performance functions of queueing systems

Shows that for queueing networks with deterministic or discrete service time distributions, the performance functions can be nondifferentiable at a dense subset of a given interval. The authors also show that when the service time densities are supported on small intervals, the performance function derivatives changes rapidly. The authors prove these results for a two-server cyclic network and then point out a potential generality to other queueing networks. The results indicate that the nonsmooth analysis may be useful in the area of stochastic discrete-event systems     

Stochastic learning and optimization—A sensitivity-based approach

We introduce a sensitivity-based view to the area of learning and optimization of stochastic dynamic systems. We show that this sensitivity-based view provides a unified framework for many different disciplines in this area, including perturbation analysis, Markov decision processes, reinforcement learning, identification and adaptive control, and singular stochastic control; and that this unified framework applies to both the discrete event dynamic systems and continuous-time continuous-state systems. Many results in these disciplines can be simply derived and intuitively explained by using two performance sensitivity formulas. In addition, we show that this sensitivity-based view leads to new results and opens up new directions for future research. For example, the n th bias optimality of Markov processes has been established and the event-based optimization may be developed; this approach has computational and other advantages over the state-based approaches.     

InPBi禁带下红外光致发光效率的Bi组分依赖研究

稀Bi半导体InPBi的光致发光（Photoluminescence, PL）主要来自缺陷能级跃迁过程,具有红外长波长、大线宽和高辐射强度等特点,因而引发广泛兴趣。针对InPBi的红外发光效率问题,本文研究了不同Bi组分InPBi的激发功率依赖红外PL光谱演化规律。实验发现,随着Bi组分增大,PL线型发生显著变化,导致发光波长总体红移;同时激发功率依赖的PL积分强度演化分析表明,发光效率随Bi组分先增大然后下降,在0.5%组分时发光效率达到峰值。发光效率增大一方面归因于Bi捕获空穴降低非辐射复合,另一方面来自Bi的表面活性剂效应;而高Bi组分引入过多缺陷从而抑制了Bi的优势,导致发光效率下降。这些结果或有助于理解InPBi的红外发射性能,表明InPBi具有红外光电子应用前景。     

Stochastic control via direct comparison

The standard approach to stochastic control is dynamic programming. In this paper, we introduce an alternative approach based on direct comparison of the performance of any two policies. This is achieved by modeling the state process as a continuous-time and continuous-state Markov process and applying the same ideas as for the discrete-time and discrete-state case. This approach is simple and intuitively clear; it applies to different problems with, finite and infinite horizons, discounted and long-run-average performance, continuous and jump diffusions, in the same way. Discounting is not needed when dealing with long-run average performance. The approach provides a unified framework for stochastic control and other optimization theory and methodologies, including Markov decision processes, perturbation analysis, and reinforcement learning.     

Potential-based online policy iteration algorithms for Markov decision processes

Performance potentials play a crucial role in performance sensitivity analysis and policy iteration of Markov decision processes. The potentials can be estimated on a single sample path of a Markov process. In this paper, we propose two potential-based online policy iteration algorithms for performance optimization of Markov systems. The algorithms are based on online estimation of potentials and stochastic approximation. We prove that with these two algorithms the optimal policy can be attained after a finite number of iterations. A simulation example is given to illustrate the main ideas and the convergence rates of the algorithms.     

Stochastic learning and optimization — Ideas vs mathematics?

What are the roles that ideas and mathematics play in research of engineering subjects? This article tries to answer this question with the author’s own research experience. In the past 30 years, the author’s research started from perturbation analysis (PA) of queueing networks, to PA of Markov systems, to Markov decision processes (MDP), and to stochastic control; and based on these research, the author has successfully developed a sensitivity-based optimization approach to the area of learning and optimization of stochastic systems, leading to a simple and unified framework for the area, new research directions, and new results in the area. This paper reviews the above research topics and the history of their development with an emphasis on what the roles that ideas and mathematics play in each of the advances along the path.     

沙漠车用陶瓷活塞顶的研制

氮化硅陶瓷活塞顶是无冷机的关键部件之一．本工作选用高熔点的稀土氧化物为添加剂,在气氛加压炉内1800～1960℃,0．1～6MPaN₂压力下进行烧结,制备了性能优越的氮化硅陶瓷材料．其常温抗弯强度为750MPa,断裂韧性为7．2MPa·m^1/2,Weibull模数为12．3．使用该材料制备的氮化硅陶瓷活塞顶,装在6105型无水冷柴油机中,在规定的条件下进行耐久考核,已完成700h以上的台架实验,氮化硅陶瓷活塞顶装在EQ2060沙漠车上进行道路试验,已行驶1900km以上,经检验氮化硅陶瓷活塞顶完好无损．