Implications of dissipativity,inverse optimal control,and stability margins for nonlinear stochastic feedback regulators

In this paper, we derive stability margins for optimal and inverse optimal stochastic feedback regulators. Specifically, gain, sector, and disk margin guarantees are obtained for nonlinear stochastic dynamical systems controlled by nonlinear optimal and inverse optimal Hamilton‐Jacobi‐Bellman controllers that minimize a nonlinear‐nonquadratic performance criterion with cross‐weighting terms. Furthermore, using the newly developed notion of stochastic dissipativity, we derive a return difference inequality to provide connections between stochastic dissipativity and optimality of nonlinear controllers for stochastic dynamical systems. In particular, using extended Kalman‐Yakubovich‐Popov conditions characterizing stochastic dissipativity, we show that our optimal feedback control law satisfies a return difference inequality predicated on the infinitesimal generator of a controlled Markov diffusion process if and only if the controller is stochastically dissipative with respect to a specific quadratic supply rate.     

Joint production and supply control in three levels flexible manufacturing systems

This paper considers a stochastic optimal control problem of a three stages Flexible Manufacturing System. The supplier (i.e., upstream FMS) and the transformation stage (i.e., FMS) are both subject to random events. Our objective is to find a feedback control policy for the supply and production activities that minimizes the incurred cost. It is shown that the considered joint production and supply control problem is difficult to tackle using a dissociated analytical approach. A simulation based approach is thus proposed to achieve a close approximation of the optimal policy. The advantages of the approach include possible extensions after numerical characterization of the optimal control policy.     

输入随机的不可靠柔性制造系统的反馈控制

对于具有随机输入和随机需求的一类不可靠柔性制造系统,利用转移率一致化技术和随机动态规划方法,给出了输入率和服务率分配的最优反馈控制策略,指出系统的最优控制具有bang-bang形式的天关结构,数值例子验证了文中的结果。     

An equivalence result in linear-quadratic theory

We consider the zero-endpoint infinite-horizon LQ problem. We show that the existence of an optimal policy in the class of feedback controls is a sufficient condition for the existence of a stabilizing solution to the algebraic Riccati equation. This result is shown without assuming positive definiteness of the state weighting matrix. The feedback formulation of the optimization problem is natural in the context of differential games and we provide a characterization of feedback Nash equilibria both in a deterministic and stochastic context.     

The drilling problem: A stochastic modeling and control example in manufacturing

A machining economics problem is considered where feed rate selection and tool replacement policies are to be determined. A new stochastic model for tool wear, called a diffusion-threshold model, is proposed. This tool wear model allows the machining economics problem to be formulated as a stochastic optimal control problem incorporating measurement feedback of tool wear. Two types of control policies are described. One is a traditional machining economics policy and the other utilizes tool wear feedback and allows on-line decision making. The optimal policy is described for both types. An example problem based on actual data is worked out that compares the two approaches and demonstrates the utility of information feedback and on-line control.     

Linear‐Quadratic Optimal Control Problems for Mean‐Field Stochastic Differential Equations with Jumps

In this paper, we study a linear‐quadratic optimal control problem for mean‐field stochastic differential equations driven by a Poisson random martingale measure and a one‐dimensional Brownian motion. Firstly, the existence and uniqueness of the optimal control is obtained by the classic convex variation principle. Secondly, by the duality method, the optimality system, also called the stochastic Hamilton system which turns out to be a linear fully coupled mean‐field forward‐backward stochastic differential equation with jumps, is derived to characterize the optimal control. Thirdly, applying a decoupling technique, we establish the connection between two Riccati equations and the stochastic Hamilton system and then prove the optimal control has a state feedback representation.     

The replenishment policy of agri-products with stochastic demand in integrated agricultural supply chains

Market demand of agri-products is influenced by uncertain factors, such as weather, temperature, and customer preferences. In integrated agricultural supply chains, traditional inventory models are useless because of the stochastic demand and deteriorative characteristic of agri-products. This paper provides a method to determine the optimal replenishment policy of integrated agricultural supply chains with stochastic demand. In these EOQ/EPQ models, shortages are allowed and are backlogged if market demand is stochastic. The objective function is to minimize the total cost of the supply chain in the planning horizon. The total cost includes the ordering cost, the holding cost, the shortage cost and the purchasing cost. Thinking of the nonlinear relationship and dynamic forces in models, a system dynamic (SD) simulation model is constructed to find the optimal lot size and replenishment interval. Finally, an example is given to make a sensitivity analysis of the simulation model. Compared to traditional methods (such as equalize stochastic demand), the total cost decreases by 16.27% if the supply chains adopt the new replenishment policy. The results illustrated that the new replenishment policy (with intelligent method) is beneficial to help supply chain make decision scientifically. Moreover, the intelligent method can simulate stochastic demand perfectly, and it is effectively for solving the complicated and mathematically intractable replenishment problem.     

Proof of quasi-adaptivity for the m-measurement feedback class of stochastic control policies

It is shown that them-measurement feedback (mM) policy for nonlinear stochastic control, performs as well or better than the open-loop optimal policy and hence becomes one of only a few suboptimal policies presently known to be quasi-adaptive in the sense of Witsenhausen.     

An adaptive automaton controller for discrete-time markov processes

The computation of an optimal feedback controller characteristic for a non-linear stochastic system may be facilitated by the use of a stochastic automaton as a system model. A problem of particular interest is that of a long duration stationary Markov process in which the state is observable but the process dynamics and disturbance characteristics are initially unknown. The determination of a suitable control algorithm, in the form of an adaptive automation in the feedback loop, is considered in this paper for such a process.

Since the algorithm is to be used on-line to perform simultaneously the functions of estimation and control, it must constitute an efficient convergent multi-stage dual control strategy. It is shown that an existing method for dual control of a repetitive single-stage stochastic process may be extended to apply to the present case. A method is introduced of calculating successive policy estimates recursively, so that the task of updating the estimated optimal feedback policy at each stage of the process is rendered feasible. The application of the automaton controller is illustrated by the simulated adaptive control of a non-linear conditionally stable heat treatment process disturbed by multiplicative noise.     

Optimal measurement policies for control purposes

A method is presented for off-line calculation of optimal measurement subsets to be used during implementation of optimal feedback control for linear stochastic systems. It is seen that feedback control imposes a penalty on noisy measurements which must be included with measurement cost in the optimization. The combinatorial problem involved with subset selection is avoided by creating a non-linear ‘ optimal control ’ problem which can be solved using available algorithms. Results for a scalar example show that, with costly measurements, it is most economical to use a number of short measurement periods.     

Shortest path stochastic control for hybrid electric vehicles

When a hybrid electric vehicle (HEV) is certified for emissions and fuel economy, its power management system must be charge sustaining over the drive cycle, meaning that the battery state of charge (SOC) must be at least as high at the end of the test as it was at the beginning of the test. During the test cycle, the power management system is free to vary the battery SOC so as to minimize a weighted combination of fuel consumption and exhaust emissions. This paper argues that shortest path stochastic dynamic programming (SP‐SDP) offers a more natural formulation of the optimal control problem associated with the design of the power management system because it allows deviations of battery SOC from a desired setpoint to be penalized only at key off. This method is illustrated on a parallel hybrid electric truck model that had previously been analyzed using infinite‐horizon stochastic dynamic programming with discounted future cost. Both formulations of the optimization problem yield a time‐invariant causal state‐feedback controller that can be directly implemented on the vehicle. The advantages of the shortest path formulation include that a single tuning parameter is needed to trade off fuel economy and emissions versus battery SOC deviation, as compared with two parameters in the discounted, infinite‐horizon case, and for the same level of complexity as a discounted future‐cost controller, the shortest‐path controller demonstrates better fuel and emission minimization while also achieving better SOC control when the vehicle is turned off. Linear programming is used to solve both stochastic dynamic programs. Copyright © 2007 John Wiley & Sons, Ltd.     

动态电源管理的随机切换模型与在线优化

考虑系统参数未知情况下的动态电源管理问题，提出一种基于强化学习的在线策略优化算法. 通过建立事件驱动的随机切换分析模型，将动态电源管理问题转化为带约束的Markov 决策过程的策略优化问题. 利用此模型的动态结构特性，结合在线学习估计梯度与随机逼近改进策略，提出动态电源管理策略的在线优化算法．随机切换模型对电源管理系统的动态特性描述精确，在线优化算法自适应性强，运算量小，精度高，具有较高的实际应用价值．     

An application of modern control theory to a time-dependent queuing system for optimal operation

‘ Bang-bang ’ optimal closed-loop service policy for a time dependent M/M/l queuing system is derived using optimal control theory. The policy is based on probabilistic (and not stochastic) behaviour of the queue. Computational results are obtained for an illustrative example with non-bang-bang service policy using the conjugate gradient algorithm with bounded control variables. It is interesting to note that the optimal service policy is not sensitive to the arrival rate but to the mean service cost of a customer.     

一类双线性随机系统M量测不定LQ控制

研究了带有乘性噪声和受扰动观测的离散时间随机系统不定线性二次(Linear quadratic, LQ) 最优输出反馈控制问题. 对此类问题而言,二次成本函数的加权矩阵不定号,并且最优控制具有对偶效果.为在最优性和计算复杂度间 进行折衷,本文采用了一种M量测反馈控制设计方法.基于动态规划方法,将未来的测量结合到当前控制 计算当中的M量测反馈控制可以通过倒向求解一类与原系统维数相同的广义差分Riccati方程(Generalized difference Riccati equation, GDRE)得到.仿真结果 表明本文提出的算法与目前普遍采用的确定等价性方法相比具有优越性.     

The mean squared loss control problem for a partially observed Markov chain

In this article, we study the optimal control of a partially observed Markov chain for which a mean squared cost functional is minimised. Both the terminal cost and the running cost are considered. Minimum principles are established. In both cases, if the optimal control is Markov feedback, more explicit forms for the stochastic integrands and adjoint processes are obtained.     

输入随机的不可靠串行生产线的反馈控制

对于具有随机输入和随机需求的一类串行生产纡系统,利用转移率一致化技术和随机动态规划方法,给出了输入率和服务率分配的最优反馈控制策略,指出 最优控制具有邦一邦形式的开关结构。仿真例子验证了方法的有效性。     

Optimal production control in a discrete manufacturing system withunreliable machines and random demands

We consider a production control problem in a manufacturing system with a failure-prone machine and a stochastic demand. The objective is to minimize a discounted inventory holding and backlog cost over an infinite planning horizon. The optimal production control of continuous, stochastic manufacturing systems with a failure-prone machine and a constant demand has been considered in Akella and Kumar (1986). However, the problem of optimal production control for discrete stochastic manufacturing systems with uncertain demands remains open. In this paper, we investigate a case where the exogenous demand forms a homogeneous Poisson flow. Primarily, we show that the optimal production control for such a system is of the threshold control type. In addition, the explicit form of production control policy and the objective functions are provided. Numerical examples are included to demonstrate the results obtained in the paper and to compare with the one in Akella and Kumar     

Optimum checking request policies

We discuss checking policies, in which an inspector is requested to perform the check, for a ono-unit system. Applying the unique modifications of Markov renewal processes, we analyse the stochastic behaviour of each CR policy (checking request policy) and CRSL policy (checking request and surveillance limit policy), where it is assumed that the failure time for each unit and the inspector's arrival time havo arbitrary distributions. Introducing replacement eosts, the request for the inspector to perform the check, system failure and surveillance, we derive the optimal policies which maximize the cost effectiveness under suitablo conditions for each model. In particular, wo discuss the optimal policies as a non-linear programming problem with two variables.     

不确定线性时滞随机系统的最优保性能控制

对一类具有状态时滞的不确定线性随机系统,研究了保性能状态反馈控制律的设计问题。采用线性矩阵不等式方法和伊藤公式,导出了保性能控制律的存在条件。进而,通过求解一个线性矩阵不等式约束的凸优化问题,提出了最优保性能控制律设计方法。最后用数值例子说明了该方法的有效性。     

Finite-dimensional optimal controllers for nonlinear plants

Optimal risk sensitive feedback controllers are now available for very general stochastic nonlinear plants and performance indices. They consist of nonlinear static feedback of so called information states from an information state filter. In general, these filters are linear, but infinite dimensional, and the information state feedback gains are derived from (doubly) infinite dimensional dynamic programming. The challenge is to achieve optimal finite dimensional controllers using finite dimensional calculations for practical implementation.This paper derives risk sensitive optimality results for finite-dimensional controllers. The controllers can be conveniently derived for ‘linearized’ (approximate) models (applied to nonlinear stochastic systems). Performance indices for which the controllers are optimal for the nonlinear plants are revealed. That is, inverse risk-sensitive optimal control results for nonlinear stochastic systems with finite dimensional linear controllers are generated. It is instructive to see from these results that as the nonlinear plants approach linearity, the risk sensitive finite dimensional controllers designed using linearized plant models and risk sensitive indices with quadratic cost kernels, are optimal for a risk sensitive cost index which approaches one with a quadratic cost kernel. Also even far from plant linearity, as the linearized model noise variance becomes suitably large, the index optimized is dominated by terms which can have an interesting and practical interpretation.Limiting versions of the results as the noise variances approach zero apply in a purely deterministic nonlinear H_∞ setting. Risk neutral and continuous-time results are summarized.More general indices than risk sensitive indices are introduced with the view to giving useful inverse optimal control results in non-Gaussian noise environments.