Approximations for optimal stopping of a piecewise-deterministic process

This paper deals with approximation techniques for the optimal stopping of a piecewise-deterministic Markov process (P.D.P.). Such processes consist of a mixture of deterministic motion and random jumps. In the first part of the paper (Section 3) we study the optimal stopping problem with lower semianalytic gain function; our main result is the construction of ε-optimal stopping times. In the second part (Section 4) we consider a P.D.P. satisfying some smoothness conditions, and forN integer we construct a discretized P.D.P. which retains the main characteristics of the original process. By iterations of the single jump operator from ℝ ^N to ℝ ^N , each iteration consisting ofN one-dimensional minimizations, we can calculate the payoff function of the discretized process. We demonstrate the convergence of the payoff functions, and for the case when the state space is compact we construct ε-optimal stopping times for the original problem using the payoff function of the discretized problem. A numerical example is presented.     

Optimal control problems with multiple characteristic time points in the objective and constraints

In this paper, we develop a computational method for a class of optimal control problems where the objective and constraint functionals depend on two or more discrete time points. These time points can be either fixed or variable. Using the control parametrization technique and a time scaling transformation, this type of optimal control problem is approximated by a sequence of approximate optimal parameter selection problems. Each of these approximate problems can be viewed as a finite dimensional optimization problem. New gradient formulae for the cost and constraint functions are derived. With these gradient formulae, standard gradient-based optimization methods can be applied to solve each approximate optimal parameter selection problem. For illustration, two numerical examples are solved.     

Design of optimal controllers for spatially invariant systems with finite communication speed

We consider the problem of designing optimal distributed controllers whose impulse response has limited propagation speed. We introduce a state-space framework in which all spatially invariant systems with this property can be characterized. After establishing the closure of such systems under linear fractional transformations, we formulate the H₂ optimal control problem using the model-matching framework. We demonstrate that, even though the optimal control problem is non-convex with respect to some state-space design parameters, a variety of numerical optimization algorithms can be employed to relax the original problem, thereby rendering suboptimal controllers. In particular, for the case in which every subsystem has scalar input disturbance, scalar measurement, and scalar actuation signal, we investigate the application of the Steiglitz–McBride, Gauss–Newton, and Newton iterative schemes to the optimal distributed controller design problem. We apply this framework to examples previously considered in the literature to demonstrate that, by designing structured controllers with infinite impulse response, superior performance can be achieved compared to finite impulse response structured controllers of the same temporal degree.     

Consistent approximation of a nonlinear optimal control problem with uncertain parameters

This paper focuses on a non-standard constrained nonlinear optimal control problem in which the objective functional involves an integration over a space of stochastic parameters as well as an integration over the time domain. The research is inspired by the problem of optimizing the trajectories of multiple searchers attempting to detect non-evading moving targets. In this paper, we propose a framework based on the approximation of the integral in the parameter space for the considered uncertain optimal control problem. The framework is proved to produce a zeroth-order consistent approximation in the sense that accumulation points of a sequence of optimal solutions to the approximate problem are optimal solutions of the original problem. In addition, we demonstrate the convergence of the corresponding adjoint variables. The accumulation points of a sequence of optimal state-adjoint pairs for the approximate problem satisfy a necessary condition of Pontryagin Minimum Principle type, which facilitates assessment of the optimality of numerical solutions.     

State waypoint approach to continuous‐time nonlinear optimal control problems

In this paper, we propose an optimal control technique for a class of continuous‐time nonlinear systems. The key idea of the proposed approach is to parametrize continuous state trajectories by sequences of a finite number of intermediate target states; namely, waypoint sequences. It is shown that the optimal control problem for transferring the state from one waypoint to the next is given an explicit‐form suboptimal solution, by means of linear approximation. Thus the original continuous‐time nonlinear control problem reduces to a finite‐dimensional optimization problem of waypoint sequences. Any efficient numerical optimization method, such as the interior‐reflection Newton method, can be applied to solve this optimization problem. Finally, we solve the optimal control problem for a simple nonlinear system example to illustrate the effectiveness of this approach. Copyright © 2009 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

Optimally switched linear systems

In this paper we address the problem of optimal switching for switched linear systems. The uniqueness of our approach lies in describing the switching action by multiple control inputs. This allows us to embed the switched system in a larger family of systems and apply Pontryagin’s Minimum Principle for solving the optimal control problem. This approach imposes no restriction on the switching sequence or the number of switchings. This is in contrast to search based algorithms where a fixed number of switchings is set a priori. In our approach, the optimal solution can be determined by solving the ensuing two-point boundary value problem. Results of numerical simulations are provided to support the proposed method.     

Adaptive dynamic programming for linear impulse systems

We investigate the optimization of linear impulse systems with the reinforcement learning based adaptive dynamic programming （ADP） method. For linear impulse systems, the optimal objective function is shown to be a quadric form of the pre-impulse states. The ADP method provides solutions that iteratively converge to the optimal objective function. If an initial guess of the pre-impulse objective function is selected as a quadratic form of the pre-impulse states, the objective function iteratively converges to the optimal one through ADP. Though direct use of the quadratic objective function of the states within the ADP method is theoretically possible, the numerical singularity problem may occur due to the matrix inversion therein when the system dimensionality increases. A neural network based ADP method can circumvent this problem. A neural network with polynomial activation functions is selected to approximate the pr~impulse objective function and trained iteratively using the ADP method to achieve optimal control. After a successful training, optimal impulse control can be derived. Simulations are presented for illustrative purposes.     

Optimal control of switching time in switched stochastic systems with multi-switching times and different costs

In this paper, we solve an optimal control problem for a class of time-invariant switched stochastic systems with multi-switching times, where the objective is to minimise a cost functional with different costs defined on the states. In particular, we focus on problems in which a pre-specified sequence of active subsystems is given and the switching times are the only control variables. Based on the calculus of variation, we derive the gradient of the cost functional with respect to the switching times on an especially simple form, which can be directly used in gradient descent algorithms to locate the optimal switching instants. Finally, a numerical example is given, highlighting the validity of the proposed methodology.     

Optimal Control of Switching Times in Switched Stochastic Systems

In this paper, we solve an optimal control problem for switched stochastic systems using calculus of variations, where the objective is to minimize a cost functional defined on the state and the switching times are the sole control variables. In particular, we focus on the problem in which a pre‐specified sequence of active subsystems is given. For one switching time case, the derivative of the cost functional with respect to the switching time is derived, which has an especially simple form and can be directly used in gradient descent algorithms to locate the optimal switching instant. Then, we propose an approach to deal with the problem with multi‐switching times case. Finally, two numerical examples are given, highlighting the viability and advantages of the proposed methodology.     

A perspective-based convex relaxation for switched-affine optimal control

We consider the switched-affine optimal control problem, i.e., the problem of selecting a sequence of affine dynamics from a finite set in order to minimize a sum of convex functions of the system state. We develop a new reduction of this problem to a mixed-integer convex program (MICP), based on perspective functions. Relaxing the integer constraints of this MICP results in a convex optimization problem, whose optimal value is a lower bound on the original problem value. We show that this bound is at least as tight as similar bounds obtained from two other well-known MICP reductions (via conversion to a mixed logical dynamical system, and by generalized disjunctive programming), and our numerical study indicates it is often substantially tighter. Using simple integer-rounding techniques, we can also use our formulation to obtain an upper bound (and corresponding sequence of control inputs). In our numerical study, this bound was typically within a few percent of the optimal value, making it attractive as a stand-alone heuristic, or as a subroutine in a global algorithm such as branch and bound. We conclude with some extensions of our formulation to problems with switching costs and piecewise affine dynamics.     

Maximizing biogas production from the anaerobic digestion

This paper presents an optimal control law policy for maximizing biogas production of anaerobic digesters. In particular, using a simple model of the anaerobic digestion process, we derive a control law to maximize the biogas production over a period T using the dilution rate as the control variable. Depending on initial conditions and constraints on the actuator (the dilution rate D(·)), the search for a solution to the optimal control problem reveals very different levels of difficulty. In the present paper, we consider that there are no severe constraints on the actuator. In particular, the interval in which the input flow rate lives includes the value which allows the biogas to be maximized at equilibrium. For this case, we solve the optimal control problem using classical tools of differential equations analysis. Numerical simulations illustrate the robustness of the control law with respect to several parameters, notably with respect to initial conditions. We use these results to show that the heuristic control law proposed by Steyer et al., 1999 [20] is optimal in a certain sense. The optimal trajectories are then compared with those given by a purely numerical optimal control solver (i.e. the “BOCOP” toolkit) which is an open-source toolbox for solving optimal control problems. When the exact analytical solution to the optimal control problem cannot be found, we suggest that such numerical tool can be used to intuiter optimal solutions.     

Solution for state constrained optimal control problems applied to power split control for hybrid vehicles

This paper presents a numerical solution for scalar state constrained optimal control problems. The algorithm rewrites the constrained optimal control problem as a sequence of unconstrained optimal control problems which can be solved recursively as a two point boundary value problem. The solution is obtained without quantization of the state and control space. The approach is applied to the power split control for hybrid vehicles for a predefined power and velocity trajectory and is compared with a Dynamic Programming solution. The computational time is at least one order of magnitude less than that for the Dynamic Programming algorithm for a superior accuracy.     

Annihilator structure of a principal ideal : Relation to optimal compensators

Given a linear, time-invariant, discrete-time plant, we consider the optimal control problem of minimizing, by choice of a stabilizing compensator, the seminorm of a selected closed-loop map in a basic feedback system. The seminorm can be selected to reflect any chosen performance feature and must satisfy only a mild condition concerning finite impulse responses. We show that if the plant has no poles or zeros on the unit circle, then the calculation of the minimum achievable seminorm is equivalent to the maximization of a linear objective over a convex set in a low-dimensional Euclidean space. Hence, for a wide variety of optimal control problems, one can compute the answer to an infinite-dimensional optimization by a finite-dimensional procedure. This allows the use of effective numerical methods for computation.     

Numerical solution of a conspicuous consumption model with constant control delay

We derive optimal pricing strategies for conspicuous consumption products in periods of recession. To that end, we formulate and investigate a two-stage economic optimal control problem that takes uncertainty of the recession period length and delay effects of the pricing strategy into account.This non-standard optimal control problem is difficult to solve analytically, and solutions depend on the variable model parameters. Therefore, we use a numerical result-driven approach. We propose a structure-exploiting direct method for optimal control to solve this challenging optimization problem. In particular, we discretize the uncertainties in the model formulation by using scenario trees and target the control delays by introduction of slack control functions.Numerical results illustrate the validity of our approach and show the impact of uncertainties and delay effects on optimal economic strategies. During the recession, delayed optimal prices are higher than the non-delayed ones. In the normal economic period, however, this effect is reversed and optimal prices with a delayed impact are smaller compared to the non-delayed case.     

The FEM optimization of active vibration control in structures by solving the corresponding two-point-boundary-value problem

An algorithm for solving optimal active vibration control problems by the finite element method (FEM) is presented. The optimality equations for the problem are derived from Pontryagin’s principle in the form of a set of the fourth order ordinary differential equations that, together with the initial and final boundary conditions, constitute the boundary value problem in the time domain, which in control is referred to as a two-point-boundary-value problem. These equations decouple in the modal space and can be solved by the FEM technique. An analogy between the optimality equations and the governing equations for a set of certain static beams permits obtaining numerical solutions to the optimal control problem with the help of standard ‘structural’ FEM software. The optimal action of actuators is automatically calculated by applying the independent modal space control concept. The structure’s response to actuation forces is also determined and can independently be verified for spillover effects. As an illustration, the algorithm is used for the analysis of optimal action of actuators to attenuate vibrations of an elastic fin.     

Optimality of affine control system of several species in competition on a sequential batch reactor

In this paper, we analyse the optimality of affine control system of several species in competition for a single substrate on a sequential batch reactor, with the objective being to reach a given (low) level of the substrate. We allow controls to be bounded measurable functions of time plus possible impulses. A suitable modification of the dynamics leads to a slightly different optimal control problem, without impulsive controls, for which we apply different optimality conditions derived from Pontryagin principle and the Hamilton–Jacobi–Bellman equation. We thus characterise the singular trajectories of our problem as the extremal trajectories keeping the substrate at a constant level. We also establish conditions for which an immediate one impulse (IOI) strategy is optimal. Some numerical experiences are then included in order to illustrate our study and show that those conditions are also necessary to ensure the optimality of the IOI strategy.     

Suboptimal control for nonlinear systems: a successive approximation approach

This paper presents a successive approximation approach (SAA) designing optimal controllers for a class of nonlinear systems with a quadratic performance index. By using the SAA, the nonlinear optimal control problem is transformed into a sequence of nonhomogeneous linear two-point boundary value (TPBV) problems. The optimal control law obtained consists of an accurate linear feedback term and a nonlinear compensation term which is the limit of an adjoint vector sequence. By using the finite-step iteration of the nonlinear compensation sequence, we can obtain a suboptimal control law. Simulation examples are employed to test the validity of the SAA.     

Time-optimal control of a self-propelled particle in a spatiotemporal flow field

We address a minimum-time problem that constitutes an extension of the classical Zermelo navigation problem in higher dimensions. In particular, we address the problem of steering a self-propelled particle to a prescribed terminal position with free terminal velocity in the presence of a spatiotemporal flow field. Furthermore, we assume that the norm of the rate of change of the particle's velocity relative to the flow is upper bounded by an explicit upper bound. To address the problem, we first employ Pontryagin's minimum principle to parameterise the set of candidate time-optimal control laws in terms of a parameter vector that belongs to a compact set. Subsequently, we develop a simple numerical algorithm for the computation of the minimum time-to-come function that is tailored to the particular parametrisation of the set of the candidate time-optimal control laws of our problem. The proposed approach bypasses the task of converting the optimal control problem to a parameter optimisation problem, which can be computationally intense, especially when one is interested in characterising the optimal synthesis of the minimum-time problem. Numerical simulations that illustrate the theoretical developments are presented.     

Optimal design of 2D conducting graded materials by minimizing quadratic functionals in the field

We explore a typical optimal design problem in 2D conductivity: given fixed amounts of two isotropic dielectric materials, decide how we are to mix them in a 2D domain so as to minimize a certain cost functional depending on the underlying electric field. We rely on a reformulation of the optimal design problem as a vector variational problem and examine its relaxation, taking advantage of the explicit formulae for the relaxed integrands recently computed in Pedregal (2003). We provide numerical evidence, based on our relaxation, that Tartar’s result Tartar (1994) is verified when the target field is zero (also for divergence-free fields) and optimal solutions are given by first-order laminates. This same evidence also holds for a general quadratic functional in the field.     

A generalized multi-period mean-variance portfolio optimization with Markov switching parameters

In this paper, we deal with a generalized multi-period mean-variance portfolio selection problem with market parameters subject to Markov random regime switchings. Problems of this kind have been recently considered in the literature for control over bankruptcy, for cases in which there are no jumps in market parameters (see [Zhu, S. S., Li, D., & Wang, S. Y. (2004). Risk control over bankruptcy in dynamic portfolio selection: A generalized mean variance formulation. IEEE Transactions on Automatic Control, 49, 447-457]). We present necessary and sufficient conditions for obtaining an optimal control policy for this Markovian generalized multi-period mean-variance problem, based on a set of interconnected Riccati difference equations, and on a set of other recursive equations. Some closed formulas are also derived for two special cases, extending some previous results in the literature. We apply the results to a numerical example with real data for risk control over bankruptcy in a dynamic portfolio selection problem with Markov jumps selection problem.