Generalized hamilton-jacobi-bellman formulation -based neural network control of affine nonlinear discrete-time systems

In this paper, we consider the use of nonlinear networks towards obtaining nearly optimal solutions to the control of nonlinear discrete-time (DT) systems. The method is based on least squares successive approximation solution of the generalized Hamilton-Jacobi-Bellman (GHJB) equation which appears in optimization problems. Successive approximation using the GHJB has not been applied for nonlinear DT systems. The proposed recursive method solves the GHJB equation in DT on a well-defined region of attraction. The definition of GHJB, pre-Hamiltonian function, HJB equation, and method of updating the control function for the affine nonlinear DT systems under small perturbation assumption are proposed. A neural network (NN) is used to approximate the GHJB solution. It is shown that the result is a closed-loop control based on an NN that has been tuned a priori in offline mode. Numerical examples show that, for the linear DT system, the updated control laws will converge to the optimal control, and for nonlinear DT systems, the updated control laws will converge to the suboptimal control.     

Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses

In this paper, the stochastic optimal control of linear networked control system (NCS) with uncertain system dynamics and in the presence of network imperfections such as random delays and packet losses is derived. The proposed stochastic optimal control method uses an adaptive estimator (AE) and ideas from Q-learning to solve the infinite horizon optimal regulation of unknown NCS with time-varying system matrices. Next, a stochastic suboptimal control scheme which uses AE and Q-learning is introduced for the regulation of unknown linear time-invariant NCS that is derived using certainty equivalence property. Update laws for online tuning the unknown parameters of the AE to obtain the Q-function are derived. Lyapunov theory is used to show that all signals are asymptotically stable (AS) and that the estimated control signals converge to optimal or suboptimal control inputs. Simulation results are included to show the effectiveness of the proposed schemes. The result is an optimal control scheme that operates forward-in-time manner for unknown linear systems in contrast with standard Riccati equation-based schemes which function backward-in-time.     

Reinforcement learning controller design for affine nonlinear discrete-time systems using online approximators

In this paper, reinforcement learning state- and output-feedback-based adaptive critic controller designs are proposed by using the online approximators (OLAs) for a general multi-input and multioutput affine unknown nonlinear discretetime systems in the presence of bounded disturbances. The proposed controller design has two entities, an action network that is designed to produce optimal signal and a critic network that evaluates the performance of the action network. The critic estimates the cost-to-go function which is tuned online using recursive equations derived from heuristic dynamic programming. Here, neural networks (NNs) are used both for the action and critic whereas any OLAs, such as radial basis functions, splines, fuzzy logic, etc., can be utilized. For the output-feedback counterpart, an additional NN is designated as the observer to estimate the unavailable system states, and thus, separation principle is not required. The NN weight tuning laws for the controller schemes are also derived while ensuring uniform ultimate boundedness of the closed-loop system using Lyapunov theory. Finally, the effectiveness of the two controllers is tested in simulation on a pendulum balancing system and a two-link robotic arm system.     

Parallel imports and unparallel laws: an examination of the exhaustion doctrine through the lens of pharmaceutical products

Parallel imports, the natural consequence of exhaustion doctrine, represent a complex interaction between the issue of free flow of international trade and the protection of intellectual property rights. There is considerable divergence among scholars, both economic and legal, about the need for harmonisation of principles of exhaustion, and consequently parallel import laws. In this article, we examine the need for harmonisation of parallel import laws through the lens of pharmaceutical products. We highlight the necessity for the affirmative norm of exhaustion doctrine in the Trade Related Intellectual Property Rights (TRIPS) Agreement and suggest amendment to TRIPS Agreement Article 6 mandating international exhaustion doctrine as international legal standard with limited exceptions as an intermediate approach to strike a balance between the interests of the IP owners and consumers, and concurrently addressing the concerns of developed and developing countries.     

Randomized leader election

We present an efficient randomized algorithm for leader election in large-scale distributed systems. The proposed algorithm is optimal in message complexity (O(n) for a set of n total processes), has round complexity logarithmic in the number of processes in the system, and provides high probabilistic guarantees on the election of a unique leader. The algorithm relies on a balls and bins abstraction and works in two phases. The main novelty of the work is in the first phase where the number of contending processes is reduced in a controlled manner. Probabilistic quorums are used to determine a winner in the second phase. We discuss, in detail, the synchronous version of the algorithm, provide extensions to an asynchronous version and examine the impact of failures.     

Neural Network Controller Development and Implementation for Spark Ignition Engines With High EGR Levels

Past research has shown substantial reductions in the oxides of nitrogen (NO_x) concentrations by using 10% -25% exhaust gas recirculation (EGR) in spark ignition (SI) engines (see Dudek and Sain, 1989). However, under high EGR levels, the engine exhibits strong cyclic dispersion in heat release which may lead to instability and unsatisfactory performance preventing commercial engines to operate with high EGR levels. A neural network (NN)-based output feedback controller is developed to reduce cyclic variation in the heat release under high levels of EGR even when the engine dynamics are unknown by using fuel as the control input. A separate control loop was designed for controlling EGR levels. The stability analysis of the closed-loop system is given and the boundedness of the control input is demonstrated by relaxing separation principle, persistency of excitation condition, certainty equivalence principle, and linear in the unknown parameter assumptions. Online training is used for the adaptive NN and no offline training phase is needed. This online learning feature and model-free approach is used to demonstrate the applicability of the controller on a different engine with minimal effort. Simulation results demonstrate that the cyclic dispersion is reduced significantly using the proposed controller when implemented on an engine model that has been validated experimentally. For a single cylinder research engine fitted with a modern four-valve head (Ricardo engine), experimental results at 15% EGR indicate that cyclic dispersion was reduced 33% by the controller, an improvement of fuel efficiency by 2%, and a 90% drop in NO_x from stoichiometric operation without EGR was observed. Moreover, unburned hydrocarbons (uHC) drop by 6% due to NN control as compared to the uncontrolled scenario due to the drop in cyclic dispersion. Similar performance was observed with the controller on a different engine.     

An assessment of the BWR fuel box-lattice-cell homogenization procedures with the Tarapur BWR incore and excore measurements

A code system has been developed to provide the incorefuel-management guidelines to the Tarapur BWR reactors. Constant checking of the design calculational methods is rendered possible by the steady flow of operating data from the Tarapur units over the last few cycles. The operating data include cold/hot criticals and detailed flux/power maps. Besides these, the burnups and isotopic composition of a few irradiated fuel pins obtained by mass-spectrometric analyses, are also available for validation of the BWR core and lattice-cell modelling.The calculated eigen values for different power levels and at different core average burnups are found to have a spread of less than 0.25% ΔK. Analyses of a number of TIP measurements show that the core power distribution can be predicted in a satisfactory manner for uncontrolled fuel bundles and non-peripheral fuel assemblies (<10%). For prediction of cold-criticals the void-history effects are found to be unimportant.The pin burnups and isotopic densities of important U and Pu isotopes relative to ²³⁸U have been compared with mass-spectrometric measurements. The pin-burnup profile comparison is found to be good for fuel pins, which are not near water gaps. Deviation histograms of various isotopes are presented in this paper. ²³⁵U is predicted within ± 3% (r.m.s.). The total Pu is overpredicted by 5–8%, while the quality of Pu is predicted within ± 1.0% (r.m.s.).     

Evaluation of the finite-element-synthesis model using the 3-D finite-element technique

For the solution of multi group diffusion theory equations a 3-D finite-element (FE) code finerc has been developed. 3-D element shapes with the base orthogonal to the third direction are considered in finerc. For these elements the 3-D element submatrices of the FE formulation are easily computed in terms of the corresponding lower-dimensional element submatrices. The 3-D problems are tackled in finerc with a degree of complexity equivalent to that of 2-D problems.The 3-D FE technique is still somewhat expensive for routine design computations. The method can, however, be used for assessing the accuracy of other faster calculational methods. In this paper we have compared the results of the 3-D FE method with those of the FE-synthesis method which was previously developed by the present author. It is noted that the FE-synthesis method gives, at negligible computational cost, accurate eigenvalue estimates and reasonably good predictions of reactor core power profiles for the 3-D benchmark problems. The FE-synthesis method may be used for a number of survey-type analyses with occasional counter-checking by the 3-D FE technique.