Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations

ABSTRACT

Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the limited-memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) update typically do not require manually tuning hyper-parameters but suffer from approximating a potentially indefinite Hessian with a positive-definite matrix. Hessian-free methods leverage the ability to perform Hessian-vector multiplication without needing the entire Hessian matrix, but each iteration's complexity is significantly greater than quasi-Newton methods. In this paper we propose an alternative approach for solving ML problems based on a quasi-Newton trust-region framework for solving large-scale optimization problems that allow for indefinite Hessian approximations. Numerical experiments on a standard testing data set show that with a fixed computational time budget, the proposed methods achieve better results than the traditional limited-memory BFGS and the Hessian-free methods.     

STABILITY ANALYSIS: AN APPROACH TO THE EVALUATION OF SYSTEM DESIGN

Abstract

The performance of most modern day, large-scale systems is defined in terms of their degree of achievement for various measures of effectiveness (e.g., cost, power output, environmental impact, etc.). These measures, in turn, are often derived from multiple, conflicting, and (usually) nonlinear functions which serve to reflect the system's design specifications. Considerable effort has been devoted to the development of various design tools, both conventional (i.e., single objective models) and multiobjective (e.g., utility theory, goal programming), but far less attention seems to have been paid to the development of a practical methodology for system sensitivity analysis. Unfortunately, in real world problems, the sensitivity of the system with respect to variations in system parameters may be as important, or more so, than just the determination of the static, “optimal” design. In this paper we deal with this aspect of system design and demonstrate our proposed approach via actual implementation to the specific design problem associated with beam pattern forming in phased antenna arrays.     

Different Zhang functions leading to different ZNN models illustrated via time-varying matrix square roots finding

In view of the great potential in parallel processing and ready implementation via hardware, neural networks are now often employed to solve online nonlinear matrix equation problems. Recently, a novel class of neural networks, termed Zhang neural network (ZNN), has been formally proposed by Zhang et al. for solving online time-varying problems. Such a neural-dynamic system is elegantly designed by defining an indefinite matrix-valued error-monitoring function, which is called Zhang function (ZF). The dynamical system is then cast in the form of a first-order differential equation by using matrix notation. In this paper, different indefinite ZFs, which lead to different ZNN models, are proposed and developed as the error-monitoring functions for time-varying matrix square roots finding. Towards the final purpose of field programmable gate array (FPGA) and application-specific integrated circuit (ASIC) realization, the MATLAB Simulink modeling and verifications of such ZNN models are further investigated for online solution of time-varying matrix square roots. Both theoretical analysis and modeling results substantiate the efficacy of the proposed ZNN models for time-varying matrix square roots finding.     

A novel multi-objective immune algorithm with a decomposition-based clonal selection

In recent years, a number of multi-objective immune algorithms (MOIAs) have been proposed as inspired by the information processing in biologic immune system. Since most MOIAs encourage to search around some boundary and less-crowded areas using the clonal selection principle, they have been validated to show the effectiveness on tackling various kinds of multi-objective optimization problems (MOPs). The crowding distance metric is often used in MOIAs as a diversity metric to reflect the status of population’s diversity, which is employed to clone less-crowded individuals for evolution. However, this kind of cloning may encounter some difficulties when tackling some complicated MOPs (e.g., the UF problems with variable linkages). To alleviate the above difficulties, a novel MOIA with a decomposition-based clonal selection strategy (MOIA-DCSS) is proposed in this paper. Each individual is associated to one subproblem using the decomposition approach and then the performance enhancement on each subproblem can be easily quantified. Then, a novel decomposition-based clonal selection strategy is designed to clone the solutions with the larger improvements for the subproblems, which encourages to search around these subproblems. Moreover, differential evolution is employed in MOIA-DCSS to strength the exploration ability and also to improve the population’s diversity. To evaluate the performance of MOIA-DCSS, twenty-eight test problems are used with the complicated Pareto-optimal sets and fronts. The experimental results validate the superiority of MOIA-DCSS over four state-of-the-art multi-objective algorithms (i.e., NSLS, MOEA/D-M2M, MOEA/D-DRA and MOEA/DD) and three competitive MOIAs (i.e., NNIA, HEIA, and AIMA).     

互联电力系统鲁棒分布式模型预测负荷频率控制

负荷频率控制是现代互联电力系统运行的重要保障.本文针对含有不确定因素和负荷扰动的多区域互联电力系统提出了一种基于线性矩阵不等式参数可调节的鲁棒分布式预测控制算法.设计各个区域控制器目标函数引入相邻区域的状态变量和输入变量,同时考虑发电机变化速率约束和阀门位置约束,将求解一组凸优化问题转化成线性矩阵不等式求解,得到各个区域的控制律,在线性矩阵不等式中引入一组可调参数,将优化一个上限值转化成优化吸引区,降低算法的保守性.仿真结果验证了该算法在负荷扰动、系统参数不确定和结构不确定性情况下具有鲁棒性.     

Business analytics for strategic management: Identifying and assessing corporate challenges via topic modeling

Strategic management requires an assessment of a firm's internal and external environments. Our work extends the body of management tools (e.g., SWOT analysis or growth-share matrix) by proposing an automated text mining framework. Here we draw on narrative materials from firms (e.g., financial disclosures) and perform topic modeling so as to identify the key issues faced by an organization. We then quantify the use of language along two dimensions: risk and optimism. This reveals a firm's strengths and weaknesses by identifying business units, activities, and processes subject to risk, while also comparing it with competitors or the market.     

Modeling and simulation of chaotic phenomena in electrical power systems

Modeling and simulation of nonlinear systems under chaotic behavior is presented. Nonlinear systems and their relation to chaos as a result of nonlinear interaction of different elements in the system are presented. Application of chaotic theory for power systems is discussed through simulation results. Simulation of some mathematical equations, e.g. Vander Pol's equation, Lorenz's equation, Duffing's equation and double scroll equations are presented. Theoretical aspects of dynamical systems, the existence of chaos in power system and their dependency on system parameters and initial conditions using computer simulations are discussed. From the results one can easily understand the strange attractor and transient stages to voltage collapse, angle instability or voltage collapse and angle divergence simultaneously. Important simulation results of chaos for a model three bus system are presented and discussed.     

Robust fuzzy filter design for nonlinear systems with persistent bounded disturbances.

To date, nonlinear Linfinity-gain filtering problems have not been solved by conventional methods for nonlinear dynamic systems with persistent bounded disturbances. This study introduces a fuzzy filtering design to deal with the nonlinear Linfinity-gain filtering problem. First, the Takagi and Sugeno fuzzy model is employed to approximate the nonlinear dynamic system. Next, based on the fuzzy model, a fuzzy filter is developed to minimize the upper bound of Linfinity-gain of the estimation error under some linear matrix inequality (LMI) constraints. Therefore, the nonlinear Linfinity-gain filtering problem is transformed into a suboptimal filtering problem, i.e., to minimize the upper bound of the Linfinity-gain of the estimation error subject to some LMI constraints. In this situation, the nonlinear Linfinity-gain filtering problem can be easily solved by an LMI-based optimization method. The proposed methods, which efficiently attenuate the peak of estimation error due to persistent bounded disturbances, extend the Linfinity-gain filtering problems from linear dynamic systems to nonlinear dynamic systems.     

Robust distributed MPC for load frequency control of uncertain power systems

Reliable Load frequency control (LFC) is crucial to the operation and design of modern electric power systems. However, the power systems are always subject to uncertainties and external disturbances. Considering the LFC problem of a multi-area interconnected power system, this paper presents a robust distributed model predictive control (RDMPC) based on linear matrix inequalities. The proposed algorithm solves a series of local convex optimization problems to minimize an attractive range for a robust performance objective by using a time-varying state-feedback controller for each control area. The scheme incorporates the two critical nonlinear constraints, e.g., the generation rate constraint (GRC) and the valve limit, into convex optimization problems. Furthermore, the algorithm explores the use of an expanded group of adjustable parameters in LMI to transform an upper bound into an attractive range for reducing conservativeness. Good performance and robustness are obtained in the presence of power system dynamic uncertainties.     

Pseudo-transient continuation-based variable relaxation solver in nonlinear magnetohydrodynamic simulations

Efficient and robust Variable Relaxation Solver, based on pseudo-transient continuation, is developed to solve nonlinear anisotropic thermal conduction arising from fusion plasma simulations. By adding first- and/or second-order artificial time derivatives to the system, this type of method advances the resulting time-dependent nonlinear PDEs to steady state, which is the solution to be sought. In this process, only the stiffness matrix itself is involved so that the numerical complexity and errors can be greatly reduced. In fact, this work is an extension of integrating efficient linear elliptic solvers for fusion simulation on Cray X1E. Two schemes are derived in this work, first- and second-order variable relaxations. Four factors are observed to be critical for efficiency and preservation of solution's symmetric structure arising from periodic boundary condition: refining meshes in different coordinate directions, initializing nonlinear process, varying time steps in both temporal and spatial directions, and accurately generating nonlinear stiffness matrix. First finer mesh scale should be taken in strong transport direction; next the system is carefully initialized by the solution with linear conductivity; third, time step and relaxation factor are vertex-based varied and optimized at each time step; finally, the nonlinear stiffness matrix is updated by just scaling corresponding linear one with the vector generated from nonlinear thermal conductivity.     

Optimal linear regulators: The discrete-time case

In this paper the optimal discrete-time linear-quadratic regulator problem is carefully presented and the basic results are reviewed. Dynamic programming is used to determine the optimization equations. Special attention is given to problems unique to the discrete-time case; this includes, for example, the possibility of a singular system matrix and a singular control-effort weighting matrix. Some problems associated with sampled-data systems are also summarized, e.g., sensitivity to sampling time, and loss of controllability due to sampling. Computational methods for the solution of the optimization equations are outlined and a simple example is included to illustrate the various computational approaches.     

Fast automatic two-stage nonlinear model identification based on the extreme learning machine

It is convenient and effective to solve nonlinear problems with a model that has a linear-in-the-parameters (LITP) structure. However, the nonlinear parameters (e.g. the width of Gaussian function) of each model term needs to be pre-determined either from expert experience or through exhaustive search. An alternative approach is to optimize them by a gradient-based technique (e.g. Newton's method). Unfortunately, all of these methods still need a lot of computations. Recently, the extreme learning machine (ELM) has shown its advantages in terms of fast learning from data, but the sparsity of the constructed model cannot be guaranteed. This paper proposes a novel algorithm for automatic construction of a nonlinear system model based on the extreme learning machine. This is achieved by effectively integrating the ELM and leave-one-out (LOO) cross validation with our two-stage stepwise construction procedure [1]. The main objective is to improve the compactness and generalization capability of the model constructed by the ELM method. Numerical analysis shows that the proposed algorithm only involves about half of the computation of orthogonal least squares (OLS) based method. Simulation examples are included to confirm the efficacy and superiority of the proposed technique.     

Unified Computation of Strict Maximum Likelihood for Geometric Fitting

A new numerical scheme is presented for computing strict maximum likelihood (ML) of geometric fitting problems having an implicit constraint. Our approach is orthogonal projection of observations onto a parameterized surface defined by the constraint. Assuming a linearly separable nonlinear constraint, we show that a theoretically global solution can be obtained by iterative Sampson error minimization. Our approach is illustrated by ellipse fitting and fundamental matrix computation. Our method also encompasses optimal correction, computing, e.g., perpendiculars to an ellipse and triangulating stereo images. A detailed discussion is given to technical and practical issues about our approach.     

Optimization problems of transportation in communication networks with variable capacities

In this paper, formulations of the synthesis and analysis problems in multiterminal communication networks with variable edge capacities are considered. In contrast with traditional formulations, the notions of flow density, flow speed, and flow intensity are introduced and the time characteristics of a transfer of given volumes through a network are analyzed. Related problems are found in various fields of human activities, e.g., with studies of traffic flows in evacuation models and, e.g., in models of the requalification of people on the labor market. The algorithm for solving the transportation problem in a multiterminal network with variable capacities is presented; under certain conditions, this algorithm allows one to avoid dealing with complex nonlinear large-scale problems of mathematical programming.     

Structured low-rank approximation and its applications

Fitting data by a bounded complexity linear model is equivalent to low-rank approximation of a matrix constructed from the data. The data matrix being Hankel structured is equivalent to the existence of a linear time-invariant system that fits the data and the rank constraint is related to a bound on the model complexity. In the special case of fitting by a static model, the data matrix and its low-rank approximation are unstructured.We outline applications in system theory (approximate realization, model reduction, output error, and errors-in-variables identification), signal processing (harmonic retrieval, sum-of-damped exponentials, and finite impulse response modeling), and computer algebra (approximate common divisor). Algorithms based on heuristics and local optimization methods are presented. Generalizations of the low-rank approximation problem result from different approximation criteria (e.g., weighted norm) and constraints on the data matrix (e.g., nonnegativity). Related problems are rank minimization and structured pseudospectra.     

FUZZY SCHEDULING OF REGIONAL QFT CONTROLLERS

A novel optimization-based controller synthesis method is developed for nonlinear dynamic systems with structured parametric uncertainty. Fuzzy logic is used to smoothly schedule independently designed regional robust controllers over the plant's operational envelope. These linear controllers are synthesized using established conventional control design techniques, e.g., quantitative feedback theory. The resulting full envelope nonlinear dynamic controller handles complex dynamic systems which cannot otherwise be addressed by simple fuzzy logic control (FLC). An analytical representation of the membership functions of FLC allows the optimization to chose the location parameters of the regional controllers. The scheduled controller's valid region of operation is maximized, thus efficiently achieving full envelope operation, while guaranteeing pre-specified tracking performance. © 1997 by John Wiley & Sons, Ltd. This paper was produced under the auspices of the US Government and it is therefore not subject to copyright in the US.     

Accuracy statistics for judging soft classification

In the literature one can find different accuracy measures that are built from the error matrix. However, standard accuracy assessment, which is based on the error matrix, is incomplete when dealing with fuzzy sets or when errors do not have the same importance. In this paper, we propose an extension of the error concept for soft (or crisp) classification that will be able to extend standard accuracy measures (e.g., overall, producer's, user's or Kappa statistic) that can be used in any framework: errors with different importance, soft classifier and crisp reference data (expert) or with a fuzzy expert. In particular, a weighted measure is built that takes into account the preferences of the decision maker in order to differentiate some errors that must not be considered equal.     

Stability Analysis of an Underactuated Autonomous Underwater Vehicle Using Extended-Routh’s Stability Method

In this paper, a new approach to stability analysis of nonlinear dynamics of an underactuated autonomous underwater vehicle (AUV) is presented. AUV is a highly nonlinear robotic system whose dynamic model includes coupled terms due to the hydrodynamic damping factors. It is difficult to analyze the stability of a nonlinear dynamical system through Routh’s stability approach because it contains nonlinear dynamic parameters owing to hydrodynamic damping coefficients. It is also difficult to analyze the stability of AUVs using Lyapunov’s criterion and LaSalle’s invariance principle. In this paper, we proposed the extended-Routh’s stability approach to verify the stability of such nonlinear dynamic systems. This extended-Routh’s stability approach is much easier as compared to the other existing methods. Numerical simulations are presented to demonstrate the efficacy of the proposed stability verification of the nonlinear dynamic systems, e.g., an AUV system dynamics.     

On the accurate determination of nonisolated solutions of nonlinear equations

A simple but efficient method to obtain accurate solutions of a system of nonlinear equations with a singular Jacobian at the solution is presented. This is achieved by enlarging the system to a higher dimensional one whose solution in question is isolated. Thus it can be computed e. g. by Newton's method, which is locally at least quadratically convergent and selfcorrecting, so that high accuracy is attainable.     

Nonlinear Finite Element Analysis using an Object-Oriented Philosophy – Application to Beam Elements and to the Cosserat Continuum

An Object-Oriented Programming (OOP) frame-work is presented for solving nonlinear structural mechanics problems by means of the Finite Element Method (FEM). Emphasis is placed on engineering applications (geometrically nonlinear beam model, and elastoplastic Cosserat continuum), and OOP is employed as an effective tool, which plays an important role in the FEM treatment of such applications. The implementation is based on computational abstractions of both mathematical and physical concepts associated to structural mechanics problems involving geometrical and material nonlinearities. The overall class organization for nonlinear mechanics modeling is discussed in detail. All the analyses rely on a generic control class where several classical and modern nonlinear solution schemes are available. Examples which explore, demonstrate and validate the main features of the overall computational system are presented and discussed.