Singularities affect dynamics of learning in neuromanifolds

The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities, ARMA time-series model, and many other cases. The standard statistical paradigm of the Cramér-Rao theorem does not hold, and the singularity gives rise to strange behaviors in parameter estimation, hypothesis testing, Bayesian inference, model selection, and in particular, the dynamics of learning from examples. Prevailing theories so far have not paid much attention to the problem caused by singularity, relying only on ordinary statistical theories developed for regular (nonsingular) models. Only recently have researchers remarked on the effects of singularity, and theories are now being developed.This article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures. We demonstrate our recent results on these problems. Simple toy models are also used to show explicit solutions. We explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically, because the Fisher information matrix degenerates, that the model selection criteria such as AIC, BIC, and MDL fail to hold in these models, that a smooth Bayesian prior becomes singular in such models, and that the trajectories of dynamics of learning are strongly affected by the singularity, causing plateaus or slow manifolds in the parameter space. The natural gradient method is shown to perform well because it takes the singular geometrical structure into account. The generalization error and the training error are studied in some examples.     

Dynamics of learning near singularities in layered networks

We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the universal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depending on the stability of the singularity and the initial parameters of learning. The results presented in this letter are universally applicable to a wide class of hierarchical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate work.     

Difficulty of singularity in population coding

Fisher information has been used to analyze the accuracy of neural population coding. This works well when the Fisher information does not degenerate, but when two stimuli are presented to a population of neurons, a singular structure emerges by their mutual interactions. In this case, the Fisher information matrix degenerates, and the regularity condition ensuring the Cramér-Rao paradigm of statistics is violated. An animal shows pathological behavior in such a situation. We present a novel method of statistical analysis to understand information in population coding in which algebraic singularity plays a major role. The method elucidates the nature of the pathological case by calculating the Fisher information. We then suggest that synchronous firing can resolve singularity and show a method of analyzing the binding problem in terms of the Fisher information. Our method integrates a variety of disciplines in population coding, such as nonregular statistics, Bayesian statistics, singularity in algebraic geometry, and synchronous firing, under the theme of Fisher information.     

Riemannian manifold learning

Recently, manifold learning has been widely exploited in pattern recognition, data analysis, and machine learning. This paper presents a novel framework, called Riemannian manifold learning (RML), based on the assumption that the input high-dimensional data lie on an intrinsically low-dimensional Riemannian manifold. The main idea is to formulate the dimensionality reduction problem as a classical problem in Riemannian geometry, i.e., how to construct coordinate charts for a given Riemannian manifold? We implement the Riemannian normal coordinate chart, which has been the most widely used in Riemannian geometry, for a set of unorganized data points. First, two input parameters (the neighborhood size k and the intrinsic dimension d) are estimated based on an efficient simplicial reconstruction of the underlying manifold. Then, the normal coordinates are computed to map the input high-dimensional data into a low-dimensional space. Experiments on synthetic data as well as real world images demonstrate that our algorithm can learn intrinsic geometric structures of the data, preserve radial geodesic distances, and yield regular embeddings.     

Restrictions on the singularity method applied to elasticity

The choice of which fundamental singular solutions to use as basis functions in the so-called singularity method applied to elasticity appears to be restricted by boundary condition types and region geometry. In practice, only the point load solution is consistently reliable but even it fails in certain instances when the elastic body is not a convex region and almost always when the region is not simply connected. Fortunately, all multiply connected regions can be partitioned into simply-connected (although not always convex) ones. A method for doing this is presented along with examples of fundamental solutions which do not work in the singularity method.     

Adaptive method of realizing natural gradient learning for multilayer perceptrons

The natural gradient learning method is known to have ideal performances for on-line training of multilayer perceptrons. It avoids plateaus, which give rise to slow convergence of the backpropagation method. It is Fisher efficient, whereas the conventional method is not. However, for implementing the method, it is necessary to calculate the Fisher information matrix and its inverse, which is practically very difficult. This article proposes an adaptive method of directly obtaining the inverse of the Fisher information matrix. It generalizes the adaptive Gauss-Newton algorithms and provides a solid theoretical justification of them. Simulations show that the proposed adaptive method works very well for realizing natural gradient learning.     

Inverse dynamics of redundant manipulators using a minimum number of control forces

The number of the control actuators used by the inverse kinematics and dynamics algorithms that have been developed in the literature for generating redundant robot joint trajectories is equal to the number of the degrees of freedom of the manipulator. In this article, an inverse dynamics algorithm that performs the tasks using only a minimum number of actuators is proposed. The number of actuators is equal to the dimension of the task space, and the control forces are solved simultaneously with the corresponding system motion. It is shown that because all degrees of freedom are not actuated, the control forces may lose the ability to make an instantaneous effect on the end-effector acceleration at certain configurations, yielding the dynamical equation set of the system to be singular. The dynamical equations are modified in the neighborhood of the singular configurations by utilizing higher-order derivative information, so that the singularities in the numerical procedure are avoided. Asymptotically stable inverse dynamics closed-loop control in the presence of perturbations is also discussed. The algorithm is further generalized to closed chain manipulators. Three-link and two-link redundant planar manipulators are analyzed to illustrate the validity of the approach. © 2995 John Wiley & Sons, Inc.     

Integrated Fisher linear discriminants: An empirical study

This paper studies Fisher linear discriminants (FLDs) based on classification accuracies for imbalanced datasets. An optimal threshold is found out from a series of empirical formulas developed, which is related not only to sample sizes but also to distribution regions. A mixed binary–decimal coding system is suggested to make the very dense datasets sparse and enlarge the class margins on condition that the neighborhood relationships of samples are nearly preserved. The within-class scatter matrices being or approximately singular should be moderately reduced in dimensionality but not added with tiny perturbations. The weight vectors can be further updated by a kind of epoch-limited (three at most) iterative learning strategy provided that the current training error rates come down accordingly. Putting the above ideas together, this paper proposes a type of integrated FLDs. The extensive experimental results over real-world datasets have demonstrated that the integrated FLDs have obvious advantages over the conventional FLDs in the aspects of learning and generalization performances for the imbalanced datasets.     

Learning of Spatio–Temporal Codes in a Coupled Oscillator System

In this paper, we consider a learning strategy that allows one to transmit information between two coupled phase oscillator systems (called teaching and learning systems) via frequency adaptation. The dynamics of these systems can be modeled with reference to a number of partially synchronized cluster states and transitions between them. Forcing the teaching system by steady but spatially nonhomogeneous inputs produces cyclic sequences of transitions between the cluster states, that is, information about inputs is encoded via a “winnerless competition” process into spatio–temporal codes. The large variety of codes can be learned by the learning system that adapts its frequencies to those of the teaching system. We visualize the dynamics using “weighted order parameters (WOPs)” that are analogous to “local field potentials” in neural systems. Since spatio–temporal coding is a mechanism that appears in olfactory systems, the developed learning rules may help to extract information from these neural ensembles.      

Accelerating SGD using flexible variance reduction on large-scale datasets

Neural Computing and Applications - Stochastic gradient descent (SGD) is a popular optimization method widely used in machine learning, while the variance of gradient estimation leads to slow...     

Aggregation of a class of interconnected, linear dynamical systems

In this paper, we introduce the notion of a “meaningful” average of a collection of dynamical systems as distinct from an “ensemble” average. Such a notion is useful for the study of a variety of dynamical systems such as traffic flow, power systems, and econometric systems. We also address the associated issue of the existence and computation of such an average for a class of interconnected, linear, time invariant dynamical systems. Such an “average” dynamical system is not only attractive from a computational perspective, but also represents the average behavior of the interconnected dynamical systems. The problem of analysis and control of heirarchical, large scale control systems can be simplified by approximating the lower level dynamics of such systems with such an average dynamical system.     

黎曼流形上的多模态医学图像配准

针对传统多模态配准方法忽视图像的结构信息和像素间的空间关系,并假定灰度全局一致的前提。本文提出了一种在黎曼流形上的多模态医学图像配准算法。首先采用线性动态模型捕捉图像的高维空间的非线性结构和局部信息,然后通过参数化动态模型构造出一种李群群元,形成黎曼流形,继而将流形嵌入到高维的再生核希尔伯特空间,再在核空间上学习出相似性测度。仿真和临床数据实验结果表明本文算法在刚体配准和仿射配准精度上均优于传统互信息方法和基于邻域的相似性测度学习方法。     

Three dimensional singular element

The finite element method has become a powerful tool for computation of stress intensity factors in fracture mechanics. The simulation of singular behavior in the stress field is accomplished using “quarter points,” following the methods of Barsoum[1] and Henshell-Shaw[2]. The analysis has also been extended to cubic elements [3] and transition elements [4]. However, these concepts cannot be easily extended to three dimensional cases without additional conditions. Progress has been hampered firstly due to a variety of possible shapes the element may possess near the singular edge of the crack, and secondly due to the complexity of algebraic expressions that have to be manipulated.

In the present investigation we extensively used MACSYMA[5], a large symbolic manipulation program at MIT, thereby alleviating some of these difficulties. A simple condition between mid-side nodes has been derived which simulates the proper singular behavior along the crack.

In the investigation we first study a simple collapsed brick element. This is then generalized to a curved crack front. A few results are derived which can be used to compute the stress intensity factors. The concept of the transitional element has also been outlined. The stability of singular element has been discussed. Some of these ideas have been applied to a specific problem with unusual crack geometry. The analysis was carried out using ADINA on VAX machine. ADINA was implemented on VAX by W. E. Lorensen.     

The calculation of singular coefficients

We consider in this paper the calculation of the ‘singular coefficients’ associated with the solution of an elliptic partial differential equation near a singular point; a re-entrant corner, or a crack tip, etc. These are the coefficients in the relevant singular expansion of the solution near the point of singularity; they often have physical relevance, and it is of interest to be able to calculate them accurately. We consider the problem in the context of the global element method; this is a variable-order finite element method designed to be capable of producing highly accurate solutions for singular problems, even in the neighbourhood of the singularity. If the values of the singular coefficients are needed, these must be extracted from the computed solution; we show in this paper that a suitably defined least-squares fitting procedure allows the calculation of values for the leading singular coefficients which are as accurate as the underlying solution.     

Histogram-based embedding for learning on statistical manifolds

A novel binning and learning framework is presented for analyzing and applying large data sets that have no explicit knowledge of distribution parameterizations, and can only be assumed generated by the underlying probability density functions (PDFs) lying on a nonparametric statistical manifold. For models’ discretization, the uniform sampling-based data space partition is used to bin flat-distributed data sets, while the quantile-based binning is adopted for complex distributed data sets to reduce the number of under-smoothed bins in histograms on average. The compactified histogram embedding is designed so that the Fisher–Riemannian structured multinomial manifold is compatible to the intrinsic geometry of nonparametric statistical manifold, providing a computationally efficient model space for information distance calculation between binned distributions. In particular, without considering histogramming in optimal bin number, we utilize multiple random partitions on data space to embed the associated data sets onto a product multinomial manifold to integrate the complementary bin information with an information metric designed by factor geodesic distances, further alleviating the effect of over-smoothing problem. Using the equipped metric on the embedded submanifold, we improve classical manifold learning and dimension estimation algorithms in metric-adaptive versions to facilitate lower-dimensional Euclidean embedding. The effectiveness of our method is verified by visualization of data sets drawn from known manifolds, visualization and recognition on a subset of ALOI object database, and Gabor feature-based face recognition on the FERET database.     

Robust adaptive fuzzy tracking control for a class of perturbed strict-feedback nonlinear systems via small-gain approach

This paper presents a novel robust adaptive fuzzy tracking controller (RAFTC) for a wide class of perturbed strict-feedback nonlinear systems with both unknown system and virtual control gain nonlinearities. For unknown system nonlinearities, two types for them are included: one naturally satisfies the “triangularity condition” and may possess a class of unstructured uncertain functions which are not linearly parameterized, while the other is partially known and consists of parametric uncertainties and known “bounding functions”. The Takagi–Sugeno type fuzzy logic systems are used to approximate unknown system nonlinearities and a systematic design procedure is developed for synthesis of RAFTC by combining the backstepping technique and generalized small-gain approach. The algorithm proposed is highlighted by three advantages: (i) the semi-global uniform ultimate bound of RAFTC in the presence of perturbed uncertainties and unknown virtual control gain nonlinearities can be guaranteed, (ii) the adaptive mechanism with minimal learning parameterizations is obtained and (iii) the possible controller singularity problem in some of the existing adaptive control schemes with feedback linearization techniques can be removed. Performance and limitations of proposed method are discussed and illustrated with simulation results.     

Local bifurcations and feasibility regions indifferential-algebraic systems

The dynamics of a large class of physical systems such as the general power system can be represented by parameter-dependent differential-algebraic models of the form x˙=f and 0=g. Typically, such constrained models have singularities. This paper analyzes the generic local bifurcations including those which are directly related to the singularity. The notion of a feasibility region is introduced and analyzed. It consists of all equilibrium states that can be reached quasistatically from the current operating point without loss of local stability. It is shown that generically loss of stability at the feasibility boundary is caused by one of three different local bifurcations, namely the saddle-node and Hopf bifurcations and a new bifurcation called the singularity induced bifurcation which is analyzed precisely here for the first time. The latter results when an equilibrium point is at the singular surface. Under certain transversality conditions, the change in the eigenstructure of the system Jacobian at the equilibrium is established and the local dynamical structure of the trajectories near this bifurcation point is analyzed     

FILIP: A fuzzy intelligent information system with learning capabilities

FILIP (fuzzy intelligent learning information processing) system is designed with the goal to model human information processing. The issues addressed are uncertain knowledge representation and approximate reasoning based on fuzzy set theory, and knowledge acquisition by “being told” or by “learning from examples”. Concepts that can be “learned” by the system can be imprecise (fuzzy), or the knowledge can be incomplete. In the latter case, FILIP uses the concept of similarity to extrapolate the knowledge to cases that were not covered by examples provided by the user. Concepts are stored in the Knowledge Base and employed in intelligent query processing, based on flexible inference that supports approximate matches between the data in the database and the query.

The architecture of FILIP is discussed, the learning algorithm is described, and examples of the system's performance in the knowledge acquisition and querying modes, together with its explanatory capabilities are shown.     

机器人操作器的奇异性鲁棒逆

奇异性问题在用关节联接的机器人操作器的控制中是一个固有的问题.本文中.我们从考虑关节运动的精确性和可行性出发来确定操作器末端器所需运动时的关节运动.这种确定关节运动的方法称为具有奇异鲁棒逆的逆运动学解.之所以说它具有鲁棒性,是因为它在奇异点也能提供连续解.即使雅可比矩阵的逆或广义逆表示的道运动学解在奇异点或其周围不可行时,雅可比矩阵的奇异鲁棒逆也能为操作器末端器提供一个期望坐标轨迹的近似运动.对奇异鲁棒逆的特性与广义逆的特性进行了比较.并考虑了可行性的标量加权值.