多层感知器模型互反奇异性区域学习动态的理论分析

多层感知器神经网络(MLPs)的学习过程经常发生一些奇异性行为,容易陷入平坦区,这都和MLPs的参数空间中存在的奇异性区域有直接关系.当MLPs的两个隐节点的权值接近互反时,置换对称性会导致学习困难.对MLPs的互反奇异性区域附近的学习动态进行分析.本文首先得到了平均学习方程的解析表达式,然后给出了互反奇异性区域附近的理论学习轨迹,并通过数值方法得到了其附近的实际学习轨迹.通过仿真实验,分别观察了MLPs的平均学习动态,批处理学习动态和在线学习动态,并进行了比较分析.     

Singularities affect dynamics of learning in neuromanifolds

The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities, ARMA time-series model, and many other cases. The standard statistical paradigm of the Cramér-Rao theorem does not hold, and the singularity gives rise to strange behaviors in parameter estimation, hypothesis testing, Bayesian inference, model selection, and in particular, the dynamics of learning from examples. Prevailing theories so far have not paid much attention to the problem caused by singularity, relying only on ordinary statistical theories developed for regular (nonsingular) models. Only recently have researchers remarked on the effects of singularity, and theories are now being developed.This article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures. We demonstrate our recent results on these problems. Simple toy models are also used to show explicit solutions. We explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically, because the Fisher information matrix degenerates, that the model selection criteria such as AIC, BIC, and MDL fail to hold in these models, that a smooth Bayesian prior becomes singular in such models, and that the trajectories of dynamics of learning are strongly affected by the singularity, causing plateaus or slow manifolds in the parameter space. The natural gradient method is shown to perform well because it takes the singular geometrical structure into account. The generalization error and the training error are studied in some examples.     

Error-driven active learning in growing radial basis function networks for early robot learning

In this paper, we describe a new error-driven active learning approach to self-growing radial basis function networks for early robot learning. There are several mappings that need to be set up for an autonomous robot system for sensorimotor coordination and transformation of sensory information from one modality to another, and these mappings are usually highly nonlinear. Traditional passive learning approaches usually cause both large mapping errors and nonuniform mapping error distribution compared to active learning. A hierarchical clustering technique is introduced to group large mapping errors and these error clusters drive the system to actively explore details of these clusters. Higher level local growing radial basis function subnetworks are used to approximate the residual errors from previous mapping levels. Plastic radial basis function networks construct the substrate of the learning system and a simplified node-decoupled extended Kalman filter algorithm is presented to train these radial basis function networks. Experimental results are given to compare the performance among active learning with hierarchical adaptive RBF networks, passive learning with adaptive RBF networks and hierarchical mixtures of experts, as well as their robustness under noise conditions.     

Dynamics of Learning in Multilayer Perceptrons Near Singularities

The dynamical behavior of learning is known to be very slow for the multilayer perceptron, being often trapped in the “plateau.” It has been recently understood that this is due to the singularity in the parameter space of perceptrons, in which trajectories of learning are drawn. The space is Riemannian from the point of view of information geometry and contains singular regions where the Riemannian metric or the Fisher information matrix degenerates. This paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity. We give explicit asymptotic analytical solutions (trajectories) both for the standard gradient (SGD) and natural gradient (NGD) methods. It is clearly shown, in the case of the SGD method, that the plateau phenomenon appears in a neighborhood of the critical regions, where the dynamical behavior is extremely slow. The analysis of the NGD method is much more difficult, because the inverse of the Fisher information matrix diverges. We conquer the difficulty by introducing the “blow-down” technique used in algebraic geometry. The NGD method works efficiently, and the state converges directly to the true parameters very quickly while it staggers in the case of the SGD method. The analytical results are compared with computer simulations, showing good agreement. The effects of singularities on learning are thus qualitatively clarified for both standard and NGD methods.      

Dynamic fuzzy neural networks-a novel approach to functionapproximation

In this paper, an architecture of dynamic fuzzy neural networks (D-FNN) implementing Takagi-Sugeno-Kang (TSK) fuzzy systems based on extended radial basis function (RBF) neural networks is proposed. A novel learning algorithm based on D-FNN is also presented. The salient characteristics of the algorithm are: 1) hierarchical on-line self-organizing learning is used; 2) neurons can be recruited or deleted dynamically according to their significance to the system's performance; and 3) fast learning speed can be achieved. Simulation studies and comprehensive comparisons with some other learning algorithms demonstrate that a more compact structure with higher performance can be achieved by the proposed approach.     

Clustering-based algorithms for single-hidden-layer sigmoid perceptron

Gradient-descent type supervised learning is the most commonly used algorithm for design of the standard sigmoid perceptron (SP). However, it is computationally expensive (slow) and has the local-minima problem. Moody and Darken (1989) proposed an input-clustering based hierarchical algorithm for fast learning in networks of locally tuned neurons in the context of radial basis function networks. We propose and analyze input clustering (IC) and input-output clustering (IOC)-based algorithms for fast learning in networks of globally tuned neurons in the context of the SP. It is shown that "localizing' the input layer weights of the SP by the IC and the IOC minimizes an upper bound to the SP output error. The proposed algorithms could possibly be used also to initialize the SP weights for the conventional gradient-descent learning. Simulation results offer that the SPs designed by the IC and the IOC yield comparable performance in comparison with its radial basis function network counterparts.     

An efficient multilayered shielded microwave circuit analysis method based on neural networks

In previous works, a neural network based technique to analyze multilayered shielded microwave circuits was developed. The method is based on the approximation of the shielded media Green's functions by radial‐basis‐function neural networks (RBFNNs). The trained neural networks, substitute the original Green's functions during the application of the integral equation approach, allowing a faster analysis than the direct solution. In this article, new and important improvements are applied to the training of the RBFNNs, which permit a reduction in the approximation error introduced by the neural networks. Furthermore, outstanding time reductions in the analysis of printed circuits are achieved, clearly outperforming the former technique. The main improvement consists on a better processing of the Green's function singularity near the source. The singularity produces rapid variations near the source that makes difficult the neural network training. In this work, the singularity is extracted in a more suitable fashion than in previous works. The functions resulting from the singularity extraction present a smooth behavior, so they can be easily approximated by neural networks. In addition, a new subdivision strategy for the input space is proposed to efficiently train the neural networks. Two practical microwave filters are analyzed using the new techniques. Comparisons with measured results are also presented for validation. © 2010 Wiley Periodicals, Inc. Int J RF and Microwave CAE, 2010.     

A comparison between HMLP and HRBF for attitude control

In this paper the problem of controlling the attitude of a rigid body, such as a Spacecraft, in three-dimensional space is approached by introducing two new control strategies developed in hypercomplex algebra. The proposed approaches are based on two parallel controllers, both derived in quaternion algebra. The first is a feedback controller of the proportional derivative (PD) type, while the second is a feedforward controller, which is implemented either by means of a hypercomplex multilayer perceptron (HMLP) neural network or by means of a hypercomplex radial basis function (HRBF) neural network. Several simulations show the performance of the two approaches. The results are also compared with a classical PD controller and with an adaptive controller, showing the improvements obtained by using neural networks, especially when an external disturbance acts on the rigid body. In particular the HMLP network gave better results when considering trajectories not presented during the learning phase.     

一种多层前馈网参数可分离学习算法

目前大部分神经网络学习算法都是对网络所有的参数同时进行学习.当网络规模较大时,这种做法常常很耗时.由于许多网络,例如感知器、径向基函数网络、概率广义回归网络以及模糊神经网络,都是一种多层前馈型网络,它们的输入输出映射都可以表示为一组可变基的线性组合.网络的参数也表现为二类:可变基中的参数是非线性的,组合系数是线性的.为此,提出了一个将这二类参数进行分离学习的算法.仿真结果表明,这个学习算法加快了学习过程,提高了网络的逼近性能.     

Centroid based Multilayer Perceptron Networks

In this study we investigate a hybrid neural network architecture for modelling purposes. The proposed network is based on the multilayer perceptron (MLP) network. However, in addition to the usual hidden layers the first hidden layer is selected to be a centroid layer. Each unit in this new layer incorporates a centroid that is located somewhere in the input space. The output of these units is the Euclidean distance between the centroid and the input. The centroid layer clearly resembles the hidden layer of the radial basis function (RBF) networks. Therefore the centroid based multilayer perceptron (CMLP) networks can be regarded as a hybrid of MLP and RBF networks. The presented benchmark experiments show that the proposed hybrid architecture is able to combine the good properties of MLP and RBF networks resulting fast and efficient learning, and compact network structure.     

Combined projection and kernel basis functions for classification in evolutionary neural networks

This paper proposes a hybrid neural network model using a possible combination of different transfer projection functions (sigmoidal unit, SU, product unit, PU) and kernel functions (radial basis function, RBF) in the hidden layer of a feed-forward neural network. An evolutionary algorithm is adapted to this model and applied for learning the architecture, weights and node typology. Three different combined basis function models are proposed with all the different pairs that can be obtained with SU, PU and RBF nodes: product–sigmoidal unit (PSU) neural networks, product–radial basis function (PRBF) neural networks, and sigmoidal–radial basis function (SRBF) neural networks; and these are compared to the corresponding pure models: product unit neural network (PUNN), multilayer perceptron (MLP) and the RBF neural network. The proposals are tested using ten benchmark classification problems from well known machine learning problems. Combined functions using projection and kernel functions are found to be better than pure basis functions for the task of classification in several datasets.     

Degeneracy conditions of the dynamic model of parallel robots

Despite their well-known advantages in terms of higher intrinsic rigidity, larger payload-to-weight ratio, and higher velocity and acceleration capacities, parallel robots have drawbacks. Among them, the most important one is surely the presence of singularities in the workspace, which divide the workspace into different aspects (each aspect corresponding to one or more assembly modes) and near which the performance is considerably reduced.In order to increase the reachable workspace of parallel robots, a promising solution consists in the definition of optimal trajectories passing through the singularities to change either the leg working modes or the robot assembly modes. Previous works on the field have shown that it is possible to define optimal trajectories that allow the passing through the robot type 2 singularities. Such trajectories must respect a physical criterion that can be obtained through the analysis of the degeneracy conditions of the parallel robot inverse dynamic model.However, the mentioned works were not complete: they lacked a degeneracy condition of the parallel robot inverse dynamic model, which is not due to type 2 singularity anymore, but to a serial singularity. Crossing a serial singularity is appealing as in that case we can change the robot leg working mode and then potentially access to other workspace zones. This absence is due to the fact that the authors used a reduced dynamic model, which was not taking into account all link dynamic parameters.The present paper aims to fill this gap by providing a complete study of the degeneracy conditions of the parallel robot dynamic model and by demonstrating that it is possible to cross the type 2, but also serial singularity, by defining trajectories that respect some given criteria obtained from the analysis of the degeneracy of the robot dynamic model. It also aims to demonstrate that the serial singularities have impacts on the robot effort transmission, which is a point that is usually bypassed in the literature. All theoretical developments are validated through simulations and experiments.     

Combined genetic algorithm optimization and regularized orthogonalleast squares learning for radial basis function networks

Presents a two-level learning method for radial basis function (RBF) networks. A regularized orthogonal least squares (ROLS) algorithm is employed at the lower level to construct RBF networks while the two key learning parameters, the regularization parameter and the RBF width, are optimized using a genetic algorithm (GA) at the upper level. Nonlinear time series modeling and prediction is used as an example to demonstrate the effectiveness of this hierarchical learning approach.     

基于遗传算法的前向神经网络结构优化

对近几年应用遗传算法（Genetic Algorithm,GA）优化设计前向神经网络结构的研究进行了评述。指出了神经网络结构优化设计的重要性和目前各种方法存在的不足。介绍了神经网络结构设计原理和应用GA优化设计神经网络应着重考虑的两个问题：即结构表达策略和适应度函数设计。分别对近来应用GA优化设计多层感知器、径向基函数神经网络和径向基概率神经网络结构的研究进行了细致介绍和分析。指出了目前研究工作的不足和未来研究工作的发展方向。     

On the construction and training of reformulated radial basis function neural networks

Presents a systematic approach for constructing reformulated radial basis function (RBF) neural networks, which was developed to facilitate their training by supervised learning algorithms based on gradient descent. This approach reduces the construction of radial basis function models to the selection of admissible generator functions. The selection of generator functions relies on the concept of the blind spot, which is introduced in the paper. The paper also introduces a new family of reformulated radial basis function neural networks, which are referred to as cosine radial basis functions. Cosine radial basis functions are constructed by linear generator functions of a special form and their use as similarity measures in radial basis function models is justified by their geometric interpretation. A set of experiments on a variety of datasets indicate that cosine radial basis functions outperform considerably conventional radial basis function neural networks with Gaussian radial basis functions. Cosine radial basis functions are also strong competitors to existing reformulated radial basis function models trained by gradient descent and feedforward neural networks with sigmoid hidden units.     

层次泛函网络整体学习算法

文中设计了一类单输人单输出泛函网络与双输人单输出泛函网络作为构造层次泛函网络基本模型,提出了一种层次泛函网络模型,给出了层次泛函网络构造方法和整体学习算法,而层次泛函网络的参数利用解方程组来进行逐层学习．以非线性代数方程组为例,指出人们熟知的一些数学解题方法可以用层次泛函网络来表达,探讨了基于层次泛函网络求解非线性代数方程组学习算法实现的一些技术问题．相对传统方法,层次泛函网络更适合于具有层次结构的应用领域．计算机仿真结果表明,这种层次学习方法具有较快的收敛速度和良好的逼近性能．     

On the versatility of radial basis function neural networks: A case study in the field of intrusion detection

Classifiers based on radial basis function neural networks have a number of useful properties that can be exploited in many practical applications. Using sample data, it is possible to adjust their parameters (weights), to optimize their structure, and to select appropriate input features (attributes). Moreover, interpretable rules can be extracted from a trained classifier and input samples can be identified that cannot be classified with a sufficient degree of “certainty”. These properties support an analysis of radial basis function classifiers and allow for an adaption to “novel” kinds of input samples in a real-world application. In this article, we outline these properties and show how they can be exploited in the field of intrusion detection (detection of network-based misuse). Intrusion detection plays an increasingly important role in securing computer networks. In this case study, we first compare the classification abilities of radial basis function classifiers, multilayer perceptrons, the neuro-fuzzy system NEFCLASS, decision trees, classifying fuzzy-k-means, support vector machines, Bayesian networks, and nearest neighbor classifiers. Then, we investigate the interpretability and understandability of the best paradigms found in the previous step. We show how structure optimization and feature selection for radial basis function classifiers can be done by means of evolutionary algorithms and compare this approach to decision trees optimized using certain pruning techniques. Finally, we demonstrate that radial basis function classifiers are basically able to detect novel attack types. The many advantageous properties of radial basis function classifiers could certainly be exploited in other application fields in a similar way.     

Wavelet basis function neural networks for sequential learning.

In this letter, we develop the wavelet basis function neural networks (WBFNNs). It is analogous to radial basis function neural networks (RBFNNs) and to wavelet neural networks (WNNs). In WBFNNs, both the scaling function and the wavelet function of a multiresolution approximation (MRA) are adopted as the basis for approximating functions. A sequential learning algorithm for WBFNNs is presented and compared to the sequential learning algorithm of RBFNNs. Experimental results show that WBFNNs have better generalization property and require shorter training time than RBFNNs.     

基于小波框架的自适应径向基函数网络

给出了由高斯径向基函数生成的一组小波框架,建立在小波框架理论的基础上,构造性地证明了高斯径向基函数网络可以任意精度地逼近L2(Rd)中的函数.在此基础上,利用高斯径向基函数的时频局部化性质和自适应投影原理,进一步给出了构造和训练网络的自适应学习算法.应用到信号的重构和去噪,获得了良好的效果.     

Adaptive neural decentralized control for strict feedback nonlinear interconnected systems via backstepping

An approximation based adaptive neural decentralized output tracking control scheme for a class of large-scale unknown nonlinear systems with strict-feedback interconnected subsystems with unknown nonlinear interconnections is developed in this paper. Within this scheme, radial basis function RBF neural networks are used to approximate the unknown nonlinear functions of the subsystems. An adaptive neural controller is designed based on the recursive backstepping procedure and the minimal learning parameter technique. The proposed decentralized control scheme has the following features. First, the controller singularity problem in some of the existing adaptive control schemes with feedback linearization is avoided. Second, the numbers of adaptive parameters required for each subsystem are not more than the order of this subsystem. Lyapunov stability method is used to prove that the proposed adaptive neural control scheme guarantees that all signals in the closed-loop system are uniformly ultimately bounded, while tracking errors converge to a small neighborhood of the origin. The simulation example of a two-spring interconnected inverted pendulum is presented to verify the effectiveness of the proposed scheme.