Learning without local minima in radial basis function networks

Learning from examples plays a central role in artificial neural networks. The success of many learning schemes is not guaranteed, however, since algorithms like backpropagation may get stuck in local minima, thus providing suboptimal solutions. For feedforward networks, optimal learning can be achieved provided that certain conditions on the network and the learning environment are met. This principle is investigated for the case of networks using radial basis functions (RBF). It is assumed that the patterns of the learning environment are separable by hyperspheres. In that case, we prove that the attached cost function is local minima free with respect to all the weights. This provides us with some theoretical foundations for a massive application of RBF in pattern recognition.     

On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks

This paper investigates new learning algorithms (LF I and LF II) based on Lyapunov function for the training of feedforward neural networks. It is observed that such algorithms have interesting parallel with the popular backpropagation (BP) algorithm where the fixed learning rate is replaced by an adaptive learning rate computed using convergence theorem based on Lyapunov stability theory. LF II, a modified version of LF I, has been introduced with an aim to avoid local minima. This modification also helps in improving the convergence speed in some cases. Conditions for achieving global minimum for these kind of algorithms have been studied in detail. The performances of the proposed algorithms are compared with BP algorithm and extended Kalman filtering (EKF) on three bench-mark function approximation problems: XOR, 3-bit parity, and 8-3 encoder. The comparisons are made in terms of number of learning iterations and computational time required for convergence. It is found that the proposed algorithms (LF I and II) are much faster in convergence than other two algorithms to attain same accuracy. Finally, the comparison is made on a complex two-dimensional (2-D) Gabor function and effect of adaptive learning rate for faster convergence is verified. In a nutshell, the investigations made in this paper help us better understand the learning procedure of feedforward neural networks in terms of adaptive learning rate, convergence speed, and local minima.     

基于回归神经网络的非线性时变系统辨识

为克服基于前馈神经网络的非线性系统辨识算法存在需预先估计系统输入输出滞后阶数的缺陷，提出一种基于回归神经网络的非线性时变系统的辨识算法，针对现有的回归网络学习算法大多采用梯度算法，收敛速度缓慢问题，提出一种具有快速收敛性的扩展卡尔曼滤波学习算法，大大提高了学习收敛速度，并推导了一种基于单个神经元的局部化算法，减少了计算量，仿真实例证明，所提出的算法是有效的。     

一种新型随机二进制神经网络

二进制数据表示具有简洁高效的特点,随机噪声有助于系统摆脱局部极小.新型的随 机神经网络模型采用随机加权联接,内部数据表示为随机二进制序列形式,实现十分高效.文中 分别就前馈型网络和反馈型网络进行了深入的讨论,给出了前馈型网络的梯度下降学习算法, 为反馈型网络设计了快速有效的模拟退火算法和渐进式Boltzmann学习算法.通过对PARITY 问题的测试,发现了新模型的一些有趣特征,实验结果表明梯度下降学习效果显著.利用渐进式 Boltzmann学习,反馈型网络被成功地用于带噪声人脸识别.     

Existence and uniqueness results for neural network approximations

Some approximation theoretic questions concerning a certain class of neural networks are considered. The networks considered are single input, single output, single hidden layer, feedforward neural networks with continuous sigmoidal activation functions, no input weights but with hidden layer thresholds and output layer weights. Specifically, questions of existence and uniqueness of best approximations on a closed interval of the real line under mean-square and uniform approximation error measures are studied. A by-product of this study is a reparametrization of the class of networks considered in terms of rational functions of a single variable. This rational reparametrization is used to apply the theory of Pade approximation to the class of networks considered. In addition, a question related to the number of local minima arising in gradient algorithms for learning is examined.     

智能油漆配色系统的改进BP算法

BP算法具有数学意义明确、学习规则简单等优点,是前向多次神经网络的典型学习算法。但是,BP算法在学习过程中容易陷入局部最小问题。针对这一问题,提出一种修正Sigmoid函数的改进BP算法。实验证明,改进BP算法可以有效克服局部最小,显著提高收敛速度。     

An improved approximation approach incorporating particle swarm optimization and a priori information into neural networks

In this paper, an improved approach incorporating adaptive particle swarm optimization (APSO) and a priori information into feedforward neural networks for function approximation problem is proposed. It is well known that gradient-based learning algorithms such as backpropagation algorithm have good ability of local search, whereas PSO has good ability of global search. Therefore, in the improved approach, the APSO algorithm encoding the first-order derivative information of the approximated function is used to train network to near global minima. Then, with the connection weights produced by APSO, the network is trained with a modified gradient-based algorithm with magnified gradient function. The modified gradient-based algorithm can reduce input-to-output mapping sensitivity and lessen the chance of being trapped into local minima. By combining APSO with local search algorithm and considering a priori information, the improved approach has better approximation accuracy and convergence rate. Finally, simulation results are given to verify the efficiency and effectiveness of the proposed approach.     

Fast Learning Algorithms for Feedforward Neural Networks

In order to improve the training speed of multilayer feedforward neural networks (MLFNN), we propose and explore two new fast backpropagation (BP) algorithms obtained: (1) by changing the error functions, in case using the exponent attenuation (or bell impulse) function and the Fourier kernel function as alternative functions; and (2) by introducing the hybrid conjugate-gradient algorithm of global optimization for dynamic learning rate to overcome the conventional BP learning problems of getting stuck into local minima or slow convergence. Our experimental results demonstrate the effectiveness of the modified error functions since the training speed is faster than that of existing fast methods. In addition, our hybrid algorithm has a higher recognition rate than the Polak-Ribieve conjugate gradient and conventional BP algorithms, and has less training time, less complication and stronger robustness than the Fletcher-Reeves conjugate-gradient and conventional BP algorithms for real speech data.     

Memetic cooperative coevolution of Elman recurrent neural networks

Cooperative coevolution decomposes an optimisation problem into subcomponents and collectively solves them using evolutionary algorithms. Memetic algorithms provides enhancement to evolutionary algorithms with local search. Recently, the incorporation of local search into a memetic cooperative coevolution method has shown to be efficient for training feedforward networks on pattern classification problems. This paper applies the memetic cooperative coevolution method for training recurrent neural networks on grammatical inference problems. The results show that the proposed method achieves better performance in terms of optimisation time and robustness.     

Simulated annealing and weight decay in adaptive learning: theSARPROP algorithm

A problem with gradient descent algorithms is that they can converge to poorly performing local minima. Global optimization algorithms address this problem, but at the cost of greatly increased training times. This work examines combining gradient descent with the global optimization technique of simulated annealing (SA). Simulated annealing in the form of noise and weight decay is added to resiliant backpropagation (RPROP), a powerful gradient descent algorithm for training feedforward neural networks. The resulting algorithm, SARPROP, is shown through various simulations not only to be able to escape local minima, but is also able to maintain, and often improve the training times of the RPROP algorithm. In addition, SARPROP may be used with a restart training phase which allows a more thorough search of the error surface and provides an automatic annealing schedule.     

基于神经网络的机器人学习与控制：回顾与展望

机器人因其高效的感知、决策和执行能力,在人工智能、信息技术和智能制造等领域中具有巨大的应用价值。目前,机器人学习与控制已成为机器人研究领域的重要前沿技术之一。各种基于神经网络的智能算法被设计,从而为机器人系统提供同步学习与控制的规划框架。首先从神经动力学(ND)算法、前馈神经网络(FNNs)、递归神经网络(RNNs)和强化学习(RL)四个方面介绍了基于神经网络的机器人学习与控制的研究现状,回顾了近30年来面向机器人学习与控制的智能算法和相关应用技术。最后展望了该领域存在的问题和发展趋势,以期促进机器人学习与控制理论的推广及应用场景的拓展。     

脉冲神经网络:模型、学习算法与应用

脉冲神经网络是目前最具有生物解释性的人工神经网络,是类脑智能领域的核心组成部分.首先介绍各类常用的脉冲神经元模型以及前馈和循环型脉冲神经网络结构;然后介绍脉冲神经网络的时间编码方式,在此基础上,系统地介绍脉冲神经网络的学习算法,包括无监督学习和监督学习算法,其中监督学习算法按照梯度下降算法、结合STDP规则的算法和基于脉冲序列卷积核的算法3大类别分别展开详细介绍和总结;接着列举脉冲神经网络在控制领域、模式识别领域和类脑智能研究领域的应用,并在此基础上介绍各国脑计划中,脉冲神经网络与神经形态处理器相结合的案例;最后分析脉冲神经网络目前所存在的困难和挑战.     

Relationships between the A priori and A posteriori errors in nonlinear adaptive neural filters

The lower bounds for the a posteriori prediction error of a nonlinear predictor realized as a neural network are provided. These are obtained for a priori adaptation and a posteriori error networks with sigmoid nonlinearities trained by gradient-descent learning algorithms. A contractivity condition is imposed on a nonlinear activation function of a neuron so that the a posteriori prediction error is smaller in magnitude than the corresponding a priori one. Furthermore, an upper bound is imposed on the learning rate eta so that the approach is feasible. The analysis is undertaken for both feedforward and recurrent nonlinear predictors realized as neural networks.     

Learning algorithms for feedforward networks based on finitesamples

We present two classes of convergent algorithms for learning continuous functions and regressions that are approximated by feedforward networks. The first class of algorithms, applicable to networks with unknown weights located only in the output layer, is obtained by utilizing the potential function methods of Aizerman et al. (1970). The second class, applicable to general feedforward networks, is obtained by utilizing the classical Robbins-Monro style stochastic approximation methods (1951). Conditions relating the sample sizes to the error bounds are derived for both classes of algorithms using martingale-type inequalities. For concreteness, the discussion is presented in terms of neural networks, but the results are applicable to general feedforward networks, in particular to wavelet networks. The algorithms can be directly adapted to concept learning problems.     

Novel approaches for online playout delay prediction in VoIP applications using time series models

Voice over IP (VoIP) applications requires a buffer at the receiver to minimize the packet loss due to late arrival. Several algorithms are available in the literature to estimate the playout buffer delay. Classic estimation algorithms are non-adaptive, i.e. they differ from more recent approaches basically due to the absence of learning mechanisms. This paper introduces two new formulations of adaptive algorithms for online learning and prediction of the playout buffer delay, the first one being based on the standard Box-Jenkins autoregressive model, while the second one being based on the feedforward and recurrent neural networks. The obtained results indicate that the proposed algorithms present better overall performance than the classic ones.     

Adding learning to cellular genetic algorithms for trainingrecurrent neural networks

This paper proposes a hybrid optimization algorithm which combines the efforts of local search (individual learning) and cellular genetic algorithms (GA) for training recurrent neural nets (RNN). Each RNN weight is encoded as a floating point number, and a concatenation of numbers forms a chromosome. Reproduction takes place locally in a square grid, each grid point representing a chromosome. Lamarckian and Baldwinian (1896) mechanisms for combining cellular GA and learning are compared. Different hill-climbing algorithms are incorporated into the cellular GA. These include the real-time recurrent learning (RTRL) and its simplified versions, and the delta rule. RTRL has been successively simplified by freezing some of the weights to form simplified versions. The delta rule, the simplest form of learning, has been implemented by considering the RNN as feedforward networks. The hybrid algorithms are used to train the RNN to solve a long-term dependency problem. The results show that Baldwinian learning is inefficient in assisting the cellular GA. It is conjectured that the more difficult it is for genetic operations to produce the genotypic changes that match the phenotypic changes due to learning, the poorer is the convergence of Baldwinian learning. Most of the combinations using the Lamarckian mechanism show an improvement in reducing the number of generations for an optimum network; however, only a few can reduce the actual time taken. Embedding the delta rule in the cellular GA is the fastest method. Learning should not be too extensive.     

基于MPSO的BP网络及其在入侵检测中的应用

提出一种基于变异粒子群优化(MPSO)的BP网络学习算法,该算法用PSO算法替代了传统BP算法,且在学习过程中,引入变异操作,克服传统BP算法易陷入局部极小和PSO算法早熟的不足。并把该算法应用于入侵检测中,通过KDD99 CUP数据集分别对基于不同算法的BP神经网络进行了仿真实验比较,结果表明,该算法的收敛速度快,迭代次数较少,而且测试平均准确率高达96.5%。     

Learning in multilayered networks used as autoassociators

Gradient descent learning algorithms may get stuck in local minima, thus making the learning suboptimal. In this paper, we focus attention on multilayered networks used as autoassociators and show some relationships with classical linear autoassociators. In addition, by using the theoretical framework of our previous research, we derive a condition which is met at the end of the learning process and show that this condition has a very intriguing geometrical meaning in the pattern space.     

限定记忆的前向神经网络在线学习算法研究

从理论上分析了隐含层激励函数满足Mercer条件的前向神经网络的数学本质,给出了网络学习的指导方向.提出3种网络在线学习算法,它们通过动态调整网络结构和权值来提高网络在线预测性能.算法完全符合统计学习理论提出的结构风险最小化原则,具有较快的学习收敛速度和良好的抗噪声能力.最后通过具体数值实验验证了上述算法的可行性和优越性.     

Interpolation functions of feedforward neural networks

Mathematical essence and structures of the feedforward neural networks are investigated in this paper. The interpolation mechanisms of the feedforward neural networks are explored. For example, the well-known result, namely, that a neural network is an universal approximator, can be concluded naturally from the interpolative representations. Finally, the learning algorithms of the feedforward neural networks are discussed.