The layer-wise method and the backpropagation hybrid approach tolearning a feedforward neural network

Feedforward neural networks (FNNs) have been proposed to solve complex problems in pattern recognition and classification and function approximation. Despite the general success of learning methods for FNNs, such as the backpropagation (BP) algorithm, second-order optimization algorithms and layer-wise learning algorithms, several drawbacks remain to be overcome. In particular, two major drawbacks are convergence to a local minima and long learning time. We propose an efficient learning method for a FNN that combines the BP strategy and optimization layer by layer. More precisely, we construct the layer-wise optimization method using the Taylor series expansion of nonlinear operators describing a FNN and propose to update weights of each layer by the BP-based Kaczmarz iterative procedure. The experimental results show that the new learning algorithm is stable, it reduces the learning time and demonstrates improvement of generalization results in comparison with other well-known methods.     

非线性权函数感知器的一种线性快速学习算法

高阶感知器是神经元状态变量的非线性化，它是一阶感知器的非线性推广，除了神经元状态变量的非线性化推广外，对权向量函数的非线性推广而得到的感知器，文中定义为具有非线性权向量函数的感知器，由于感知器的权重及作用函数都是非性函数，当感知器接近最优点时，其连接权调节幅度很小，采用对非线性权函数及非线性作用函数分别进行Taylor展开，并取其一式近似逼近原函数，从而使其非线性权函数及非线性作用函数都转化为线性函数，简化了感知器学习过程的计算量，加快了感知的学习过程。最后，给出了具有非线性权函数感知器的线性化学习算法。     

Curvature-driven smoothing: a learning algorithm for feedforwardnetworks

The performance of feedforward neural networks in real applications can often be improved significantly if use is made of a priori information. For interpolation problems this prior knowledge frequently includes smoothness requirements on the network mapping, and can be imposed by the addition to the error function of suitable regularization terms. The new error function, however, now depends on the derivatives of the network mapping, and so the standard backpropagation algorithm cannot be applied. In this letter, we derive a computationally efficient learning algorithm, for a feedforward network of arbitrary topology, which can be used to minimize such error functions. Networks having a single hidden layer, for which the learning algorithm simplifies, are treated as a special case.     

A new optimization algorithm for single hidden layer feedforward neural networks

Feedforward neural networks are the most commonly used function approximation techniques in neural networks. By the universal approximation theorem, it is clear that a single-hidden layer feedforward neural network (FNN) is sufficient to approximate the corresponding desired outputs arbitrarily close. Some researchers use genetic algorithms (GAs) to explore the global optimal solution of the FNN structure. However, it is rather time consuming to use GA for the training of FNN. In this paper, we propose a new optimization algorithm for a single-hidden layer FNN. The method is based on the convex combination algorithm for massaging information in the hidden layer. In fact, this technique explores a continuum idea which combines the classic mutation and crossover strategies in GA together. The proposed method has the advantage over GA which requires a lot of preprocessing works in breaking down the data into a sequence of binary codes before learning or mutation can apply. Also, we set up a new error function to measure the performance of the FNN and obtain the optimal choice of the connection weights and thus the nonlinear optimization problem can be solved directly. Several computational experiments are used to illustrate the proposed algorithm, which has good exploration and exploitation capabilities in search of the optimal weight for single hidden layer FNNs.     

一种前馈神经网的快速算法

前馈神经网已经被大量用于非线性信号处理． 经典反向传播算法是一种标准的前馈网络学习算法,但是,对许多应用,反向传播算法的收 敛速度却很慢．本文根据对网络的非线性单元进行线性化而提出一种新的算法,该算法在非 线性信号处理中在精度和收敛速度方面都优于传统的反向传播算法．     

Optimal control of terminal processes using neural networks

Feedforward neural networks are capable of approximating continuous multivariate functions and, as such, can implement nonlinear state-feedback controllers. Training methods such as backpropagation-through-time (BPTT), however, do not deal with terminal control problems in which the specified cost function includes the elapsed trajectory-time. In this paper, an extension to BPTT is proposed which addresses this limitation. The controller design is reformulated as a constrained optimization problem defined over the entire field of extremals and in which the set of trajectory times is incorporated into the cost function. Necessary first-order stationary conditions are derived which correspond to standard BPTT with the addition of certain transversality conditions. The new gradient algorithm based on these conditions, called time-optimal backpropagation through time, is tested on two benchmark minimum-time control problems.     

Fast training of multilayer perceptrons

Training a multilayer perceptron by an error backpropagation algorithm is slow and uncertain. This paper describes a new approach which is much faster and certain than error backpropagation. The proposed approach is based on combined iterative and direct solution methods. In this approach, we use an inverse transformation for linearization of nonlinear output activation functions, direct solution matrix methods for training the weights of the output layer; and gradient descent, the delta rule, and other proposed techniques for training the weights of the hidden layers. The approach has been implemented and tested on many problems. Experimental results, including training times and recognition accuracy, are given. Generally, the approach achieves accuracy as good as or better than perceptrons trained using error backpropagation, and the training process is much faster than the error backpropagation algorithm and also avoids local minima and paralysis.     

Selection of optimal set of weights in a layered network using genetic algorithms

Genetic algorithms represent a class of highly parallel robust adaptive search processes for solving a wide range of optimization and machine learning problems. The present work is an attempt to demonstrate their effectiveness to search a global optimal solution to select a decision boundary for a pattern recognition problem using a multilayer perceptron. The proposed method incorporates a new concept of nonlinear selection for creating mating pools and a weighted error as a fitness function. Since there is no need for the backpropagation technique, the algorithm is computationally efficient and avoids all the drawbacks of the backpropagation algorithm. Moreover, it does not depend on the sequence of the training data. The performance of the method along with the convergence has been experimentally demonstrated for both linearly separable and nonseparable pattern classes.     

Optimized approximation algorithm in neural networks without overfitting.

In this paper, an optimized approximation algorithm (OAA) is proposed to address the overfitting problem in function approximation using neural networks (NNs). The optimized approximation algorithm avoids overfitting by means of a novel and effective stopping criterion based on the estimation of the signal-to-noise-ratio figure (SNRF). Using SNRF, which checks the goodness-of-fit in the approximation, overfitting can be automatically detected from the training error only without use of a separate validation set. The algorithm has been applied to problems of optimizing the number of hidden neurons in a multilayer perceptron (MLP) and optimizing the number of learning epochs in MLP's backpropagation training using both synthetic and benchmark data sets. The OAA algorithm can also be utilized in the optimization of other parameters of NNs. In addition, it can be applied to the problem of function approximation using any kind of basis functions, or to the problem of learning model selection when overfitting needs to be considered.     

Cost optimization of hybrid composite flywheel rotors for energy storage

A novel approach to composite flywheel rotor design is proposed. Flywheel development has been dominated by mobile applications where minimizing mass is critical. This technology is also attractive for various industrial applications. For these stationary applications, the design is considerably cost-driven. Hence, the energy-per-cost ratio was used as the objective function. Based on an analytical approach for calculating stresses in multi-rim hybrid composite rotors, the nonlinear optimization problem was solved using a multi-strategy optimization scheme that combines an evolutionary algorithm with a nonlinear interior-point method. The problem was solved for a sample rotor with varying cost ratio of the rim materials. Instead of an optimal solution per cost ratio, only four optimal designs were obtained with a sharp transition between designs at specific cost ratios. This sharp transition is explained by the intricate interplay that exists between the objective function and the nonlinear constraints imposed by the applied failure criteria.     

An improved algorithm for learning long-term dependency problems in adaptive processing of data structures

Many researchers have explored the use of neural-network representations for the adaptive processing of data structures. One of the most popular learning formulations of data structure processing is backpropagation through structure (BPTS). The BPTS algorithm has been successful applied to a number of learning tasks that involve structural patterns such as logo and natural scene classification. The main limitations of the BPTS algorithm are attributed to slow convergence speed and the long-term dependency problem for the adaptive processing of data structures. In this paper, an improved algorithm is proposed to solve these problems. The idea of this algorithm is to optimize the free learning parameters of the neural network in the node representation by using least-squares-based optimization methods in a layer-by-layer fashion. Not only can fast convergence speed be achieved, but the long-term dependency problem can also be overcome since the vanishing of gradient information is avoided when our approach is applied to very deep tree structures.     

A sparse kernel algorithm for online time series data prediction

Kernel based methods have been widely applied for signal analysis and processing. In this paper, we propose a sparse kernel based algorithm for online time series prediction. In classical kernel methods, the kernel function number is very large which makes them of a high computational cost and only applicable for off-line or batch learning. In online learning settings, the learning system is updated when each training sample is obtained and it requires a higher computational speed. To make the kernel methods suitable for online learning, we propose a sparsification method based on the Hessian matrix of the system loss function to continuously examine the significance of the new training sample in order to select a sparse dictionary (support vector set). The Hessian matrix is equivalent to the correlation matrix of sample inputs in the kernel weight updating using the recursive least square (RLS) algorithm. This makes the algorithm able to be easily implemented with an affordable computational cost for real-time applications. Experimental results show the ability of the proposed algorithm for both real-world and artificial time series data forecasting and prediction.     

Information theoretic clustering

Clustering is an important topic in pattern recognition. Since only the structure of the data dictates the grouping (unsupervised learning), information theory is an obvious criteria to establish the clustering rule. The paper describes a novel valley seeking clustering algorithm using an information theoretic measure to estimate the cost of partitioning the data set. The information theoretic criteria developed here evolved from a Renyi entropy estimator (A. Renyi, 1960) that was proposed recently and has been successfully applied to other machine learning applications (J.C. Principe et al., 2000). An improved version of the k-change algorithm is used in optimization because of the stepwise nature of the cost function and existence of local minima. Even when applied to nonlinearly separable data, the new algorithm performs well, and was able to find nonlinear boundaries between clusters. The algorithm is also applied to the segmentation of magnetic resonance imaging data (MRI) with very promising results     

基于粒计算的极限学习机模型设计与应用

在数据智能处理中属性重要度差异很大且具有高度非线性的特征,在这种情况下直接应用机器学习进行建模处理往往很难获得问题的有效解。针对此问题,文中探索了基于粒计算的属性重要度的排序方法且结合排序结果应用二元关系实现粒层划分算法;应用极限学习机对不同划分获得的粒层空间进行学习,进而对不同粒层空间的学习结果进行对比分析,从而获得最优划分与粒层;此外,将提出的粒度极限学习机模型应用于空气质量的预报问题,不仅加快了预报速度,而且获得的结果与实际预测高度吻合,实证了粒度极限学习机模型的有效性和可靠性。     

一种基于ADMM的非光滑损失在线优化算法

交替方向乘子法（ADMM）在机器学习问题研究中已有一些高效的实际应用,但为了适应大规模数据的处理和求解非光滑损失凸优化问题,文中提出对原ADMM进行改进,得到了损失函数线性化的ADMM的在线优化算法。该在线算法相较原算法具有操作简单、计算高效等特点。通过详尽的理论分析,文中证明了新在线算法的收敛性,并得到其在一般凸条件下具有目前最优的Regret界以及随机收敛速度。最后在与当今流行在线算法的对比实验中验证了新在线算法的高效可行性。     

基于新的改进粒子群算法的BP神经网络在拟合非线性函数中的应用

介绍了一种基于新的改进粒子群算法(NIPSO)的BP神经网络来解决拟合非线性函数所出现的误差较大的问题。此算法在粒子群优化算法基础上,分别让权重和学习因子非线性和线性变化,建立基于新的粒子群优化算法的新模型,再与BP神经网络结合之后来拟合非线性函数。结果表明,新的改进粒子群优化算法更加合理且高效地提高了BP神经网络的拟合能力,减小了拟合误差,提高了拟合精度。     

A fast feedforward training algorithm using a modified form of thestandard backpropagation algorithm

In this letter, a new approach for the learning process of multilayer feedforward neural network is introduced. This approach minimizes a modified form of the criterion used in the standard backpropagation algorithm. This criterion is based on the sum of the linear and the nonlinear quadratic errors of the output neuron. The quadratic linear error signal is appropriately weighted. The choice of the weighted design parameter is evaluated via rank convergence series analysis and asymptotic constant error values. The new proposed modified standard backpropagation algorithm (MBP) is first derived on a single neuron-based net and then extended to a general feedforward neural network. Simulation results of the 4-b parity checker and the circle in the square problem confirm that the performance of the MBP algorithm exceed the standard backpropagation (SBP) in the reduction of the total number of iterations and in the learning time.     

Fast Blind Equalization Using Complex-Valued MLP

The blind equalizers based on complex valued feedforward neural networks, for linear and nonlinear communication channels, yield better performance as compared to linear equalizers. The learning algorithms are, generally, based on stochastic gradient descent, as they are simple to implement. However, these algorithms show a slow convergence rate. In the blind equalization problem, the unavailability of the desired output signal and the presence of nonlinear activation functions make the application of recursive least squares algorithm difficult. In this letter, a new scheme using recursive least squares algorithm is proposed for blind equalization. The learning of weights of the output layer is obtained by using a modified version of constant modulus algorithm cost function. For the learning of weights of hidden layer neuron space adaptation approach is used. The proposed scheme results in faster convergence of the equalizer.     

异步粒子群优化算法在QoS组播路由中的应用

QoS组播路由问题是一个非线性的组合优化问题,已证明了该问题是NP完全问题。为适应下一代IP网络对实时信息传输的要求,在异步模式粒子群优化算法基础上,给出包含延迟、延迟抖动、带宽、丢包率和最小花费5个约束条件在内的QoS组播路由算法。该算法首先给出数学模型,设计适应度函数,再给出受限的网络模型,通过粒子群优化(PSO)算法最大化适应度函数来求解最优Steiner树。算法仿真实验结果表明:与遗传算法和同步模式的粒子群优化算法相比,该算法有较好的收敛速度和寻优效果。     

A Modified Backpropagation Training Algorithm for Feedforward Neural Networks*

In this paper, a new efficient learning procedure for training single hidden layer feedforward network is proposed. This procedure trains the output layer and the hidden layer separately. A new optimization criterion for the hidden layer is proposed. Existing methods to find fictitious teacher signal for the output of each hidden neuron, modified standard backpropagation algorithm and the new optimization criterion are combined to train the feedforward neural networks. The effectiveness of the proposed procedure is shown by the simulation results. *The work of P. Thangavel is partially supported by UGC, Government of India sponsored project.