A new approach to perceptron training

The training of perceptrons is discussed in the framework of nonsmooth optimization. An investigation of Rosenblatt's perceptron training rule shows that convergence or the failure to converge in certain situations can be easily understood in this framework. An algorithm based on results from nonsmooth optimization is proposed and its relation to the "constrained steepest descent" method is investigated. Numerical experiments verify that the "constrained steepest descent" algorithm may be further improved by the integration of methods from nonsmooth optimization.     

解灰色非线性规划问题的随机搜索算法

对带约束条件的灰色非线性规划问题进行了探讨,首先将原灰色约束非线性规划问题进行均值白化处理,转化成一个确定型的带约束条件的非线性规划问题,对该确定型的非线性约束规划问题提出一个基于分布估计算法的随机搜索方法,对所提出的求解方法的关键技术作了详细的说明并给出了具体的算法步骤。 初步的数值算例表明所提出的方法是可行有效的     

基于PSO求解随机期望值模型的混合智能算法

随机期望值模型是一类有着广泛应用背景的随机规划问题,为了寻找更为高效的求解随机期望值模型的算法,采用随机仿真产生样本训练BP网络以逼近随机函数,然后应用微粒群算法并以逼近随机函数的神经元网络作为适应值估计和实现为了检验解的可行性,从而提出了一种求解随机期望值模型的混合智能算法。最后通过两个实例的仿真结果说明了算法的正确性和有效性。     

IMPROVED CONVERGENCE THROUGH DYNAMIC PROGRAMMING APPROACH

New hybrid methods for solving the multiplayer perceptron optimization problem are proposed which use the computation capabilities of Bellman's dynamic programming (DP) method. To solve the neural network optimization problem, we consider the case of output neurons differently from that of hidden neurons. For the neurons of the output layer we apply the conventional DP and for the hidden neurons we apply a method based on gradient approach. Computer simulation shows that the new hybrid methods outperform the gradient-based optimization methods in converging speed and avoiding the local minimum.     

The solution of routing problem based on neural network and immunological algorithms

The paper offers a combined approach to training a multiple-layer perceptron with the immune optimization algorithm and simulated annealing method. The approach allows us to dynamically vary the convergence rate of the training process at different stages of best-solution search and decrease the run-time. The possibility to use a trained multiple-layer perceptron for tackling routing problems is considered.     

Sequential quadratic programming enhanced backtracking search algorithm

In this paper, we propose a new hybrid method called SQPBSA which combines backtracking search optimization algorithm (BSA) and sequential quadratic programming (SQP). BSA, as an exploration search engine, gives a good direction to the global optimal region, while SQP is used as a local search technique to exploit the optimal solution. The experiments are carried on two suits of 28 functions proposed in the CEC-2013 competitions to verify the performance of SQPBSA. The results indicate the proposed method is effective and competitive.     

A new hybrid particle swarm and simulated annealing stochastic optimization method

A novel hybrid particle swarm and simulated annealing stochastic optimization method is proposed. The proposed hybrid method uses both PSO and SA in sequence and integrates the merits of good exploration capability of PSO and good local search properties of SA. Numerical simulation has been performed for selection of near optimum parameters of the method. The performance of this hybrid optimization technique was evaluated by comparing optimization results of thirty benchmark functions of different dimensions with those obtained by other numerical methods considering three criteria. These criteria were stability, average trial function evaluations for successful runs and the total average trial function evaluations considering both successful and failed runs. Design of laminated composite materials with required effective stiffness properties and minimum weight design of a three-bar truss are addressed as typical applications of the proposed algorithm in various types of optimization problems. In general, the proposed hybrid PSO-SA algorithm demonstrates improved performance in solution of these problems compared to other evolutionary methods The results of this research show that the proposed algorithm can reliably and effectively be used for various optimization problems.     

Hybrid morphological methodology for software development cost estimation

In this paper we propose a hybrid methodology to design morphological-rank-linear (MRL) perceptrons in the problem of software development cost estimation (SDCE). In this methodology, we use a modified genetic algorithm (MGA) to optimize the parameters of the MRL perceptron, as well as to select an optimal input feature subset of the used databases, aiming at a higher accuracy level for SDCE problems. Besides, for each individual of MGA, a gradient steepest descent method is used to further improve the MRL perceptron parameters supplied by MGA. Finally, we conduct an experimental analysis with the proposed methodology using six well-known benchmark databases of software projects, where two relevant performance metrics and a fitness function are used to assess the performance of the proposed methodology, which is compared to classical machine learning models presented in the literature.     

On the initialization and optimization of multilayer perceptrons

Multilayer perceptrons are now widely used for pattern recognition, although the training remains a time consuming procedure often converging toward a local optimum. Moreover, as the optimum network size and topology are usually unknown, the search of this optimum requires a lot of networks to be trained. In this paper the authors propose a method for properly initializing the parameters (weights) of a two-layer perceptron, and for identifying (without the need for any error-backpropagation training) the most suitable network size and topology for solving the problem under investigation. The initialized network can then be optimized by means of the standard error-backpropagation (EBP) algorithm. The authors' method is applicable to any two-layer perceptron comprising concentric as well as squashing units on its hidden layer. The output units are restricted to squashing units, but direct connections from the input to the output layer are also accommodated. To illustrate the power of the method, results obtained for different classification tasks are compared to similar results obtained using a traditional error-backpropagation training starting from a random initial state.     

Second order spiking perceptrons

According to the diffusion approximation and usual approximation scheme, we present two more biologically plausible so called second order spiking perceptron (SOSP) and extended second order spiking perceptron (ESOSP) based on the integrate-and-fire model with renewal process inputs, which employ both first and second statistical representation, i.e., the means, variances and correlations of the synaptic input. We show through various examples that such perceptrons, even a single neuron, are able to perform various complex non-linear tasks like the XOR problem, which is impossible to be solved by traditional single-layer perceptrons. Here our perceptrons offer a significant advantage over classical models, in that they include the second order statistics in computations, specially in that the ESOSP considers the learning of variance in the training. Our ultimate purpose is to open up the possibility of carrying out a stochastic computation in neuronal networks.

A hybrid genetic algorithm for two-stage multi-item inventory system with stochastic demand

We study a two-stage, multi-item inventory system where stochastic demand occurs at stage 1, and nodes at stage 1 replenish their inventory from stage 2. Due to the complexity of stochastic inventory optimization in multi-echelon system, few analytical models and effective algorithms exist. In this paper, we establish exact stochastic optimization models by proposing a well-defined supply–demand process analysis and provide an efficient hybrid genetic algorithm (HGA) by introducing a heuristic search technique based on the tradeoff between the inventory cost and setup cost and improving the initial solution. Monte Carlo method is also introduced to simulate the actual demand and thus to approximate the long-run average cost. By numerical experiments, we compare the widely used installation policy and echelon policy and show that when variance of stochastic demand increase, echelon policy outperforms installation policy and, furthermore, the proposed heuristic search technique greatly enhances the search capacity of HGA.     

A quantum particle swarm optimization for the 0–1 generalized knapsack sharing problem

This study proposes a new hybrid heuristic approach that combines the quantum particle swarm optimization (QPSO) technique with a local search phase to solve the binary generalized knapsack sharing problem (GKSP). The approach also incorporates a heuristic repair operator that uses problem-specific knowledge instead of the penalty function technique commonly used for constrained problems. This study is the first to report on the application of the QPSO method to the GKSP. The efficiency of our proposed approach was tested on a large set of instances, and the results were compared to those produced by the commercial mixed integer programming solver CPLEX 12.5 of IBM-ILOG. The Experimental results demonstrated the good performance of the QPSO in solving the GKSP.     

Comparative analysis of backpropagation and the extended Kalmanfilter for training multilayer perceptrons

The relationship between backpropagation and extended Kalman filtering for training multilayer perceptrons is examined. These two techniques are compared theoretically and empirically using sensor imagery. Backpropagation is a technique from neural networks for assigning weights in a multilayer perceptron. An extended Kalman filter can also be used for this purpose. A brief review of the multilayer perceptron and these two training methods is provided. Then, it is shown that backpropagation is a degenerate form of the extended Kalman filter. The training rules are compared in two examples: an image classification problem using laser radar Doppler imagery and a target detection problem using absolute range images. In both examples, the backpropagation training algorithm is shown to be three orders of magnitude less costly than the extended Kalman filter algorithm in terms of a number of floating-point operations     

基于PSO求解随机相关机会规划的有效算法

随机相关机会规划是一类有着广泛应用背景的随机规划问题,通过采用随机仿真产生样本训练BP网络以逼近机会函数,然后应用微粒群算法并以逼近机会函数的神经网络作为适应值估计,从而提出了一种求解随机相关机会规划的混合智能算法。最后通过实例仿真说明了算法的正确性和有效性。     

Short-term wind speed forecasting based on a hybrid model

Wind power is currently one of the types of renewable energy with a large generation capacity. However, operation of wind power generation is very challenging because of the intermittent and stochastic nature of the wind speed. Wind speed forecasting is a very important part of wind parks management and the integration of wind power into electricity grids. As an artificial intelligence algorithm, radial basis function neural network (RBFNN) has been successfully applied into solving forecasting problems. In this paper, a novel approach named WTT–SAM–RBFNN for short-term wind speed forecasting is proposed by applying wavelet transform technique (WTT) into hybrid model which hybrids the seasonal adjustment method (SAM) and the RBFNN. Real data sets of wind speed in Northwest China are used to evaluate the forecasting accuracy of the proposed approach. To avoid the randomness caused by the RBFNN model or the RBFNN part of the hybrid model, all simulations in this study are repeated 30 times to get the average. Numerical results show that the WTT–SAM–RBFNN outperforms the persistence method (PM), multilayer perceptron neural network (MLP), RBFNN, hybrid SAM and RBFNN (SAM–RBFNN), and hybrid WTT and RBFNN (WTT–RBFNN). It is concluded that the proposed approach is an effective way to improve the prediction accuracy.     

Hybrid back-propagation training with evolutionary strategies

This work presents a hybrid algorithm for neural network training that combines the back-propagation (BP) method with an evolutionary algorithm. In the proposed approach, BP updates the network connection weights, and a ( \(1+1\) ) Evolutionary Strategy (ES) adaptively modifies the main learning parameters. The algorithm can incorporate different BP variants, such as gradient descent with adaptive learning rate (GDA), in which case the learning rate is dynamically adjusted by the stochastic ( \(1+1\) )-ES as well as the deterministic adaptive rules of GDA; a combined optimization strategy known as memetic search. The proposal is tested on three different domains, time series prediction, classification and biometric recognition, using several problem instances. Experimental results show that the hybrid algorithm can substantially improve upon the standard BP methods. In conclusion, the proposed approach provides a simple extension to basic BP training that improves performance and lessens the need for parameter tuning in real-world problems.     

A class of hybrid morphological perceptrons with application in time series forecasting

In this work a class of hybrid morphological perceptrons, called dilation–erosion perceptron (DEP), is presented to overcome the random walk dilemma in the time series forecasting problem. It consists of a convex combination of fundamental operators from mathematical morphology (MM) on complete lattices theory (CLT). A gradient steepest descent method is presented to design the proposed DEP (learning process), using the back propagation (BP) algorithm and a systematic approach to overcome the problem of nondifferentiability of morphological operators. The learning process includes an automatic phase fix procedure that is geared at eliminating time phase distortions observed in some time series. Finally, an experimental analysis is conducted with the proposed DEP using five real world time series, where five well-known performance metrics and an evaluation function are used to assess the forecasting performance of the proposed model. The obtained results are compared with those generated by classical forecasting models presented in the literature.     

Fast training of multilayer perceptrons

Training a multilayer perceptron by an error backpropagation algorithm is slow and uncertain. This paper describes a new approach which is much faster and certain than error backpropagation. The proposed approach is based on combined iterative and direct solution methods. In this approach, we use an inverse transformation for linearization of nonlinear output activation functions, direct solution matrix methods for training the weights of the output layer; and gradient descent, the delta rule, and other proposed techniques for training the weights of the hidden layers. The approach has been implemented and tested on many problems. Experimental results, including training times and recognition accuracy, are given. Generally, the approach achieves accuracy as good as or better than perceptrons trained using error backpropagation, and the training process is much faster than the error backpropagation algorithm and also avoids local minima and paralysis.     

An accelerated learning algorithm for multilayer perceptrons: optimization layer by layer.

Multilayer perceptrons are successfully used in an increasing number of nonlinear signal processing applications. The backpropagation learning algorithm, or variations hereof, is the standard method applied to the nonlinear optimization problem of adjusting the weights in the network in order to minimize a given cost function. However, backpropagation as a steepest descent approach is too slow for many applications. In this paper a new learning procedure is presented which is based on a linearization of the nonlinear processing elements and the optimization of the multilayer perceptron layer by layer. In order to limit the introduced linearization error a penalty term is added to the cost function. The new learning algorithm is applied to the problem of nonlinear prediction of chaotic time series. The proposed algorithm yields results in both accuracy and convergence rates which are orders of magnitude superior compared to conventional backpropagation learning.