Performing Feature Selection With Multilayer Perceptrons

An experimental study on two decision issues for wrapper feature selection (FS) with multilayer perceptrons and the sequential backward selection (SBS) procedure is presented. The decision issues studied are the stopping criterion and the network retraining before computing the saliency. Experimental results indicate that the increase in the computational cost associated with retraining the network with every feature temporarily removed before computing the saliency is rewarded with a significant performance improvement. Despite being quite intuitive, this idea has been hardly used in practice. A somehow nonintuitive conclusion can be drawn by looking at the stopping criterion, suggesting that forcing overtraining may be as useful as early stopping. A significant improvement in the overall results with respect to learning with the whole set of variables is observed.     

A nonmonotone accelerated Levenberg–Marquardt method for the -eigenvalues of symmetric tensors

The eigenvalues of tensors become more and more important in the numerical multilinear algebra. In this paper, based on the nonmonotone technique, an accelerated Levenberg–Marquardt (LM) algorithm is presented for computing the -eigenvalues of symmetric tensors, in which an LM step and an accelerated LM step are computed at each iteration. We establish the global convergence of the proposed algorithm using properties of symmetric tensors and norms. Under the local error-bound condition, the cubic convergence of the nonmonotone accelerated LM algorithm is derived. Numerical results show that this method is efficient.     

LMA: A generic and efficient implementation of the Levenberg–Marquardt Algorithm

This paper presents an open‐source, generic and efficient implementation of a very popular nonlinear optimization method: the Levenberg–Marquardt algorithm (LMA). This minimization algorithm is well known and hundreds of implementations have already been released. However, none of them offer at the same time a high level of genericity, a friendly syntax and a high computational performance. In this paper, we propose a solution to gather all those advantages in one library named LMA. The main challenge is to implement an efficient solver for every encounter problem. To overcome this difficulty, LMA uses compile time algorithms to design a code specific to the given optimization problem. The features of LMA are presented and the performances are compared with the state‐of‐the‐art best alternatives through extensive benchmarks on different kind of problems. Copyright © 2017 John Wiley & Sons, Ltd.     

Dynamics of Learning in Multilayer Perceptrons Near Singularities

The dynamical behavior of learning is known to be very slow for the multilayer perceptron, being often trapped in the “plateau.” It has been recently understood that this is due to the singularity in the parameter space of perceptrons, in which trajectories of learning are drawn. The space is Riemannian from the point of view of information geometry and contains singular regions where the Riemannian metric or the Fisher information matrix degenerates. This paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity. We give explicit asymptotic analytical solutions (trajectories) both for the standard gradient (SGD) and natural gradient (NGD) methods. It is clearly shown, in the case of the SGD method, that the plateau phenomenon appears in a neighborhood of the critical regions, where the dynamical behavior is extremely slow. The analysis of the NGD method is much more difficult, because the inverse of the Fisher information matrix diverges. We conquer the difficulty by introducing the “blow-down” technique used in algebraic geometry. The NGD method works efficiently, and the state converges directly to the true parameters very quickly while it staggers in the case of the SGD method. The analytical results are compared with computer simulations, showing good agreement. The effects of singularities on learning are thus qualitatively clarified for both standard and NGD methods.      

Evolving Multilayer Perceptrons

This paper proposes a new version of a method (G-Prop, genetic backpropagation) that attempts to solve the problem of finding appropriate initial weights and learning parameters for a single hidden layer Multilayer Perceptron (MLP) by combining an evolutionary algorithm (EA) and backpropagation (BP). The EA selects the MLP initial weights, the learning rate and changes the number of neurons in the hidden layer through the application of specific genetic operators, one of which is BP training. The EA works on the initial weights and structure of the MLP, which is then trained using QuickProp; thus G-Prop combines the advantages of the global search performed by the EA over the MLP parameter space and the local search of the BP algorithm. The application of the G-Prop algorithm to several real-world and benchmark problems shows that MLPs evolved using G-Prop are smaller and achieve a higher level of generalization than other perceptron training algorithms, such as QuickPropagation or RPROP, and other evolutive algorithms. It also shows some improvement over previous versions of the algorithm.     

Human Face Recognition Using Accelerated Multilayer Perceptrons

Standard back propagation, as with many gradient based optimization methods converges slowly as neural network training problems become larger and more complex. This paper describes the employment of two algorithms to accelerate the training procedure in an automatic human face recognition system. As compared to standard back propagation, the convergence rate is improved by up to 98% with only a minimal increase in the complexity of each iteration.     

Parameter by Parameter Algorithm for Multilayer Perceptrons

This paper presents a parameter by parameter (PBP) algorithm for speeding up the training of multilayer perceptrons (MLP). This new algorithm uses an approach similar to that of the layer by layer (LBL) algorithm, taking into account the input errors of the output layer and hidden layer. The proposed PBP algorithm, however, is not burdened by the need to calculate the gradient of the error function. In each iteration step, the weights or thresholds can be optimized directly one by one with other variables fixed. Four classes of solution equations for parameters of networks are deducted. The effectiveness of the PBP algorithm is demonstrated using two benchmarks. In comparisons with the BP algorithm with momentum (BPM) and the conventional LBL algorithms, PBP obtains faster convergences and better simulation performances.     

Development of a coupled wavelet transform and evolutionary Levenberg‐Marquardt neural networks for hydrological process modeling

This research aims to present a general framework by which the most appropriate wavelet parameters including mother wavelet, vanishing moment, and decomposition level can be chosen for a joint wavelet transform and machine learning model. This study is organized in 2 parts: the first part presents an evolutionary Levenberg‐Marquardt neural network (ELMNN) model as the most effective machine learning configuration, and the second part describes how the wavelet transform can be effectively embedded with the developed ELMNN model. In this research, the rainfall and runoff time series data of 2 distinct watersheds at 2 different time scales (daily and monthly) were used to build the proposed hybrid wavelet transform and ELMNN model. The conclusions of this study showed that the Daubechies wavelet more than other wavelet families is capable to extract the informative features of hydrologic series. The vanishing moment and decomposition level of this mother wavelet should be selected based on the watershed behavior and the time resolution of rainfall and runoff time series, respectively. The verification results for both watersheds at daily and monthly time scales indicated root mean square error, peak value criterion, low value criterion, and Kling‐Gupta efficiency as about 0.017, 0.021, 0.023, and 0.91, respectively.     

Two-Phase Construction of Multilayer Perceptrons Using Information Theory

This brief presents a two-phase construction approach for pruning both input and hidden units of multilayer perceptrons (MLPs) based on mutual information (MI). First, all features of input vectors are ranked according to their relevance to target outputs through a forward strategy. The salient input units of an MLP are thus determined according to the order of the ranking result and by considering their contributions to the network's performance. Then, the irrelevant features of input vectors can be identified and eliminated. Second, the redundant hidden units are removed from the trained MLP one after another according to a novel relevance measure. Compared with its related work, the proposed strategy exhibits better performance. Moreover, experimental results show that the proposed method is comparable or even superior to support vector machine (SVM) and support vector regression (SVR). Finally, the advantages of the MI-based method are investigated in comparison with the sensitivity analysis (SA)-based method.     

Obtaining Fault Tolerant Multilayer Perceptrons Using an Explicit Regularization

When the learning algorithm is applied to a MLP structure, different solutions for the weight values can be obtained if the parameters of the applied rule or the initial conditions are changed. Those solutions can present similar performance with respect to learning, but they differ in other aspects, in particular, fault tolerance against weight perturbations. In this paper, a backpropagation algorithm that maximizes fault tolerance is proposed. The algorithm presented explicitly adds a new term to the backpropagation learning rule related to the mean square error degradation in the presence of weight deviations in order to minimize this degradation. The results obtained demonstrate the efficiency of the learning rule proposed here in comparison with other algorithm.     

A New Method to Increase the Margin of Multilayer Perceptrons

A new method to maximize the margin of MLP classifier in classification problems is described. Thismethod is based on a new cost function which minimizes the variance ofthe mean squared error. We show that with this cost function the generalizationperformance increase. This method is tested and compared with the standard mean square errorand is applied to a face detection problem.     

Theoretical Properties of Projection Based Multilayer Perceptrons with Functional Inputs

Many real world data are sampled functions. As shown by Functional Data Analysis (FDA) methods, spectra, time series, images, gesture recognition data, etc. can be processed more efficiently if their functional nature is taken into account during the data analysis process. This is done by extending standard data analysis methods so that they can apply to functional inputs. A general way to achieve this goal is to compute projections of the functional data onto a finite dimensional sub-space of the functional space. The coordinates of the data on a basis of this sub-space provide standard vector representations of the functions. The obtained vectors can be processed by any standard method. In [43], this general approach has been used to define projection based Multilayer Perceptrons (MLPs) with functional inputs. We study in this paper important theoretical properties of the proposed model. We show in particular that MLPs with functional inputs are universal approximators: they can approximate to arbitrary accuracy any continuous mapping from a compact sub-space of a functional space to . Moreover, we provide a consistency result that shows that any mapping from a functional space to can be learned thanks to examples by a projection based MLP: the generalization mean square error of the MLP decreases to the smallest possible mean square error on the data when the number of examples goes to infinity.     

Learning with Permutably Homogeneous Multiple-Valued Multiple-Threshold Perceptrons

The (n,k,s)-perceptrons partition the input space V R ⁿ into s+1 regions using s parallel hyperplanes. Their learning abilities are examined in this research paper. The previously studied homogeneous (n,k,k–1)-perceptron learning algorithm is generalized to the permutably homogeneous (n,k,s)-perceptron learning algorithm with guaranteed convergence property. We also introduce a high capacity learning method that learns any permutably homogeneously separable k-valued function given as input.     

A Multilayer Perceptrons Model for the Stability of a Bipedal Robot

A neural network model is proposed as a means of controlling the dynamical equilibrium of a walking bipedal robot. As a criterion to determine the stability of such a robot in relation with the organization of the sensorimotor system, we have been making use of the ZMP (Zero Momentum Point). Simulations are used to check the convergence of the algorithm. In the generalization phase, it is shown that the neural network has the ability to stabilise the robot for motions which have not previously been learned. An extended model is proposed, which seeks to closely inspect the physiology of the cerebellar cortex.     

基于多层前馈型人工神经网络的抑郁症分类系统研究

多层前馈型人工神经网络(MLPANN)是应用广泛的一种人工神经网络。该文研究了用于抑郁症中医证候分类的一类MLPANN,设计了一种基于自定义网络结构及其他参数的BP训练算法的分类系统,并首次应用在抑郁症的中医证候分类研究中。该系统利用实际病症样本数据进行了训练和分类,结果表明系统具有很好的分类效果,可以用于指导抑郁症诊断和治疗。     

Error Concealment Using Adaptive Multilayer Perceptrons (MLPs) for Block-Based Image Coding

Image coding algorithms such as Vector Quantisation (VQ), JPEG and MPEG have been widely used for encoding image and video. These compression systems utilise block-based coding techniques to achieve a higher compression ratio. However, a cell loss or a random bit error during network transmission will permeate into the whole block, and then generate several damaged blocks. Therefore, an efficient Error Concealment (EC) scheme is essential for diminishing the impact of damaged blocks in a compressed image. In this paper, a novel adaptive EC algorithm is proposed to conceal the error for block-based image coding systems by using neural network techniques in the spatial domain. In the proposed algorithm, only the intra-frame information is used for reconstructing the image with damaged blocks. The information of pixels surrounding a damaged block is used to recover the errors using the neural network models. Computer simulation results show that the visual quality and the PSNR evaluation of a reconstructed image are significantly improved using the proposed EC algorithm.     

Strategies defined for the development of applications using Multilayer Perceptrons: A survey

Abstract: The success in developing an application employing the Multilayer perceptron (MLP) as knowledge representation form is very dependent on the degree of complexity that the structure of the application's domain has. Different mathematical and/or statistical techniques have been developed to subtract the maximum amount of information of this type from an available sample of the operating space associated to the task of interest. In the context of MLP it has been used to decide on the form the different intervening parameters of the network and/or related learning algorithm (LA) should have. This paper provides an overview of the processes that have been defined to generate network applications using the MLP model, giving particular attention to those based on the dynamic creation of a network's architecture through the application of different techniques for subtracting information about the operating domain in which the training set is subsumed.     

Learning Perceptrons and Piecewise Linear Classifiers Sensitive to Example Dependent Costs

This article considers the cost dependent construction of linear and piecewise linear classifiers. Classical learning algorithms from the fields of artificial neural networks and machine learning consider either no costs at all or allow only costs that depend on the classes of the examples that are used for learning. In contrast to class dependent costs, we consider costs that are example, i.e. feature and class dependent. We present a cost sensitive extension of a modified version of the well-known perceptron algorithm that can also be applied in cases, where the classes are linearly non-separable. We also present an extended version of the hybrid learning algorithm DIPOL, that can be applied in the case of linear non-separability, multi-modal class distributions, and multi-class learning problems. We show that the consideration of example dependent costs is a true extension of class dependent costs. The approach is general and can be extended to other neural network architectures like multi-layer perceptrons and radial basis function networks.     

基于多层忆阻脉冲神经网络的强化学习及应用

人工神经网络（Artificial neural networks,ANNs）与强化学习算法的结合显著增强了智能体的学习能力和效率.然而,这些算法需要消耗大量的计算资源,且难以硬件实现.而脉冲神经网络（Spiking neural networks,SNNs）使用脉冲信号来传递信息,具有能量效率高、仿生特性强等特点,且有利于进一步实现强化学习的硬件加速,增强嵌入式智能体的自主学习能力.不过,目前脉冲神经网络的学习和训练过程较为复杂,网络设计和实现方面存在较大挑战.本文通过引入人工突触的理想实现元件——忆阻器,提出了一种硬件友好的基于多层忆阻脉冲神经网络的强化学习算法.特别地,设计了用于数据——脉冲转换的脉冲神经元;通过改进脉冲时间依赖可塑性（Spiking-timing dependent plasticity,STDP）规则,使脉冲神经网络与强化学习算法有机结合,并设计了对应的忆阻神经突触;构建了可动态调整的网络结构,以提高网络的学习效率;最后,以Open AI Gym中的CartPole-v0（倒立摆）和MountainCar-v0（小车爬坡）为例,通过实验仿真和对比分析,验证了方案的有效性和相对于传统强化学习方法的优势.