The efficient incorporation of MLP features into automatic speech recognition systems

In recent years, the use of Multi-Layer Perceptron (MLP) derived acoustic features has become increasingly popular in automatic speech recognition systems. These features are typically used in combination with standard short-term spectral-based features, and have been found to yield consistent performance improvements. However there are a number of design decisions and issues associated with the use of MLP features for state-of-the-art speech recognition systems. Two modifications to the standard training/adaptation procedures are described in this work. First, the paper examines how MLP features, and the associated acoustic models, can be trained efficiently on large training corpora using discriminative training techniques. An approach that combines multiple individual MLPs is proposed, and this reduces the time needed to train MLPs on large amounts of data. In addition, to further speed up discriminative training, a lattice re-use method is proposed. The paper also examines how systems with MLP features can be adapted to a particular speakers, or acoustic environments. In contrast to previous work (where standard HMM adaptation schemes are used), linear input network adaptation is investigated. System performance is investigated within a multi-pass adaptation/combination framework. This allows the performance gains of individual techniques to be evaluated at various stages, as well as the impact in combination with other sub-systems. All the approaches considered in this paper are evaluated on an Arabic large vocabulary speech recognition task which includes both Broadcast News and Broadcast Conversation test data.     

Linear-least-squares initialization of multilayer perceptrons through backpropagation of the desired response

Training multilayer neural networks is typically carried out using descent techniques such as the gradient-based backpropagation (BP) of error or the quasi-Newton approaches including the Levenberg-Marquardt algorithm. This is basically due to the fact that there are no analytical methods to find the optimal weights, so iterative local or global optimization techniques are necessary. The success of iterative optimization procedures is strictly dependent on the initial conditions, therefore, in this paper, we devise a principled novel method of backpropagating the desired response through the layers of a multilayer perceptron (MLP), which enables us to accurately initialize these neural networks in the minimum mean-square-error sense, using the analytic linear least squares solution. The generated solution can be used as an initial condition to standard iterative optimization algorithms. However, simulations demonstrate that in most cases, the performance achieved through the proposed initialization scheme leaves little room for further improvement in the mean-square-error (MSE) over the training set. In addition, the performance of the network optimized with the proposed approach also generalizes well to testing data. A rigorous derivation of the initialization algorithm is presented and its high performance is verified with a number of benchmark training problems including chaotic time-series prediction, classification, and nonlinear system identification with MLPs.     

Two-Phase Construction of Multilayer Perceptrons Using Information Theory

This brief presents a two-phase construction approach for pruning both input and hidden units of multilayer perceptrons (MLPs) based on mutual information (MI). First, all features of input vectors are ranked according to their relevance to target outputs through a forward strategy. The salient input units of an MLP are thus determined according to the order of the ranking result and by considering their contributions to the network's performance. Then, the irrelevant features of input vectors can be identified and eliminated. Second, the redundant hidden units are removed from the trained MLP one after another according to a novel relevance measure. Compared with its related work, the proposed strategy exhibits better performance. Moreover, experimental results show that the proposed method is comparable or even superior to support vector machine (SVM) and support vector regression (SVR). Finally, the advantages of the MI-based method are investigated in comparison with the sensitivity analysis (SA)-based method.     

基于干线对的红外与可见光最优图像配准算法

提出了一种基于干线对的红外与可见光图像配准算法.该算法分4步:首先分别从基准图像和待配准图像中提取干线对,即对图像中满足特定条件的直线进行配对;然后按照一些准则寻找这两幅图像中的干线对所有可能的匹配情况,并组成一个集合;接着从该集合中寻找这样一个子集,在保证每个干线对最多出现在它的一个元素中的前提下,使得该子集所有元素的相似性测度之和最大且由它确定的配准误差最小,该文采用分支定限法解决了这一优化问题;最后由最优子集中的所有元素得到同名像点集,运用仿射变换模型,实现图像的配准.大量实验表明,文中提出的方法对红外与可见光遥感图像之间的配准是有效的.     

Nonlinear dynamic system identification using Chebyshev functionallink artificial neural networks

A computationally efficient artificial neural network (ANN) for the purpose of dynamic nonlinear system identification is proposed. The major drawback of feedforward neural networks, such as multilayer perceptrons (MLPs) trained with the backpropagation (BP) algorithm, is that they require a large amount of computation for learning. We propose a single-layer functional-link ANN (FLANN) in which the need for a hidden layer is eliminated by expanding the input pattern by Chebyshev polynomials. The novelty of this network is that it requires much less computation than that of a MLP. We have shown its effectiveness in the problem of nonlinear dynamic system identification. In the presence of additive Gaussian noise, the performance of the proposed network is found to be similar or superior to that of a MLP. A performance comparison in terms of computational complexity has also been carried out.     

Bayesian MLP neural networks for image analysis

We demonstrate the advantages of using Bayesian multi-layer perceptron (MLP) neural networks for image analysis. The Bayesian approach provides consistent way to do inference by combining the evidence from the data to prior knowledge from the problem. A practical problem with MLPs is to select the correct complexity for the model, i.e., the right number of hidden units or correct regularization parameters. The Bayesian approach offers efficient tools for avoiding overfitting even with very complex models, and facilitates estimation of the confidence intervals of the results. In this contribution we review the Bayesian methods for MLPs and present comparison results from two case studies. In the first case, MLPs were used to solve the inverse problem in electrical impedance tomography. The Bayesian MLP provided consistently better results than other methods. In the second case, the goal was to locate trunks of trees in forest scenes. With Bayesian MLP it was possible to use large number of potentially useful features and prior for determining the relevance of the features automatically.     

Face Detection Using Mixture of MLP Experts

This paper presents a face detection method which makes use of a modified mixture of experts. In order to improve the face detection accuracy, a novel structure is introduced which uses the multilayer perceptrons (MLPs), as expert and gating networks, and employs a new learning algorithm to adapt with the MLPs. We call this model Mixture of MLP Experts (MMLPE). Experiments using images from the CMU-130 test set demonstrate the robustness of our method in detecting faces with wide variations in pose, facial expression, illumination, and complex backgrounds. The MMLPE produces promising high detection rate of 98.8% with ten false positives.     

A robust cardinality-constrained model to address the machine loading problem

Several deterministic models have been proposed in the literature to solve the machine loading problem (MLP), which considers a set of product types to be produced on a set of machines using a set of tool types, and determines the quantity of each product type to be produced at each time period and the corresponding machine tool loading configuration. However, processing times are subject to random increases, which could impair the quality of a deterministic solution. Thus, we propose a robust MLP counterpart, searching for an approach that properly describes the uncertainty set of model parameters and, at the same time, ensures practical application. We exploit the cardinality-constrained approach, which considers a simple uncertainty set where all uncertain parameters belong to an interval, and allows tuning the robustness level by bounding the number of parameters that assume the worst value. The resulting plans provide accurate estimations on the minimum production level that a system achieves even in the worst conditions. The applicability of the robust MLP and the impact of robustness level have been tested on several problem variants, considering single- vs multi-machine and single- vs multi-period MLPs. We also consider the execution of the plans in a set of scenarios to evaluate the practical implications of MLP robustness. Results show the advantages of the robust formulation, in terms of improved feasibility of the plans, identification of the most critical tools and products, and evaluation of the maximum achievable performance in relation to the level of protection. Moreover, low computational times guarantee the applicability of the proposed robust MLP counterpart.     

Learning from Innate Behaviors: A Quantitative Evaluation of Neural Network Controllers

The aim was to investigate a method of developing mobile robot controllers based on ideas about how plastic neural systems adapt to their environment by extracting regularities from the amalgamated behavior of inflexible (nonplastic) innate subsystems interacting with the world. Incremental bootstrapping of neural network controllers was examined. The objective was twofold. First, to develop and evaluate the use of prewired or innate robot controllers to bootstrap backpropagation learning for Multilayer Perceptron (MLP) controllers. Second, to develop and evaluate a new MLP controller trained on the back of another bootstrapped controller. The experimental hypothesis was that MLPs would improve on the performance of controllers used to train them. The performances of the innate and bootstrapped MLP controllers were compared in eight experiments on the tasks of avoiding obstacles and finding goals. Four quantitative measures were employed: the number of sensorimotor loops required to complete a task; the distance traveled; the mean distance from walls and obstacles; the smoothness of travel. The overall pattern of results from statistical analyses of these quantities supported the hypothesis; the MLP controllers completed the tasks faster, smoother, and steered further from obstacles and walls than their innate teachers. In particular, a single MLP controller incrementally bootstrapped by a MLP subsumption controller was superior to the others.     

Learning from Innate Behaviors: A Quantitative Evaluation of Neural Network Controllers

The aim was to investigate a method of developing mobile robot controllers based on ideas about how plastic neural systems adapt to their environment by extracting regularities from the amalgamated behavior of inflexible (non-plastic) innate s ubsystems interacting with the world.Incremental bootstrapping of neural network controllers was examined. The objective was twofold. First, to develop and evaluate the use of prewired or innate robot controllers to bootstrap backpropagation learning for Multi-Layer Perceptron (MLP) controllers. Second, to develop and evaluate a new MLP controller trained on the back of another bootstrapped controller. The experimental hypothesis was that MLPs would improve on the performance of controllers used to train them. The performances of the innate and bootstrapped MLP controllers were compared in eight experiments on the tasks of avoiding obstacles and finding goals. Four quantitative measures were employed: the number of sensorimotor loops required to complete a task; the distance traveled; the mean distance from walls and obstacles; the smoothness of travel. The overall pattern of results from statistical analyses of these quantities su pported the hypothesis; the MLP controllers completed the tasks faster, smoother, and steered further from obstacles and walls than their innate teachers. In particular, a single MLP controller incrementally bootstrapped by a MLP subsumption controller was superior to the others.     

A novel training scheme for multilayered perceptrons to realize proper generalization and incremental learning

The response of a multilayered perceptron (MLP) network on points which are far away from the boundary of its training data is generally never reliable. Ideally a network should not respond to data points which lie far away from the boundary of its training data. We propose a new training scheme for MLPs as classifiers, which ensures this. Our training scheme involves training subnets for each class present in the training data. Each subnet can decide whether a data point belongs to a certain class or not. Training each subnet requires data from the class which the subnet represents along with some points outside the boundary of that class. For this purpose we propose an easy but approximate method to generate points outside the boundary of a pattern class. The trained subnets are then merged to solve the multiclass classification problem. We show through simulations that an MLP trained by our method does not respond to points which lies outside the boundary of its training sample. Also, our network can deal with overlapped classes in a better manner. In addition, this scheme enables incremental training of an MLP, i.e., the MLP can learn new knowledge without forgetting the old knowledge.     

VEP optimal channel selection using genetic algorithm for neural network classification of alcoholics.

In this letter, neural networks (NNs) classify alcoholics and nonalcoholics using features extracted from visual evoked potential (VEP). A genetic algorithm (GA) is used to select the minimum number of channels that maximize classification performance. GA population fitness is evaluated using fuzzy ARTMAP (FA) NN, instead of the widely used multilayer perceptron (MLP). MLP, despite its effective classification, requires long training time (on the order of 10(3) times compared to FA). This causes it to be unsuitable to be used with GA, especially for on-line training. It is shown empirically that the optimal channel configuration selected by the proposed method is unbiased, i.e., it is optimal not only for FA but also for MLP classification. Therefore, it is proposed that for future experiments, these optimal channels could be considered for applications that involve classification of alcoholics.     

Handwritten characters recognition based on nature-inspired computing and neuro-evolution

The enormous services obtainable by bank and postal systems are not 100 % guaranteed due to variability of handwriting styles. Various methods based on neural networks have been suggested to address this issue. Unfortunately, they often fall into local optima that arises from the use of old learning methods. Global optimization methods provided new directions for neural networks evolution that may be useful in recognition. This paper develops efficient algorithms that compute globally optimal solutions by exploiting the benefits of both swarm intelligence and neuro-evolution in a way to improve the overall performance of a character recognition system. Various adaptations implied to both MLP and RBF networks have been suggested namely: particle swarm optimization (PSO) and the bees algorithm (BA) for characters classification, MLP training or RBF design by co-evolution and effective combinations of MLPs, RBFs or SVMs as an attempt to overcome the drawbacks of old recognition methods. Results proved that networks combination proposals ensure the highest improvement compared to either standard MLP and RBF networks, the co-evolutionary alternatives or other classifiers combination based on common combination rules namely majority voting, the fusion rules of min, max, sum, average, product and Bayes, Decision template and the Behavior Knowledge Space (BKS).     

Multilayer perceptrons combination applied to handwritten character recognition

Several methods of combination of Multilayer Perceptrons (MLPs) for handwritten character recognition are presented and discussed. Recognition tests have shown that cooperation of neural networks using different features vectors can reduce significantly the overall misclassification error rate. Additionally, the MLPs that are combined are the results of the experiments that were previously performed in order to optimize the recognition process when using a single MLP. So, all the combination methods that are proposed are very easy to carry out. The final recognition system consists of a cascade association of small MLPs, which allows minimization of the overall recognition time while retaining a high recognition rate. This system appears to be 2.5 times faster than the best of the individual MLPs, while offering a recognition rate of 99.8% on unconstrained digits extracted from the NIST 3 database.     

Enhancing positioning accuracy of GPS/INS system during GPS outages utilizing artificial neural network

Integrated global positioning system and inertial navigation system (GPS/INS) have been extensively employed for navigation purposes. However, low-grade GPS/INS systems generate erroneous navigation solutions in the absence of GPS signals and drift very fast. We propose in this paper a novel method to integrate a low-grade GPS/INS with an artificial neural network (ANN) structure. Our method is based on updating the INS in a Kalman filter structure using ANN during GPS outages. This study focuses on the design, implementation and integration of such an ANN employing an optimum multilayer perceptron (MLP) structure with relevant number of layers/perceptrons and an appropriate learning. As a result, a land test is conducted with the proposed ANN + GPS/INS system and we here provide the system performance with the land trials.     

Compressing arrays of classifiers using Volterra-neural network: application to face recognition

Model compression is required when large models are used, for example, for a classification task, but there are transmission, space, time, or computing constraints that have to be fulfilled. Multilayer perceptron (MLP) models have been traditionally used as classifiers. Depending on the problem, they may need a large number of parameters (neuron functions, weights, and bias) to obtain an acceptable performance. This work proposes a technique to compress an array of MLPs, through the weights of a Volterra-neural network (Volterra-NN), maintaining its classification performance. It will be shown that several MLP topologies can be well-compressed into the first-, second-, and third-order (Volterra-NN) outputs. The obtained results show that these outputs can be used to build an array of (Volterra-NN) that needs significantly less parameters than the original array of MLPs, furthermore having the same high accuracy. The Volterra-NN compression capabilities were tested for solving a face recognition problem. Experimental results are presented on two well-known face databases: ORL and FERET.     

Task decomposition and modular single-hidden-layer perceptron classifiers for multi-class learning problems

One of keys for multilayer perceptrons (MLPs) to solve the multi-class learning problems is how to make them get good convergence and generalization performances merely through learning small-scale subsets, i.e., a small part of the original larger-scale data sets. This paper first decomposes an n-class problem into n two-class problems, and then uses n class-modular MLPs to solve them one by one. A class-modular MLP is responsible for forming the decision boundaries of its represented class, and thus can be trained only by the samples from the represented class and some neighboring ones. When solving a two-class problem, an MLP has to face with such unfavorable situations as unbalanced training data, locally sparse and weak distribution regions, and open decision boundaries. One of solutions is that the samples from the minority classes or in the thin regions are virtually reinforced by suitable enlargement factors. And next, the effective range of an MLP is localized by a correction coefficient related to the distribution of its represented class. In brief, this paper focuses on the formation of economic learning subsets, the virtual balance of imbalanced training sets, and the localization of generalization regions of MLPs. The results for the letter and the extended handwritten digital recognitions show that the proposed methods are effective.     

Genetic algorithm hybridized with multilayer perceptron to have an economical slope stability design

The present work aimed to evaluate and optimize the design of an artificial neural network (ANN) combined with an optimization algorithm of genetic algorithm (GA) for the calculation of slope stability safety factors (SF) in a pure cohesive slope. To make datasets of training and testing for the developed predictive models, 630 finite element limit equilibrium (FELE) analyses were performed. Similar to many artificial intelligence-based solutions, the database was involved in 189 testing datasets (e.g., 30% of the entire database) and 441 training datasets; for example, a range of 70% of the total database. Moreover, variables of multilayer perceptron (MLP) algorithm (for example, number of nodes in any hidden layer) and the algorithm of GA like population size was optimized by utilizing a series of trial and error process. The parameters in input, which were used in the analysis, consist of slope angle (β), setback distance ratio (b/B), applied stresses on the slope (F_y) and undrained shear strength of the cohesive soil (C_u) where the output was taken SF. The obtained network outputs for both datasets from MLP and GA-MLP models are evaluated according to many statistical indices. A total of 72 MLP trial and error (e.g., parameter study) the optimal architecture of 4 × 8 × 1 were determined for the MLP structure. Both proposed techniques result in a proper performance; however, according to the statistical indices, the GA–MLP model can somewhat accomplish the least mean square error (MSE) when compared to MLP. In an optimized GA–MLP network, coefficient of determination (R²) and root mean square error (RMSE) values of (0.975, and 0.097) and (0.969, and 0.107) were found, respectively, to both of the normalized training and testing datasets.

     

On the closure of the set of functions that can be realized by agiven multilayer perceptron

Given a multilayer perceptron (MLP) with a fixed architecture, there are functions that can be approximated up to any degree of accuracy, without having to increase the number of the hidden nodes. Those functions belong to the closure F of the set F of the maps realizable by the MLP. In this paper, we give a list of maps with this property. In particular, it is proven that: 1) rational functions belongs to F for networks with inverse tangent activation function; and 2) products of polynomials and exponentials belongs to F for networks with sigmoid activation function. Moreover, for a restricted class of MLPs, we prove that the list is complete and give an analytic definition of F.