A symbolic interpretation for back-propagation networks

Two main problems for the neural network (NN) paradigm are discussed: the output value interpretation and the symbolic content of the connection matrix. In this article, we construct a solution for a very common architecture of pattern associators: the backpropagation networks. First, we show how Zadeh's possibility theory brings a formal structure to the output interpretation. Properties and practical applications of this theory are developed. Second, a symbolic interpretation for the connection matrix is proposed by designing of an algorithm. By accepting the NN training examples as input this algorithm produces a set of implication rules. These rules accurately model the NN behavior. Moreover, they allow to understand it, especially in the cases of generalization or interference.     

On 'natural' learning and pruning in multi-layered perceptrons

Several studies have shown that natural gradient descent for on-line learning is much more efficient than standard gradient descent. In this article, we derive natural gradients in a slightly different manner and discuss implications for batch-mode learning and pruning, linking them to existing algorithms such as Levenberg-Marquardt optimization and optimal brain surgeon. The Fisher matrix plays an important role in all these algorithms. The second half of the article discusses a layered approximation of the Fisher matrix specific to multilayered perceptrons. Using this approximation rather than the exact Fisher matrix, we arrive at much faster "natural" learning algorithms and more robust pruning procedures.     

基于图嵌入正则化的人脸线性判别分析

提出一种基于图嵌入正则化的人脸线性判别分析方法。构造非监督最优类可分准则,基于图嵌入理论,求解该最优类可分准则下的最优投影向量,在非监督的图嵌入框架下利用样本局部类别信息提高人脸识别率,降低矩阵计算复杂度。在典型的人脸数据库上的实验证明了该方法的有效性。     

Parallel implementation of back-propagation algorithm in networks of workstations

This work presents an efficient mapping scheme for the multilayer perceptron (MLP) network trained using back-propagation (BP) algorithm on network of workstations (NOWs). Hybrid partitioning (HP) scheme is used to partition the network and each partition is mapped on to processors in NOWs. We derive the processing time and memory space required to implement the parallel BP algorithm in NOWs. The performance parameters like speed-up and space reduction factor are evaluated for the HP scheme and it is compared with earlier work involving vertical partitioning (VP) scheme for mapping the MLP on NOWs. The performance of the HP scheme is evaluated by solving optical character recognition (OCR) problem in a network of ALPHA machines. The analytical and experimental performance shows that the proposed parallel algorithm has better speed-up, less communication time, and better space reduction factor than the earlier algorithm. This work also presents a simple and efficient static mapping scheme on heterogeneous system. Using divisible load scheduling theory, a closed-form expression for number of neurons assigned to each processor in the NOW is obtained. Analytical and experimental results for static mapping problem on NOWs are also presented.     

Uncorrelated Multilinear Discriminant Analysis With Regularization and Aggregation for Tensor Object Recognition

This paper proposes an uncorrelated multilinear discriminant analysis (UMLDA) framework for the recognition of multidimensional objects, known as tensor objects. Uncorrelated features are desirable in recognition tasks since they contain minimum redundancy and ensure independence of features. The UMLDA aims to extract uncorrelated discriminative features directly from tensorial data through solving a tensor-to-vector projection. The solution consists of sequential iterative processes based on the alternating projection method, and an adaptive regularization procedure is incorporated to enhance the performance in the small sample size (SSS) scenario. A simple nearest-neighbor classifier is employed for classification. Furthermore, exploiting the complementary information from differently initialized and regularized UMLDA recognizers, an aggregation scheme is adopted to combine them at the matching score level, resulting in enhanced generalization performance while alleviating the regularization parameter selection problem. The UMLDA-based recognition algorithm is then empirically shown on face and gait recognition tasks to outperform four multilinear subspace solutions (MPCA, DATER, GTDA, TR1DA) and four linear subspace solutions (Bayesian, LDA, ULDA, R-JD-LDA).     

Regularization networks: fast weight calculation via Kalman filtering.

Regularization networks are nonparametric estimators obtained from the application of Tychonov regularization or Bayes estimation to the hypersurface reconstruction problem. Their main drawback back is that the computation of the weights scales as O(n(3)) where n is the number of data. In this paper, we show that for a class of monodimensional problems, the complexity can be reduced to O(n) by a suitable algorithm based on spectral factorization and Kalman filtering. Moreover, the procedure applies also to smoothing splines.     

Negative effects of sufficiently small initial weights on back-propagation neural networks

In the training of feedforward neural networks,it is usually suggested that the initial weights should be small in magnitude in order to prevent premature saturation.The aim of this paper is to point out the other side of the story:In some cases,the gradient of the error functions is zero not only for infinitely large weights but also for zero weights.Slow convergence in the beginning of the training procedure is often the result of sufficiently small initial weights.Therefore,we suggest that,in these cases,the initial values of the weights should be neither too large,nor too small.For instance,a typical range of choices of the initial weights might be something like(0.4,0.1) ∪(0.1,0.4),rather than(0.1,0.1) as suggested by the usual strategy.Our theory that medium size weights should be used has also been extended to a few commonly used transfer functions and error functions.Numerical experiments are carried out to support our theoretical findings.     

Implementation of two-dimensional systolic algorithms for multi-layered neural networks

A new two-dimensional systolic algorithm is proposed in this paper for parallel implementation of the multi-layered neural network. To reduce communication overhead, the input data flow is passed along the horizontal and vertical directions of the systolic array alternately, over different layers of the neural network. This new algorithm accelerates learning process of the neural network. Transputer implementation of the proposed algorithm and experimental results are presented to show efficiency of the new algorithm.     

基于独立分量分析和BP网络的电子鼻模式识别

为了提高电子鼻对混合气体的识别率,针对气体传感器阵列的交叉敏感特性,探讨了在电子鼻系统中基于独立分量分析(ICA)算法与BP神经网络相结合进行模式识别的可行性。并对4个气体传感器组成的电子鼻对4种气体混合物所测得的原始数据进行处理,结果表明:ICA算法对数据进行有效预分类,减少了样本之间的相关性,将生成的新样本作为BP网络的输入,使网络结构简化,在保证一定正确率的前提下,大大提高网络的学习速度。利用该方法可以提高电子鼻识别混合气体的准确率。     

Discriminant component analysis for privacy protection and visualization of big data

Multimedia Tools and Applications - Big data has many divergent types of sources, from physical (sensor/IoT) to social and cyber (web) types, rendering it messy and, imprecise, and incomplete. Due...     

引入概率分布的深度神经网络贪婪剪枝

目的 深度学习在自动驾驶环境感知中的应用,将极大提升感知系统的精度和可靠性,但是现有的深度学习神经网络模型因其计算量和存储资源的需求难以部署在计算资源有限的自动驾驶嵌入式平台上。因此为解决应用深度神经网络所需的庞大计算量与嵌入式平台有限的计算能力之间的矛盾,提出了一种基于权重的概率分布的贪婪网络剪枝方法,旨在减少网络模型中的冗余连接,提高模型的计算效率。方法 引入权重的概率分布,在训练过程中记录权重参数中较小值出现的概率。在剪枝阶段,依据训练过程中统计的权重概率分布进行增量剪枝和网络修复,改善了目前仅以权重大小为依据的剪枝策略。结果 经实验验证,在Cifar10数据集上,在各个剪枝率下本文方法相比动态网络剪枝策略的准确率更高。在ImageNet数据集上,此方法在较小精度损失的情况下,有效地将AlexNet、VGG（visual geometry group）16的参数数量分别压缩了5.9倍和11.4倍,且所需的训练迭代次数相对于动态网络剪枝策略更少。另外对于残差类型网络ResNet34和ResNet50也可以进行有效的压缩,其中对于ResNet50网络,在精度损失增加较小的情况下,相比目前最优的方法HRank实现了更大的压缩率（2.1倍）。结论 基于概率分布的贪婪剪枝策略解决了深度神经网络剪枝的不确定性问题,进一步提高了模型压缩后网络的稳定性,在实现压缩网络模型参数数量的同时保证了模型的准确率。     

Derivative and GA-based methods in metamodeling of back-propagation neural networks for constrained approximate optimization

Artificial neural networks (ANN) have been extensively used as global approximation tools in the context of approximate optimization. ANN traditionally minimizes the absolute difference between target outputs and approximate outputs thereby resulting in approximate optimal solutions being sometimes actually infeasible when it is used as a metamodel for inequality constraint functions. The paper explores the development of the efficient back-propagation neural network (BPN)-based metamodel that ensures the constraint feasibility of approximate optimal solution. The BPN architecture is optimized via two approaches of both derivative-based method and genetic algorithm (GA) to determine interconnection weights between layers in the network. The verification of the proposed approach is examined by adopting a standard ten-bar truss problem. Finally, a GA-based approximate optimization of suspension with an optical flying head is conducted to enhance the shock resistance capability in addition to dynamic characteristics.     

Cost-Sensitive back-propagation neural networks with binarization techniques in addressing multi-class problems and non-competent classifiers

Multi-class classification problems can be addressed by using decomposition strategy. One of the most popular decomposition techniques is the One-vs-One (OVO) strategy, which consists of dividing multi-class classification problems into as many as possible pairs of easier-to-solve binary sub-problems. To discuss the presence of classes with different cost, in this paper, we examine the behavior of an ensemble of Cost-Sensitive Back-Propagation Neural Networks (CSBPNN) with OVO binarization techniques for multi-class problems. To implement this, the original multi-class cost-sensitive problem is decomposed into as many sub-problems as possible pairs of classes and each sub-problem is learnt in an independent manner using CSBPNN. Then a combination method is used to aggregate the binary cost-sensitive classifiers. To verify the synergy of the binarization technique and CSBPNN for multi-class cost-sensitive problems, we carry out a thorough experimental study. Specifically, we first develop the study to check the effectiveness of the OVO strategy for multi-class cost-sensitive learning problems. Then, we develop a comparison of several well-known aggregation strategies in our scenario. Finally, we explore whether further improvement can be achieved by using the management of non-competent classifiers. The experimental study is performed with three types of cost matrices and proper statistical analysis is employed to extract the meaningful findings.     

Empirical results of using back-propagation neural networks toseparate single echoes from multiple echoes

Empirical results illustrate the pitfalls of applying an artificial neural network (ANN) to classification of underwater active sonar returns. During training, a back-propagation ANN classifier learns to recognize two classes of reflected active sonar waveforms: waveforms having two major sonar echoes or peaks and those having one major echo or peak. It is shown how the classifier learns to distinguish between the two classes. Testing the ANN classifier with different waveforms of each type generated unexpected results: the number of echo peaks was nor the feature used to separate classes.     

Symbolic interpretation of artificial neural networks

Hybrid intelligent systems that combine knowledge-based and artificial neural network systems typically have four phases, involving domain knowledge representation, mapping of this knowledge into an initial connectionist architecture, network training and rule extraction, respectively. The final phase is important because it can provide a trained connectionist architecture with explanation power and validate its output decisions. Moreover, it can be used to refine and maintain the initial knowledge acquired from domain experts. In this paper, we present three rule extraction techniques. The first technique extracts a set of binary rules from any type of neural network. The other two techniques are specific to feedforward networks, with a single hidden layer of sigmoidal units. Technique 2 extracts partial rules that represent the most important embedded knowledge with an adjustable level of detail, while the third technique provides a more comprehensive and universal approach. A rule-evaluation technique, which orders extracted rules based on three performance measures, is then proposed. The three techniques area applied to the iris and breast cancer data sets. The extracted rules are evaluated qualitatively and quantitatively, and are compared with those obtained by other approaches     

Bayesian neural networks at scale: a performance analysis and pruning study

Bayesian neural networks (BNNs) are a promising method of obtaining statistical uncertainties for neural network predictions but with a higher computational overhead which can limit their practical usage. This work explores the use of high-performance computing with distributed training to address the challenges of training BNNs at scale. We present a performance and scalability comparison of training the VGG-16 and Resnet-18 models on a Cray-XC40 cluster. We demonstrate that network pruning can speed up inference without accuracy loss and provide an open-source software package, BPrune, to automate this pruning. For certain models we find that pruning up to 80% of the network results in only a 7.0% loss in accuracy. With the development of new hardware accelerators for deep learning, BNNs are of considerable interest for benchmarking performance. This analysis of training a BNN at scale outlines the limitations and benefits compared to a conventional neural network.

     

An evaluation of back-propagation neural networks for the optimal design of structural systems: Part I. Training procedures

Design optimization using approximations based on feed-forward back-propagation neural network is the topic of much recent research. The neural network schemes that have been proposed in the literature for optimal design of structural systems differ in their architecture and training procedures. Furthermore, their utility vis-a-vis classical optimization techniques is not always clear. A systematic comparison of the efficiency and accuracy of the neural network-based solution schemes to classical structural optimization techniques is the aim of this and the companion paper. In this paper, the neural network training procedures used in the present evaluation are described in detail. When using first-order nonlinear programming algorithms with neural networks, the ability to approximate derivatives is important. Therefore, mainly for completeness of evaluation, two new training methods that use the derivative information are proposed in addition to the now common function-based training method. The first method uses the derivatives to create additional training points in the vicinity of the original points, based on Taylor's series expansion. The second method attempts to minimize the error in derivatives while imposing the error in output functions as constraint. Expressions for analytical derivatives are derived for both function-based and derivative-based training. Significant savings in computational time are reported when calculating derivatives using built-in analytical derivatives instead of using finite difference derivatives. In the companion paper the proposed methods are applied to solve five optimization problems with varying degree of complexity. Approximately 1100 test cases are executed in the companion paper to compare the accuracy and efficiency of neural network-based optimization with the classical approaches.     

The generation and interpretation of false-colour composite principal component images

The presentation of multichannel image information in terms of a false-colour composite can, in general, be achieved with less sacrifice of data if the three leading principal component images are used rather than the three conventional raw channels

A hardwired electronic system which has been developed is described in outline; it implements principal components analysis on 256 x 256 four-channel array of pixels in a few seconds. The system is under the control of an APPLE II microcomputer, and can generate colour hard-copy output in the form of composites of the principal component images

The use of this system is illustrated by reference to a LANDSAT MSS subframe of southern Spain. A standard false-colour composite of this scene contains 73 per cent of the available image variance whereas the principal component composite contains 98 per cent. The necessity of contrast stretching the minor principal component images in order to generate a visually effective colour composite is demonstrated. Some consideration is given to the choice of the most appropriate colour coding in the principal component composite images.     

基于遗传算法与人工神经网络相结合的固体推进剂燃速计算

为了提高固体推进剂燃速预示精度,将遗传算法(Genetic Algorithm)与误差反传(Back Propagation)网络结构模型相结合,设计了用遗传算法优化神经网络权重的新方法。以固体推进剂燃速数据库为基础,对推进剂的燃速进行了预估,并与BP算法进行了比较。结果显示,预估值与实际值接近,误差小于BP算法模型,具有良好的预示效果,为推进剂燃速预估提供了新方法。