Neural and Wavelet Network Models for Financial Distress Classification

This work analyzes the use of linear discriminant models, multi-layer perceptron neural networks and wavelet networks for corporate financial distress prediction. Although simple and easy to interpret, linear models require statistical assumptions that may be unrealistic. Neural networks are able to discriminate patterns that are not linearly separable, but the large number of parameters involved in a neural model often causes generalization problems. Wavelet networks are classification models that implement nonlinear discriminant surfaces as the superposition of dilated and translated versions of a single “mother wavelet” function. In this paper, an algorithm is proposed to select dilation and translation parameters that yield a wavelet network classifier with good parsimony characteristics. The models are compared in a case study involving failed and continuing British firms in the period 1997–2000. Problems associated with over-parameterized neural networks are illustrated and the Optimal Brain Damage pruning technique is employed to obtain a parsimonious neural model. The results, supported by a re-sampling study, show that both neural and wavelet networks may be a valid alternative to classical linear discriminant models.     

Discriminant component pruning. Regularization and interpretation of multi-layered back-propagation networks

Neural networks are often employed as tools in classification tasks. The use of large networks increases the likelihood of the task's being learned, although it may also lead to increased complexity. Pruning is an effective way of reducing the complexity of large networks. We present discriminant components pruning (DCP), a method of pruning matrices of summed contributions between layers of a neural network. Attempting to interpret the underlying functions learned by the network can be aided by pruning the network. Generalization performance should be maintained at its optimal level following pruning. We demonstrate DCP's effectiveness at maintaining generalization performance, applicability to a wider range of problems, and the usefulness of such pruning for network interpretation. Possible enhancements are discussed for the identification of the optimal reduced rank and inclusion of nonlinear neural activation functions in the pruning algorithm.     

Pruning recurrent neural networks for improved generalizationperformance

Determining the architecture of a neural network is an important issue for any learning task. For recurrent neural networks no general methods exist that permit the estimation of the number of layers of hidden neurons, the size of layers or the number of weights. We present a simple pruning heuristic that significantly improves the generalization performance of trained recurrent networks. We illustrate this heuristic by training a fully recurrent neural network on positive and negative strings of a regular grammar. We also show that rules extracted from networks trained with this pruning heuristic are more consistent with the rules to be learned. This performance improvement is obtained by pruning and retraining the networks. Simulations are shown for training and pruning a recurrent neural net on strings generated by two regular grammars, a randomly-generated 10-state grammar and an 8-state, triple-parity grammar. Further simulations indicate that this pruning method can have generalization performance superior to that obtained by training with weight decay.     

Influence of the optimization methods on neural state estimation quality of the drive system with elasticity

The paper deals with the implementation of optimized neural networks (NNs) for state variable estimation of the drive system with an elastic joint. The signals estimated by NNs are used in the control structure with a state-space controller and additional feedbacks from the shaft torque and the load speed. High estimation quality is very important for the correct operation of a closed-loop system. The precision of state variables estimation depends on the generalization properties of NNs. A short review of optimization methods of the NN is presented. Two techniques typical for regularization and pruning methods are described and tested in detail: the Bayesian regularization and the Optimal Brain Damage methods. Simulation results show good precision of both optimized neural estimators for a wide range of changes of the load speed and the load torque, not only for nominal but also changed parameters of the drive system. The simulation results are verified in a laboratory setup.     

引入概率分布的深度神经网络贪婪剪枝

目的 深度学习在自动驾驶环境感知中的应用,将极大提升感知系统的精度和可靠性,但是现有的深度学习神经网络模型因其计算量和存储资源的需求难以部署在计算资源有限的自动驾驶嵌入式平台上。因此为解决应用深度神经网络所需的庞大计算量与嵌入式平台有限的计算能力之间的矛盾,提出了一种基于权重的概率分布的贪婪网络剪枝方法,旨在减少网络模型中的冗余连接,提高模型的计算效率。方法 引入权重的概率分布,在训练过程中记录权重参数中较小值出现的概率。在剪枝阶段,依据训练过程中统计的权重概率分布进行增量剪枝和网络修复,改善了目前仅以权重大小为依据的剪枝策略。结果 经实验验证,在Cifar10数据集上,在各个剪枝率下本文方法相比动态网络剪枝策略的准确率更高。在ImageNet数据集上,此方法在较小精度损失的情况下,有效地将AlexNet、VGG（visual geometry group）16的参数数量分别压缩了5.9倍和11.4倍,且所需的训练迭代次数相对于动态网络剪枝策略更少。另外对于残差类型网络ResNet34和ResNet50也可以进行有效的压缩,其中对于ResNet50网络,在精度损失增加较小的情况下,相比目前最优的方法HRank实现了更大的压缩率（2.1倍）。结论 基于概率分布的贪婪剪枝策略解决了深度神经网络剪枝的不确定性问题,进一步提高了模型压缩后网络的稳定性,在实现压缩网络模型参数数量的同时保证了模型的准确率。     

A unified mathematical form for removing neurons based on orthogonal projection and crosswise propagation

It is a common practice to adjust the number of hidden neurons in training, and the removal of neurons in neural networks plays an indispensable role in this architecture manipulation. In this paper, a succinct and unified mathematical form is upgraded to the generic case for removing neurons based on orthogonal projection and crosswise propagation in a feedforward layer with different architectures of neural networks, and further developed for several neural networks with different architectures. For a trained neural network, the method is divided into three stages. In the first stage, the output vectors of the feedforward observation layer are classified to clusters. In the second stage, the orthogonal projection is performed to locate a neuron whose output vector can be approximated by the other output vectors in the same cluster with the least information loss. In the third stage, the previous located neuron is removed and the crosswise propagation is implemented in each cluster. On accomplishment of the three stages, the neural network with the pruned architecture is retrained. If the number of clusters is one, the method is degenerated into its special case with only one neuron being removed. The applications to different architectures of neural networks with an extension to the support vector machine are exemplified. The methodology supports in theory large-scale applications of neural networks in the real world. In addition, with minor modifications, the unified method is instructive in pruning other networks as far as they have similar network structure to the ones in this paper. It is concluded that the unified pruning method in this paper equips us an effective and powerful tool to simplify the architecture in neural networks.     

神经网络的两种结构优化算法研究

提出了一种基于权值拟熵的“剪枝算法”与权值敏感度相结合的新方法，在“剪枝算法”中将权值拟熵作为惩罚项加入目标函数中，使多层前向神经网络在学习过程中自动约束权值分布，并以权值敏感度作为简化标准，避免了单纯依赖权值大小剪枝的随机性．同时，又针对剪枝算法在优化多输入多输出网络过程中计算量大、效率不高的问题，提出了一种在级联—相关(cascade correlation, CC)算法的基础上从适当的网络结构开始对网络进行构建的快速“构造算法”．仿真结果表明这种快速构造算法在收敛速度、运行效率乃至泛化性能上都更胜一筹．     

基于强化学习的无标签网络剪枝

为了消除深度神经网络中的冗余结构,找到具备较好性能和复杂度之间平衡性的网络结构,提出基于无标签的全局学习方法(LFGCL).LFGCL学习基于网络体系结构表示的全局剪枝策略,可有效避免以逐层方式修剪网络而导致的次优压缩率.在剪枝过程中不依赖数据标签,输出与基线网络相似的特征,优化网络体系结构.通过强化学习推断所有层的压...     

Bayesian neural networks at scale: a performance analysis and pruning study

Bayesian neural networks (BNNs) are a promising method of obtaining statistical uncertainties for neural network predictions but with a higher computational overhead which can limit their practical usage. This work explores the use of high-performance computing with distributed training to address the challenges of training BNNs at scale. We present a performance and scalability comparison of training the VGG-16 and Resnet-18 models on a Cray-XC40 cluster. We demonstrate that network pruning can speed up inference without accuracy loss and provide an open-source software package, BPrune, to automate this pruning. For certain models we find that pruning up to 80% of the network results in only a 7.0% loss in accuracy. With the development of new hardware accelerators for deep learning, BNNs are of considerable interest for benchmarking performance. This analysis of training a BNN at scale outlines the limitations and benefits compared to a conventional neural network.

     

A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems

Optimizing the structure of neural networks is an essential step for the discovery of knowledge from data. This paper deals with a new approach which determines the insignificant input and hidden neurons to detect the optimum structure of a feedforward neural network. The proposed pruning algorithm, called as neural network pruning by significance (N2PS), is based on a new significant measure which is calculated by the Sigmoidal activation value of the node and all the weights of its outgoing connections. It considers all the nodes with significance value below the threshold as insignificant and eliminates them. The advantages of this approach are illustrated by implementing it on six different real datasets namely iris, breast-cancer, hepatitis, diabetes, ionosphere and wave. The results show that the proposed algorithm is quite efficient in pruning the significant number of neurons on the neural network models without sacrificing the networks performance.     

基于粗糙集理论的RBF神经网络剪枝算法

分析了满足给定学习误差要求的最小结构神经网络的各种实现方法.把粗糙集理论引入神经网络的结构构造中;提出了一种基于粗糙集理论的RBF神经网络剪枝算法,并将这种算法与现有剪枝算法相比较.最后将该算法应用于热工过程中过热气温动态特性建模.仿真结果表明基于该算法的神经网络模型具有较高的建模精度以及泛化能力.     

基于信息强度的RBF神经网络结构设计研究

在系统研究前馈神经网络的基础上,针对径向基函数(Radial basis function, RBF) 网络的结构设计问题,提出一种弹性RBF神经网络结构优化设计方法. 利用隐含层神经元的输出信息(Output-information, OI)以及隐含层神经元与输出层神经元间的交互信息(Multi-information, MI)分析网络的连接强度, 以此判断增加或删除RBF神经网络隐含层神经元, 同时调整神经网络的拓扑结构,有效地解决了RBF神经网络结构设计问题; 利用梯度下降的参数修正算法保证了最终RBF网络的精度, 实现了神经网络的结构和参数自校正. 通过对典型非线性函数的逼近与污水处理过程关键水质参数建模, 结果证明了该弹性RBF具有良好的动态特征响应能力和逼近能力, 尤其是在训练速度、泛化能力、最终网络结构等方面较之最小资源神经网络(Minimal resource allocation net works, MRAN)、增长修剪RBF 神经网络(Generalized growing and pruning RBF, GGAP-RBF)和自组织RBF神经网络(Self-organizing RBF, SORBF)有较大的提高.     

一种径向基函数神经网络预测在超临界温度控制系统中的应用研究

超临界温度控制系统具有较大的惯性、时滞和非线性,且动态特性随运行工况而改变,难以建立其精确的数学模型,本文采用GGAP算法的RBF神经网络构成神经网络预测控制器,将在线学习和预测控制相结合,以某超临界电厂主汽温度为研究对象,MATLAB仿真实验表明,该方法能对超临界温度控制系统实现有效的控制,动态性能较传统的PID控制有较大的提高。     

溶解氧浓度的直接自适应动态神经网络控制方法

针对污水处理过程溶解氧浓度的控制问题,提出一种直接自适应动态神经网络控制方法(direct adaptive dynamic neural network control,DADNNC).构建的控制系统主要包括神经网络控制器和补偿控制器.神经网络控制器由自组织模糊神经网络实现系统状态与控制量之间的映射;提出一种基于规则无用率的结构修剪算法,并给出结构调整后网络收敛的理论证明.同时,为保证系统稳定,设计补偿控制器减小网络逼近误差,参数调整由Layapunov理论给出.国际基准仿真平台上的实验表明,与固定结构神经网络控制器、PID和模型预测控制等已有控制方法相比,DADNNC方法具有更高的控制精度和更强的适应能力.     

A formal selection and pruning algorithm for feedforward artificialneural network optimization

A formal selection and pruning technique based on the concept of local relative sensitivity index is proposed for feedforward neural networks. The mechanism of backpropagation training algorithm is revisited and the theoretical foundation of the improved selection and pruning technique is presented. This technique is based on parallel pruning of weights which are relatively redundant in a subgroup of a feedforward neural network. Comparative studies with a similar technique proposed in the literature show that the improved technique provides better pruning results in terms of reduction of model residues, improvement of generalization capability and reduction of network complexity. The effectiveness of the improved technique is demonstrated in developing neural network models of a number of nonlinear systems including three bit parity problem, Van der Pol equation, a chemical processes and two nonlinear discrete-time systems using the backpropagation training algorithm with adaptive learning rate.     

A fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks

In this paper, we present a fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks (FAOS-PFNN), where a novel structure learning algorithm incorporating a pruning strategy into new growth criteria is developed. The proposed growing procedure without pruning not only speeds up the online learning process but also facilitates a more parsimonious fuzzy neural network while achieving comparable performance and accuracy by virtue of the growing and pruning strategy. The FAOS-PFNN starts with no hidden neurons and parsimoniously generates new hidden units according to the proposed growth criteria as learning proceeds. In the parameter learning phase, all the free parameters of hidden units, regardless of whether they are newly created or originally existing, are updated by the extended Kalman filter (EKF) method. The effectiveness and superiority of the FAOS-PFNN paradigm is compared with other popular approaches like resource allocation network (RAN), RAN via the extended Kalman filter (RANEKF), minimal resource allocation network (MRAN), adaptive-network-based fuzzy inference system (ANFIS), orthogonal least squares (OLS), RBF-AFS, dynamic fuzzy neural networks (DFNN), generalized DFNN (GDFNN), generalized GAP-RBF (GGAP-RBF), online sequential extreme learning machine (OS-ELM) and self-organizing fuzzy neural network (SOFNN) on various benchmark problems in the areas of function approximation, nonlinear dynamic system identification, chaotic time-series prediction and real-world regression problems. Simulation results demonstrate that the proposed FAOS-PFNN algorithm can achieve faster learning speed and more compact network structure with comparably high accuracy of approximation and generalization.     

Structure minimization using the impact factor in neural networks

Despite many advances, the problem of determining the proper size of a neural network is important, especially for its practical implications in such issues as learning and generalization. Unfortunately, it is not usually obvious which size is best; a system that is too small will not be able to learn the data, while one that is just big enough may learn very slowly and be very sensitive to initial conditions and learning parameters. There are two types of approach to determining the network size: pruning and growing. Pruning consists of training a network which is larger than necessary, and then removing unnecessary weights/nodes. Here, a new pruning method is developed, based on the penalty-term method. This method makes the neural networks good for generalization, and reduces the retraining time needed after pruning weights/nodes. This work was presented, in part, at the 6th International Symposium on Artificial Life and Robotics, Tokyo, Japan, January 15–17, 2001.     

Iterative pruning in second-order recurrent neural networks

An iterative pruning method for second-order recurrent neural networks is presented. Each step consists in eliminating a unit and adjusting the remaining weights so that the network performance does not worsen over the training set. The pruning process involves solving a linear system of equations in the least-squares sense. The algorithm also provides a criterion for choosing the units to be removed, which works well in practice. Initial experimental results demonstrate the effectiveness of the proposed approach over high-order architectures.     

融合弱层惩罚的卷积神经网络模型剪枝方法

深度卷积神经网络的存储和计算需求巨大,难以在一些资源受限的嵌入式设备上进行部署。为尽可能减少深度卷积神经网络模型在推理过程中的资源消耗,引入基于几何中值的卷积核重要性判断标准,提出一种融合弱层惩罚的结构化非均匀卷积神经网络模型剪枝方法。使用欧式距离计算各层卷积核间的信息距离,利用各卷积层信息距离的数据分布特征识别弱层,通过基于贡献度的归一化函数进行弱层惩罚,消除各层间的差异性。在全局层面评估卷积核重要性,利用全局掩码技术对所有卷积核实现动态剪枝。在CIFAR-10、CIFAR-100和SVHN数据集上的实验结果表明,与SFP、PFEC、FPGM和MIL剪枝方法相比,该方法剪枝得到的VGG16单分支、Resnet多分支、Mobilenet-v1轻量化网络模型在保证精度损失较小的情况下,有效地减少了模型参数量和浮点操作数。     

基于改进的PSO算法的神经网络相关性剪枝优化*

针对传统的神经网络训练算法收敛速度慢、易陷入局部最优的问题,提出了一种基于改进的分期变异微粒群优化算法（SMPSO）的神经网络相关性剪枝优化方法。SMPSO在初期使适应度过低的微粒发生变异,在后期使停滞代数过高的个体极值和全局极值发生变异,后将SMPSO用于优化神经网络相关性剪枝算法。实验结果表明,该方法与采用BP算法及标准PSO算法进行相关性剪枝相比,在训练收敛速度、剪枝效率及分类正确率三方面都有较大提高。