面向稀疏卷积神经网络的GPU性能优化方法

近些年来,深度卷积神经网络在多项任务中展现了惊人的能力,并已经被用在物体检测、自动驾驶和机器翻译等众多应用中.但这些模型往往参数规模庞大,并带来了沉重的计算负担.神经网络的模型剪枝技术能够识别并删除模型中对精度影响较小的参数,从而降低模型的参数数目和理论计算量,给模型的高效执行提供了机会.然而,剪枝后的稀疏模型却难以在GPU上实现高效执行,其性能甚至差于剪枝前的稠密模型,导致模型剪枝难以带来真正的执行性能收益.提出一种稀疏感知的代码生成方法,能够生成高效的稀疏卷积GPU程序.首先为卷积算子设计了算子模板,并结合GPU的特点对模板代码进行了多种优化.算子模板中的源代码经过编译和分析被转换为算子中间表示模板,设计了一种稀疏代码生成方法,能够结合剪枝后的稀疏参数,基于中间表示模板生成对应的稀疏卷积代码.同时,利用神经网络执行过程中的数据访问特点对数据的访问和放置进行了优化,有效提升了访存吞吐量.最后,稀疏参数的位置信息被隐式编码在生成的代码中,不需要额外的索引结构,降低了访存需求.在实验中证明了：相对于GPU上已有的稀疏神经网络执行方法,提出的稀疏感知的代码生成方法能够有效提升稀疏卷积神经网络的性能.     

Range estimation of construction costs using neural networks with bootstrap prediction intervals

Modeling of construction costs is a challenging task, as it requires representation of complex relations between factors and project costs with sparse and noisy data. In this paper, neural networks with bootstrap prediction intervals are presented for range estimation of construction costs. In the integrated approach, neural networks are used for modeling the mapping function between the factors and costs, and bootstrap method is used to quantify the level of variability included in the estimated costs. The integrated method is applied to range estimation of building projects. Two techniques; elimination of the input variables, and Bayesian regularization were implemented to improve generalization capabilities of the neural network models. The proposed modeling approach enables identification of parsimonious mapping function between the factors and cost and, provides a tool to quantify the prediction variability of the neural network models. Hence, the integrated approach presents a robust and pragmatic alternative for conceptual estimation of costs.     

A Hybrid Forward Algorithm for RBF Neural Network Construction

This paper proposes a novel hybrid forward algorithm (HFA) for the construction of radial basis function (RBF) neural networks with tunable nodes. The main objective is to efficiently and effectively produce a parsimonious RBF neural network that generalizes well. In this study, it is achieved through simultaneous network structure determination and parameter optimization on the continuous parameter space. This is a mixed integer hard problem and the proposed HFA tackles this problem using an integrated analytic framework, leading to significantly improved network performance and reduced memory usage for the network construction. The computational complexity analysis confirms the efficiency of the proposed algorithm, and the simulation results demonstrate its effectiveness     

Differential Evolution Training Algorithm for Feed-Forward Neural Networks

An evolutionary optimization method over continuous search spaces, differential evolution, has recently been successfully applied to real world and artificial optimization problems and proposed also for neural network training. However, differential evolution has not been comprehensively studied in the context of training neural network weights, i.e., how useful is differential evolution in finding the global optimum for expense of convergence speed. In this study, differential evolution has been analyzed as a candidate global optimization method for feed-forward neural networks. In comparison to gradient based methods, differential evolution seems not to provide any distinct advantage in terms of learning rate or solution quality. Differential evolution can rather be used in validation of reached optima and in the development of regularization terms and non-conventional transfer functions that do not necessarily provide gradient information. This revised version was published online in June 2006 with corrections to the Cover Date.     

应用遗传算法构建化学模式分类器

神经网络和统计分析所构建的分类器均为复杂算式,难以体现专业知识;而分类规则直接以属性值为条件,确定个体类别,易于专业分析。对于连续属性的样本数据,本文应用基于信息熵的Chi-merge方法将其离散化,并将提取最优规则转换为组合优化问题,进而采用遗传算法求解。为此,本文将规则提取演绎为种群进化,并设计了个体适应度函数。由此提取出最优的分类规则,经过修剪处理后,与判别准则一起构成模式分类器。本文将其应用于橄榄油产地判别,所建立的分类器简单明了,规则数少,性能良好,适用于化学模式分类。     

Fuzzy auto-associative neural networks for principal component extraction of noisy data.

In this paper, we propose a fuzzy auto-associative neural network for principal component extraction. The objective function is based on reconstructing the inputs from the corresponding outputs of the auto-associative neural network. Unlike the traditional approaches, the proposed criterion is a fuzzy mean squared error. We prove that the proposed objective function is an appropriate fuzzy formulation of auto-associative neural network for principal component extraction. Simulations are given to show the performances of the proposed neural networks in comparison with the existing method.     

Subgradient-based feedback neural networks for non-differentiable convex optimization problems

1 Introduction Optimization problems arise in a broad variety of scientific and engineering applica- tions. For many practice engineering applications problems, the real-time solutions of optimization problems are mostly required. One possible and very pr…     

Design optimization with back-propagation neural networks

A methodology with back-propagation neural network models is developed to explore the artificial neural nets (ANN) technology in the new application territory of design optimization. This design methodology could go beyond the Hopfield network model, Hopfield and Tank (1985), for combinatorial optimization problems In this approach, pattern classification with back-propagation network, the most demonstrated power of neural networks applications, is utilized to identify the boundaries of the feasible and the infeasible design regions. These boundaries enclose the multi-dimensional space within which designs satisfy all design criteria. A feedforward network is then incorporated to perform function approximation of the design objective function. This approximation is performed by training the feedforward network with objective functions evaluated at selected design sets in the feasible design regions. Additional optimum design sets in the classified feasible regions are calculated and included in the successive training sets to improve the function mapping. Iteration is continued until convergent criteria are satisfied. This paper demonstrates that the artificial neural nets technology provides a global perspective of the entire design space with good and near optimal solutions. ANN can indeed be a potential technology for design optimization.     

基于Rectified Adam和颜色不变性的对抗迁移攻击

深度神经网络在物体检测、图像分类、自然语言处理、语音识别等众多领域上得到广泛应用.然而,深度神经网络很容易受到对抗样本(即在原有样本上施加人眼无法察觉的微小扰动)的攻击,而且相同的扰动可以跨模型、甚至跨任务地欺骗多个分类器.对抗样本这种跨模型迁移特性,使得深度神经网络在实际生活的应用受到了很大限制.对抗样本对神经网络的威胁,激发了研究者对对抗攻击的研究兴趣.虽然研究者们已提出了不少对抗攻击方法,但是大多数这些方法(特别是黑盒攻击方法)的跨模型的攻击能力往往较差,尤其是对经过对抗训练、输入变换等的防御模型.为此,提出了一种提高对抗样本可迁移性的方法:RLI-CI-FGSM. RLI-CI-FGSM是一种基于迁移的攻击方法,在替代模型上,使用基于梯度的白盒攻击RLI-FGSM生成对抗样本,同时使用CIM扩充源模型,使RLI-FGSM能够同时攻击替代模型和扩充模型.具体而言,RLI-FGSM算法将Radam优化算法与迭代快速符号下降法相结合,并利用目标函数的二阶导信息来生成对抗样本,避免优化算法陷入较差的局部最优.基于深度神经网络具有一定的颜色变换不变性,CIM算法通过优化对颜色变换图像集合...     

Evolutionary multiobjective industrial design: the case of a racing car tire-suspension system

When dealing with multiobjective optimization (MO) of the tire-suspension system of a racing car, a large number of design variables and a large number of objectives have to be taken into account. Two different models have been used, both validated on data coming from an instrumented car, a differential equation-based physical model, and a neural network purely numerical model. Up to 23 objective functions have been defined, at least 14 of which are in strict conflict of each other. The equivalent scalar function based and the objective-as-constraint formulations are intentionally avoided due to their well-known limitations. A fuzzy definition of optima, being a generalization of Pareto optimality, is applied to the problem. The result of such an approach is that subsets of Pareto optimal solutions (on such a problem, a big portion of the entire search space) can be properly selected as a consequence of input from the designer. The obtained optimal solutions are compared with the reference vehicle and with the optima previously obtained with design of experiment techniques and different MO optimization strategies. The proposed strategy improves both the reference (actual) car and previously obtained optima (scalar preference function) in the majority of objectives with technically significant improvements. Moreover, the strategy offers an univoque criterion for the choice among tradeoff solutions in the 14-dimensional objective space. The problem is used as a test of a proposed optimal design strategy for industrial problems, integrating differential equation and neural networks modeling, design of experiments, MO, and fuzzy optimal-based decision making. Such a linked approach gives also a unified view of where to concentrate the computational effort.     

基于再生核神经网络的断层面模型重构

为了提高勘探精度及采油效率,需要更深入地了解地质构造,因此断层面模型的重构有着重要的意义,为此提出了基于再生核神经网络的断层面重构方法。再生核源于不同学科分支,目前已成为函数逼近的重要工具。将再生核与神经网络有机地结合起来,提出一种新型的神经网络———再生核神经网络,且将网络的训练归结为求解线性方程组问题,为了建立既具有足够精度又能表现系统行为的简单模型,考虑线性方程组的稀疏解是必要的,稀疏解就是具有大量零元素的近似解。虽然稀疏解整体误差较小,但可能在一些点上的误差较大,为此提出对稀疏解的误差修正方法。将再生核神经网络应用于大庆地区的断层面模型重构,实验结果表明,本文重构的断层面与传统方法重构的断层面相比,更符合大庆地区的地质情况。     

A constructive algorithm to synthesize arbitrarily connected feedforward neural networks

In this work we present a constructive algorithm capable of producing arbitrarily connected feedforward neural network architectures for classification problems. Architecture and synaptic weights of the neural network should be defined by the learning procedure. The main purpose is to obtain a parsimonious neural network, in the form of a hybrid and dedicate linear/nonlinear classification model, which can guide to high levels of performance in terms of generalization. Though not being a global optimization algorithm, nor a population-based metaheuristics, the constructive approach has mechanisms to avoid premature convergence, by mixing growing and pruning processes, and also by implementing a relaxation strategy for the learning error. The synaptic weights of the neural networks produced by the constructive mechanism are adjusted by a quasi-Newton method, and the decision to grow or prune the current network is based on a mutual information criterion. A set of benchmark experiments, including artificial and real datasets, indicates that the new proposal presents a favorable performance when compared with alternative approaches in the literature, such as traditional MLP, mixture of heterogeneous experts, cascade correlation networks and an evolutionary programming system, in terms of both classification accuracy and parsimony of the obtained classifier.     

多媒体信号处理的数学理论前沿进展

深度学习模型广泛应用于多媒体信号处理领域,通过引入非线性能够极大地提升性能,但是其黑箱结构无法解析地给出最优点和优化条件。因此如何利用传统信号处理理论,基于变换/基映射模型逼近深度学习模型,解析优化问题,成为当前研究的前沿问题。本文从信号处理的基础理论出发,分析了当前针对高维非线性非规则结构方法的数学模型和理论边界,主要包括：结构化稀疏表示模型、基于框架理论的深度网络模型、多层卷积稀疏编码模型以及图信号处理理论。详细描述了基于组稀疏性和层次化稀疏性的表示模型和优化方法,分析基于半离散框架和卷积稀疏编码构建深度/多层网络模型,进一步在非欧氏空间上扩展形成图信号处理模型,并对国内外关于记忆网络的研究进展进行了比较。最后,展望了多媒体信号处理的理论模型发展,认为图信号处理通过解析谱图模型的数学性质,解释其中的关联性,为建立广义的大规模非规则多媒体信号处理模型提供理论基础,是未来研究的重要领域之一。     

改进的粒子群算法优化神经网络及应用

针对神经网络权值选取不精确的问题,提出改进的粒子群优化算法结合BP神经网络动态选取权值的方法。在改进的粒子群优化算法中,采用动态惯性权重,并且认知参数与社会参数相互制约。同时,改进的粒子群优化算法结合差分进化算法使粒子拥有变异与交叉操作,保持粒子的多样性。基于改进的粒子群优化算法与BP神经网络,构建IPSONN神经网络模型并运用于酒类品质的预测。实验分别从训练精度、正确率及粒子多样性三方面验证了IPSONN模型的有效性。     

On Discriminative Bayesian Network Classifiers and Logistic Regression

Discriminative learning of the parameters in the naive Bayes model is known to be equivalent to a logistic regression problem. Here we show that the same fact holds for much more general Bayesian network models, as long as the corresponding network structure satisfies a certain graph-theoretic property. The property holds for naive Bayes but also for more complex structures such as tree-augmented naive Bayes (TAN) as well as for mixed diagnostic-discriminative structures. Our results imply that for networks satisfying our property, the conditional likelihood cannot have local maxima so that the global maximum can be found by simple local optimization methods. We also show that if this property does not hold, then in general the conditional likelihood can have local, non-global maxima. We illustrate our theoretical results by empirical experiments with local optimization in a conditional naive Bayes model. Furthermore, we provide a heuristic strategy for pruning the number of parameters and relevant features in such models. For many data sets, we obtain good results with heavily pruned submodels containing many fewer parameters than the original naive Bayes model.Editors: Pedro Larrañaga, Jose A. Lozano, Jose M. Peña and Iñaki Inza     

Neural and Wavelet Network Models for Financial Distress Classification

This work analyzes the use of linear discriminant models, multi-layer perceptron neural networks and wavelet networks for corporate financial distress prediction. Although simple and easy to interpret, linear models require statistical assumptions that may be unrealistic. Neural networks are able to discriminate patterns that are not linearly separable, but the large number of parameters involved in a neural model often causes generalization problems. Wavelet networks are classification models that implement nonlinear discriminant surfaces as the superposition of dilated and translated versions of a single “mother wavelet” function. In this paper, an algorithm is proposed to select dilation and translation parameters that yield a wavelet network classifier with good parsimony characteristics. The models are compared in a case study involving failed and continuing British firms in the period 1997–2000. Problems associated with over-parameterized neural networks are illustrated and the Optimal Brain Damage pruning technique is employed to obtain a parsimonious neural model. The results, supported by a re-sampling study, show that both neural and wavelet networks may be a valid alternative to classical linear discriminant models.     

Data reconciliation for simulated flotation process

This paper introduces a novel neural network-based technique called system balance-related autoassociative neural networks (SBANN) for steady state data reconciliation. This neural network has the same architecture as traditional feedforward neural networks but the main difference lies in the minimization of an objective function that includes process material and/or energy imbalance terms in addition to the traditional least-squares prediction term. Accordingly, this neural network with the system balance-related objective criterion is able to perform the two basic functions necessary for proper steady state data reconciliation: data smoothing to reduce the data variance and data correction to satisfy material and/or energy balance constraints. This novel technique is illustrated for data reconciliation of a simulated flotation circuit that is widely used in mineral processing.     

Using the Correlation Criterion to Position and Shape RBF Units for Incremental Modelling

A novel technique is proposed for the incremental construction of sparse radial basis function (RBF) networks. The correlation between an RBF regressor and the training data is used as the criterion to position and shape the RBF node, and it is shown that this is equivalent to incrementally minimise the modelling mean square error. A guided random search optimisation method, called the repeated weighted boosting search, is adopted to append RBF nodes one by one in an incremental regression modelling procedure. The experimental results obtained using the proposed method demonstrate that it provides a viable alternative to the existing state-of-the-art modelling techniques for constructing parsimonious RBF models that generalise well.     

采用智能评判的网络防御效果仿真模型研究

通过将网络防御分为3个层次,并结合免疫学的相关理论,提出了抗体的剪枝防御评判策略,指出在网元的抗体浓度低于零时,应将其剪枝隔离,不再将其纳入评判网络防御效果的防御体系计算范围内。在此基础上结合神经网络理论,建立了基于免疫和神经网络综合评判的智能评判网络防御效果仿真模型,根据该模型可以对网元的网络防御效果进行定性化计算研究。