CB3: An Adaptive Error Function for Backpropagation Training

Effective backpropagation training of multi-layer perceptrons depends on the incorporation of an appropriate error or objective function. Classification-Based (CB) error functions are heuristic approaches that attempt to guide the network directly to correct pattern classification rather than using common error minimization heuristics, such as Sum-Squared Error (SSE) and Cross-Entropy (CE), which do not explicitly minimize classification error. This work presents CB3, a novel CB approach that learns the error function to be used while training. This is accomplished by learning pattern confidence margins during training, which are used to dynamically set output target values for each training pattern. On 11 applications, CB3 significantly outperforms previous CB error functions, and also reduces average test error over conventional error metrics using 0–1 targets without weight decay by 1.8%, and by 1.3% over metrics with weight decay. The CB3 also exhibits lower model variance and tighter mean confidence interval.     

Optimized approximation algorithm in neural networks without overfitting.

In this paper, an optimized approximation algorithm (OAA) is proposed to address the overfitting problem in function approximation using neural networks (NNs). The optimized approximation algorithm avoids overfitting by means of a novel and effective stopping criterion based on the estimation of the signal-to-noise-ratio figure (SNRF). Using SNRF, which checks the goodness-of-fit in the approximation, overfitting can be automatically detected from the training error only without use of a separate validation set. The algorithm has been applied to problems of optimizing the number of hidden neurons in a multilayer perceptron (MLP) and optimizing the number of learning epochs in MLP's backpropagation training using both synthetic and benchmark data sets. The OAA algorithm can also be utilized in the optimization of other parameters of NNs. In addition, it can be applied to the problem of function approximation using any kind of basis functions, or to the problem of learning model selection when overfitting needs to be considered.     

基于改进的深度残差网络的表情识别研究

提出了一种基于改进的深度残差网络（residual network,ResNet）的表情识别算法。采用小卷积核和深网络结构,利用残差模块学习残差映射解决了随着网络深度的增加网络精度下降问题,通过迁移学习方法克服了因数据量不足导致训练不充分的缺点;网络架构使用了线性支持向量机（SVM）进行分类。实验中首先利用ImageNet数据库进行网络参数预训练,使网络具有良好的提取特征能力,根据迁移学习方法,利用FER-2013数据库以及扩充后的CK+数据库进行参数微调和训练。该算法克服了浅层网络需要依靠手工特征,深层网络难以训练等问题,在CK+数据库以及GENKI-4K数据库上分别取得了91.333%和95.775%识别率。SVM在CK+数据库的分类效果较softmax提高了1%左右。     

面向GF-2遥感影像的U-Net城市绿地分类

目的 高分2号卫星（GF-2）是首颗民用高空间分辨率光学卫星，具有亚米级高空间分辨率与宽覆盖结合的显著特点，为城市绿地信息提取等多领域提供了重要的数据支撑。本文利用GF-2卫星多光谱遥感影像，将一种改进的U-Net卷积神经网络首次应用于城市绿地分类，提出一种面向高分遥感影像的城市绿地自动分类提取技术。方法 先针对小样本训练集容易产生的过拟合问题对U-Net网络进行改进，添加批标准化（batch normalization，BN）和dropout层获得U-Net+模型；再采用随机裁剪和随机数据增强的方式扩充数据集，使得在充分利用影像信息的同时保证样本随机性，增强模型稳定性。结果 将U-Net+模型与最大似然法（maximum likelihood estimation，MLE）、神经网络（neural networks，NNs）和支持向量机（support vector machine，SVM）3种传统分类方法以及U-Net、SegNet和DeepLabv3+这3种深度学习语义分割模型进行分类结果精度对比。改进后的U-Net+模型能有效防止过拟合，模型总体分类精度比改进前提高了1.06%。基于改进的U-Net+模型的城市绿地总体分类精度为92.73%，平均F₁分数为91.85%。各分类方法按照总体分类精度从大到小依次为U-Net+（92.73%）、U-Net （91.67%）、SegNet （88.98%）、DeepLabv3+（87.41%）、SVM （81.32%）、NNs （79.92%）和MLE （77.21%）。深度学习城市绿地分类方法能充分挖掘数据的光谱、纹理及潜在特征信息，有效降低分类过程中产生的"椒盐噪声"，具有较好的样本容错能力，比传统遥感分类方法更适用于城市绿地信息提取。结论 改进后的U-Net+卷积神经网络模型能够有效提升高分遥感影像城市绿地自动分类提取精度，为城市绿地分类提供了一种新的智能解译方法。     

代表特征网络的小样本学习方法

目的 小样本学习任务旨在仅提供少量有标签样本的情况下完成对测试样本的正确分类。基于度量学习的小样本学习方法通过将样本映射到嵌入空间,计算距离得到相似性度量以预测类别,但未能从类内多个支持向量中归纳出具有代表性的特征以表征类概念,限制了分类准确率的进一步提高。针对该问题,本文提出代表特征网络,分类效果提升显著。方法 代表特征网络通过类代表特征的度量学习策略,利用类中支持向量集学习得到的代表特征有效地表达类概念,实现对测试样本的正确分类。具体地说,代表特征网络包含两个模块,首先通过嵌入模块提取抽象层次高的嵌入向量,然后堆叠嵌入向量经过代表特征模块得到各个类代表特征。随后通过计算测试样本嵌入向量与各类代表特征的距离以预测类别,最后使用提出的混合损失函数计算损失以拉大嵌入空间中相互类别间距减少相似类别错分情况。结果 经过广泛实验,在Omniglot、miniImageNet和Cifar100数据集上都验证了本文模型不仅可以获得目前已知最好的分类准确率,而且能够保持较高的训练效率。结论 代表特征网络可以从类中多个支持向量有效地归纳出代表特征用于对测试样本的分类,对比直接使用支持向量进行分类具有更好的鲁棒性,进一步提高了小样本条件下的分类准确率。     

基于深度学习的病历质量控制系统设计

医疗领域患者的主诉信息是医疗文本分类工作的关键,能为智慧医疗和信息文本归类提供有力的支持。近几年来随着深度学习的发展应用,基于传统深度学习技术的全流程病历内涵质量控制模型层出不穷,但传统模型存在很多缺点和局限性,诸如训练速度慢、精度损失、过拟合和无法处理大规模数据的问题,因此,引入改进的深度学习算法。指南指导下基于深度学习的全流程病历内涵质量控制体系实验结果为,将词向量设置成160时双向循环神经网络(Bidirectional Recurrent Neural Network,BiGRU-SA)模型效果最优,准确率为84.9% 。BiGRU-SA MODEL,精准度受向量维度的影响并不大。而改进的文本分类式前馈神经网络(Transformation-extraction-convolutional CNN,TextCNN)模型,精准度在其进行第3次和第四次迭代更新时,发生指数级增长,并在第3次迭代时,精度达到理想值,为8.3×10_-1^。随着迭代次数的增加,模型准确率呈现先增大后减小的趋势,在进行第6次迭代时模型效果最优,准确率为84.9% 。优化后的全流程病历内涵质量控制模型在变动率指标下的面积的值、准确率、F1、召回率四项指标值都有了一定的提升,以上结果能更好地解决过拟合和特征信息丢失的问题,并且实现全流程病历内涵质量的控制。     

A Novel Solution of Using Deep Learning for White Blood Cells Classification: Enhanced Loss Function with Regularization and Weighted Loss (ELFRWL)

Deep learning has been successfully applied in classification of white blood cells (WBCs), however, accuracy and processing time are found to be less than optimal hindering it from getting its full potential. This is due to imbalanced dataset, intra-class compactness, inter-class separability and overfitting problems. The main research idea is to enhance the classification and prediction accuracy of blood images while lowering processing time through the use of deep convolutional neural network (DCNN) architecture by using the modified loss function. The proposed system consists of a deep neural convolution network (DCNN) that will improve the classification accuracy by using modified loss function along with regularization. Firstly, images are pre-processed and fed through DCNN that contains different layers with different activation function for the feature extraction and classification. Along with modified loss function with regularization, weight function aids in the classification of WBCs by considering weights of samples belonging to each class for compensating the error arising due to imbalanced dataset. The processing time will be counted by each image to check the time enhancement. The classification accuracy and processing time are achieved using the dataset-master. Our proposed solution obtains better classification performance in the given dataset comparing with other previous methods. The proposed system enhanced the classification accuracy of 98.92% from 96.1% and a decrease in processing time from 0.354 to 0.216 s. Less time will be required by our proposed solution for achieving the model convergence with 9 epochs against the current convergence time of 13.5 epochs on average, epoch is the formation white blood cells (WBCs) and the development of granular cells. The proposed solution modified loss function to solve the adverse effect caused due to imbalance dataset by considering weight and use regularization technique for overfitting problem.

     

改进的弹性网模型在深度神经网络中的应用

由于具有较高的模型复杂度，深层神经网络容易产生过拟合问题，为了减少该问题对网络性能的不利影响，提出一种基于改进的弹性网模型的深度学习优化方法。首先，考虑到变量之间的相关性，对弹性网模型中的L1范数的不同变量进行自适应加权，从而得到L2范数与自适应加权的L1范数的线性组合。其次，将改进的弹性网络模型与深度学习的优化模型相结合，给出在这种新正则项约束下求解神经网络参数的过程。然后，推导出改进的弹性网模型在神经网络优化中具有群组选择能力和Oracle性质，进而从理论上保证该模型是一种更加鲁棒的正则化方法。最后，在多个回归问题和分类问题的实验中，相对于L1、L2和弹性网正则项，该方法的回归测试误差可分别平均降低87.09、88.54和47.02，分类测试准确度可分别平均提高3.98、2.92和3.58个百分点。由此，在理论和实验两方面验证了改进的弹性网模型可以有效地增强深层神经网络的泛化能力，提升优化算法的性能，解决深度学习的过拟合问题。     

A novel objective function for improved phoneme recognition usingtime-delay neural networks

Single-speaker and multispeaker recognition results are presented for the voice-stop consonants /b,d,g/ using time-delay neural networks (TDNNs) with a number of enhancements, including a new objective function for training these networks. The new objective function, called the classification figure of merit (CFM), differs markedly from the traditional mean-squared-error (MSE) objective function and the related cross entropy (CE) objective function. Where the MSE and CE objective functions seek to minimize the difference between each output node and its ideal activation, the CFM function seeks to maximize the difference between the output activation of the node representing incorrect classifications. A simple arbitration mechanism is used with all three objective functions to achieve a median 30% reduction in the number of misclassifications when compared to TDNNs trained with the traditional MSE back-propagation objective function alone.     

Boosted Bayesian network classifiers

The use of Bayesian networks for classification problems has received a significant amount of recent attention. Although computationally efficient, the standard maximum likelihood learning method tends to be suboptimal due to the mismatch between its optimization criteria (data likelihood) and the actual goal of classification (label prediction accuracy). Recent approaches to optimizing classification performance during parameter or structure learning show promise, but lack the favorable computational properties of maximum likelihood learning. In this paper we present boosted Bayesian network classifiers, a framework to combine discriminative data-weighting with generative training of intermediate models. We show that boosted Bayesian network classifiers encompass the basic generative models in isolation, but improve their classification performance when the model structure is suboptimal. We also demonstrate that structure learning is beneficial in the construction of boosted Bayesian network classifiers. On a large suite of benchmark data-sets, this approach outperforms generative graphical models such as naive Bayes and TAN in classification accuracy. Boosted Bayesian network classifiers have comparable or better performance in comparison to other discriminatively trained graphical models including ELR and BNC. Furthermore, boosted Bayesian networks require significantly less training time than the ELR and BNC algorithms.     

A study of overfitting in optimization of a manufacturing quality control procedure

Quality control of the commutator manufacturing process can be automated by means of a machine learning model that can predict the quality of commutators as they are being manufactured. Such a model can be constructed by combining machine vision, machine learning and evolutionary optimization techniques. In this procedure, optimization is used to minimize the model error, which is estimated using single cross-validation. This work exposes the overfitting that emerges in such optimization. Overfitting is shown for three machine learning methods with different sensitivity to it (trees, additionally pruned trees and random forests) and assessed in two ways (repeated cross-validation and validation on a set of unseen instances). Results on two distinct quality control problems show that optimization amplifies overfitting, i.e., the single cross-validation error estimate for the optimized models is overly optimistic. Nevertheless, minimization of the error estimate by single cross-validation in general results in minimization of the other error estimates as well, showing that optimization is indeed beneficial in this context.     

自适应多视角学习及其在图像分类中的应用

针对现有多视角学习算法在构建近邻图时缺乏数据自适应性问题,提出一种自适应多视角学习(AMVL)算法。该算法首先利用L1范数具有自动数据样本选择的特性,对不同视角分别构建有向的L1图;然后根据得到的L1图,最小化各个视角下的低维重建误差;最后对不同视角间进行多视角全局坐标对齐,得到自适应多视角学习算法的目标函数。此外,还提出一种迭代优化求解方法来对所提目标函数进行优化求解。将该算法应用到图像分类问题,在Corel5K和NUS-WIDE-OBJECT两个公共图像数据集上与现有算法进行对比。实验结果表明：所提方法在这两个数据集上可以分别提高最高5%和2%的分类准确率;优化求解算法可以保证在100次迭代内收敛;算法所得到的近邻数目具有数据自适应性。     

高分辨卫星图像卷积神经网络分类模型

目的 卫星图像往往目标、背景复杂而且带有噪声,因此使用人工选取的特征进行卫星图像的分类就变得十分困难。提出一种新的使用卷积神经网络进行卫星图像分类的方案。使用卷积神经网络可以提取卫星图像的高层特征,进而提高卫星图像分类的识别率。方法 首先,提出一个包含六类图像的新的卫星图像数据集来解决卷积神经网络的有标签训练样本不足的问题。其次,使用了一种直接训练卷积神经网络模型和3种预训练卷积神经网络模型来进行卫星图像分类。直接训练模型直接在文章提出的数据集上进行训练,预训练模型先在ILSVRC（the ImageNet large scale visual recognition challenge）-2012数据集上进行预训练,然后在提出的卫星图像数据集上进行微调训练。完成微调的模型用于卫星图像分类。结果 提出的微调预训练卷积神经网络深层模型具有最高的分类正确率。在提出的数据集上,深层卷积神经网络模型达到了99.50%的识别率。在数据集UC Merced Land Use上,深层卷积神经网络模型达到了96.44%的识别率。结论 本文提出的数据集具有一般性和代表性,使用的深层卷积神经网络模型具有很强的特征提取能力和分类能力,且是一种端到端的分类模型,不需要堆叠其他模型或分类器。在高分辨卫星图像的分类上,本文模型和对比模型相比取得了更有说服力的结果。     

跨模态多标签生物医学图像分类建模识别

目的 生物医学文献中的图像经常是包含多种模式的复合图像,自动标注其类别,将有助于提高图像检索的性能,辅助医学研究或教学。方法 融合图像内容和说明文本两种模态的信息,分别搭建基于深度卷积神经网络的多标签分类模型。视觉分类模型借用自然图像和单标签的生物医学简单图像,实现异质迁移学习和同质迁移学习,捕获通用领域的一般特征和生物医学领域的专有特征,而文本分类模型利用生物医学简单图像的说明文本,实现同质迁移学习。然后,采用分段式融合策略,结合两种模态模型输出的结果,识别多标签医学图像的相关模式。结果 本文提出的跨模态多标签分类算法,在ImageCLEF2016生物医学图像多标签分类任务数据集上展开实验。基于图像内容的混合迁移学习方法,比仅采用异质迁移学习的方法,具有更低的汉明损失和更高的宏平均F1值。文本分类模型引入同质迁移学习后,能够明显提高标签的分类性能。最后,融合两种模态的多标签分类模型,获得与评测任务最佳成绩相近的汉明损失,而宏平均F1值从0.320上升到0.488,提高了约52.5%。结论 实验结果表明,跨模态生物医学图像多标签分类算法,融合图像内容和说明文本,引入同质和异质数据进行迁移学习,缓解生物医学图像领域标注数据规模小且标签分布不均衡的问题,能够更有效地识别复合医学图像中的模式信息,进而提高图像检索性能。     

分类错误指导的分层B-CNN模型用于细粒度分类

目的 细粒度分类近年来受到了越来越多研究者的广泛关注,其难点是分类目标间的差异非常小。为此提出一种分类错误指导的分层双线性卷积神经网络模型。方法 该模型的核心思想是将双线性卷积神经网络算法（B-CNN）容易分错、混淆的类再分别进行重新训练和分类。首先,为得到易错类,提出分类错误指导的聚类算法。该算法基于受限拉普拉斯秩（CLR）聚类模型,其核心“关联矩阵”由“分类错误矩阵”构造。其次,以聚类结果为基础,构建了新的分层B-CNN模型。结果 用分类错误指导的分层B-CNN模型在CUB-200-2011、 FGVC-Aircraft-2013b和Stanford-cars 3个标准数据集上进行了实验,相比于单层的B-CNN模型,分类准确率分别由84.35%,83.56%,89.45%提高到了84.67%,84.11%,89.78%,验证了本文算法的有效性。结论 本文提出了用分类错误矩阵指导聚类从而进行重分类的方法,相对于基于特征相似度而构造的关联矩阵,分类错误矩阵直接针对分类问题,可以有效提高易混淆类的分类准确率。本文方法针对比较相近的目标,尤其是有非常相近的目标的情况,通过将容易分错、混淆的目标分组并进行再训练和重分类,使得分类效果更好,适用于细粒度分类问题。     

持续学习改进的人脸表情识别

目的 大量标注数据和深度学习方法极大地提升了图像识别性能。然而,表情识别的标注数据缺乏,训练出的深度模型极易过拟合,研究表明使用人脸识别的预训练网络可以缓解这一问题。但是预训练的人脸网络可能会保留大量身份信息,不利于表情识别。本文探究如何有效利用人脸识别的预训练网络来提升表情识别的性能。方法 本文引入持续学习的思想,利用人脸识别和表情识别之间的联系来指导表情识别。方法指出网络中对人脸识别整体损失函数的下降贡献最大的参数与捕获人脸公共特征相关,对表情识别来说为重要参数,能够帮助感知面部特征。该方法由两个阶段组成：首先训练一个人脸识别网络,同时计算并记录网络中每个参数的重要性;然后利用预训练的模型进行表情识别的训练,同时通过限制重要参数的变化来保留模型对于面部特征的强大感知能力,另外非重要参数能够以较大的幅度变化,从而学习更多表情特有的信息。这种方法称之为参数重要性正则。结果 该方法在RAF-DB（real-world affective faces database）,CK+（the extended Cohn-Kanade database）和Oulu-CASIA这3个数据集上进行了实验评估。在主流数据集RAF-DB上,该方法达到了88.04%的精度,相比于直接用预训练网络微调的方法提升了1.83%。其他数据集的实验结果也表明了该方法的有效性。结论 提出的参数重要性正则,通过利用人脸识别和表情识别之间的联系,充分发挥人脸识别预训练模型的作用,使得表情识别模型更加鲁棒。     

Radial Basis Function network learning using localized generalization error bound

Training a classifier with good generalization capability is a major issue for pattern classification problems. A novel training objective function for Radial Basis Function (RBF) network using a localized generalization error model (L-GEM) is proposed in this paper. The localized generalization error model provides a generalization error bound for unseen samples located within a neighborhood that contains all training samples. The assumption of the same width for all dimensions of a hidden neuron in L-GEM is relaxed in this work. The parameters of RBF network are selected via minimization of the proposed objective function to minimize its localized generalization error bound. The characteristics of the proposed objective function are compared with those for regularization methods. For weight selection, RBF networks trained by minimizing the proposed objective function consistently outperform RBF networks trained by minimizing the training error, Tikhonov Regularization, Weight Decay or Locality Regularization. The proposed objective function is also applied to select center, width and weight in RBF network simultaneously. RBF networks trained by minimizing the proposed objective function yield better testing accuracies when compared to those that minimizes training error only.     

融合判别式深度特征学习的图像识别算法

目的 卷积神经网络在图像识别算法中得到了广泛应用。针对传统卷积神经网络学习到的特征缺少更有效的鉴别能力而导致图像识别性能不佳等问题,提出一种融合线性判别式思想的损失函数LDloss（linear discriminant loss）并用于图像识别中的深度特征提取,以提高特征的鉴别能力,进而改善图像识别性能。方法 首先利用卷积神经网络搭建特征提取所需的深度网络,然后在考虑样本分类误差最小化的基础上,对于图像多分类问题,引入LDA（linear discriminant analysis）思想构建新的损失函数参与卷积神经网络的训练,来最小化类内特征距离和最大化类间特征距离,以提高特征的鉴别能力,从而进一步提高图像识别性能,分析表明,本文算法可以获得更有助于样本分类的特征。其中,学习过程中采用均值分批迭代更新的策略实现样本均值平稳更新。结果 该算法在MNIST数据集和CK+数据库上分别取得了99.53%和94.73%的平均识别率,与现有算法相比较有一定的提升。同时,与传统的损失函数Softmax loss和Hinge loss对比,采用LDloss的深度网络在MNIST数据集上分别提升了0.2%和0.3%,在CK+数据库上分别提升了9.21%和24.28%。结论 本文提出一种新的融合判别式深度特征学习算法,该算法能有效地提高深度网络的可鉴别能力,从而提高图像识别精度,并且在测试阶段,与Softmax loss相比也不需要额外的计算量。     

Discrete recurrent neural networks for grammatical inference

Describes a novel neural architecture for learning deterministic context-free grammars, or equivalently, deterministic pushdown automata. The unique feature of the proposed network is that it forms stable state representations during learning-previous work has shown that conventional analog recurrent networks can be inherently unstable in that they cannot retain their state memory for long input strings. The authors have previously introduced the discrete recurrent network architecture for learning finite-state automata. Here they extend this model to include a discrete external stack with discrete symbols. A composite error function is described to handle the different situations encountered in learning. The pseudo-gradient learning method (introduced in previous work) is in turn extended for the minimization of these error functions. Empirical trials validating the effectiveness of the pseudo-gradient learning method are presented, for networks both with and without an external stack. Experimental results show that the new networks are successful in learning some simple pushdown automata, though overfitting and non-convergent learning can also occur. Once learned, the internal representation of the network is provably stable; i.e., it classifies unseen strings of arbitrary length with 100% accuracy.     

应用级联多分类器的高光谱图像分类

目的 高光谱分类任务中,由于波段数量较多,图像中存在包含噪声以及各类地物样本分布不均匀等问题,导致分类精度与训练效率不能平衡,在小样本上分类精度低。因此,提出一种基于级联多分类器的高光谱图像分类方法。方法 首先采用主成分分析方法将高度相关的高维特征合成无关的低维特征,以加快Gabor滤波器提取纹理特征的速度;然后使用Gabor滤波器提取图像在各个尺寸、方向上的纹理信息,每一个滤波器会生成一张特征图,在特征图中以待分类样本为中心取一个d×d的邻域,计算该邻域内数据的均值和方差来作为待分类样本的空间信息,再将空间信息和光谱信息融合,以降低光线与噪声的影响;最后将谱—空联合特征输入级联多分类器中,得到预测样本关于类别的概率分布的平均值。结果 实验采用Indian Pines、Pavia University和Salinas 3个数据集,与经典算法如支持向量机和卷积神经网络进行比较,并利用总体分类精度、平均分类精度和Kappa系数作为评价标准进行分析。本文方法总体分类精度在3个数据集上分别达到97.24%、99.57%和99.46%,相对于基于径向基神经网络（RBF）核函数的支持向量机方法提高了13.2%、4.8%和5.68%,相对于加入谱—空联合特征的RBF-SVM （radial basis function-support vector machine）方法提高了2.18%、0.36%和0.83%,相对于卷积神经网络方法提高了3.27%、3.2%和0.3%;Kappa系数分别是0.968 6、0.994 3和0.995 6,亦有提高。结论 实验结果表明,本文方法应用于高光谱图像分类具有较优的分类效果,训练效率较高,无需依赖GPU,而且在小样本上也具有较高的分类精度。