Task decomposition and module combination based on class relations: a modular neural network for pattern classification.

We propose a method for decomposing pattern classification problems based on the class relations among training data. By using this method, we can divide a K-class classification problem into a series of ((2)(K)) two-class problems. These two-class problems are to discriminate class C(i) from class C(j) for i=1, ..., K and j=i+1, while the existence of the training data belonging to the other K-2 classes is ignored. If the two-class problem of discriminating class C(i) from class C(j) is still hard to be learned, we can further break down it into a set of two-class subproblems as small as we expect. Since each of the two-class problems can be treated as a completely separate classification problem with the proposed learning framework, all of the two-class problems can be learned in parallel. We also propose two module combination principles which give practical guidelines in integrating individual trained network modules. After learning of each of the two-class problems with a network module, we can easily integrate all of the trained modules into a min-max modular (M(3)) network according to the module combination principles and obtain a solution to the original problem. Consequently, a large-scale and complex K-class classification problem can be solved effortlessly and efficiently by learning a series of smaller and simpler two-class problems in parallel.     

A parallel scheduling algorithm for reinforcement learning in large state space

The main challenge in the area of reinforcement learning is scaling up to larger and more complex problems. Aiming at the scaling problem of reinforcement learning, a scalable reinforcement learning method, DCS-SRL, is proposed on the basis of divide-and-conquer strategy, and its convergence is proved. In this method, the learning problem in large state space or continuous state space is decomposed into multiple smaller subproblems. Given a specific learning algorithm, each subproblem can be solved independently with limited available resources. In the end, component solutions can be recombined to obtain the desired result. To address the question of prioritizing subproblems in the scheduler, a weighted priority scheduling algorithm is proposed. This scheduling algorithm ensures that computation is focused on regions of the problem space which are expected to be maximally productive. To expedite the learning process, a new parallel method, called DCS-SPRL, is derived from combining DCS-SRL with a parallel scheduling architecture. In the DCS-SPRL method, the subproblems will be distributed among processors that have the capacity to work in parallel. The experimental results show that learning based on DCS-SPRL has fast convergence speed and good scalability.     

基于机器学习的平行语料库应用效果评价方法

针对传统翻译系统在时态翻译中不准确的问题,结合当前的机器学习算法,提出一种基于DBN的平行语料库时态翻译方法。为实现该方法,首先对时态标注模型和DBN基本理论进行介绍,并提出汉英语句时态翻译的思路;而在进行DBN平行语料库特征提取的过程中,采用自动时态标注算法对时态进行标注,并对得到的数据进行时态树编码;然后以编码数据作为输入,运用DBN网络对时态进行训练预测,得到中文语句中可能的时态;最后通过Transformer翻译器对语句进行翻译,得到对应的翻译时态句子;最后结合LDC等平行语料库中的翻译句子,采用上述方法进行翻译,结果表明提出的时态翻译方法无论是在准确率,还是在召回率等方面,都有很大的优势,说明提出的机器学习算法在平行语料库的翻译中更具有准确性。     

Neural network modeling of vector multivariable functions in ill-posed approximation problems

A neural network solution of the ill-posed inverse approximation problem of a multivariable vector function based on of a committee of multilayer perceptrons is proposed. A nonlinear adaptive decision-making rule by the committee is developed that improves the accuracy compared with other neural network solutions of the inverse problem. Using a model example, the accuracy characteristics of the method are shown. An applied engineering problem is considered and the results of its solution by the proposed method are presented.     

Statistical active learning in multilayer perceptrons

Proposes methods for generating input locations actively in gathering training data, aiming at solving problems unique to muitilayer perceptrons. One of the problems is that optimum input locations, which are calculated deterministically, sometimes distribute densely around the same point and cause local minima in backpropagation training. Two probabilistic active learning methods, which utilize the statistical variance of locations, are proposed to solve this problem. One is parametric active learning and the other is multipoint-search active learning. Another serious problem in applying active learning to multilayer perceptrons is that a Fisher information matrix can be singular, while many methods, including the proposed ones, assume its regularity. A technique of pruning redundant hidden units is proposed to keep the Fisher information matrix regular. Combined with this technique, active learning can be applied stably to multilayer perceptrons. The effectiveness of the proposed methods is demonstrated through computer simulations on simple artificial problems and a real-world problem of color conversion.     

基于无指导机器学习的全文词义自动标注方法

为实现汉语全文词义自动标注，本文采用了一种新的基于无指导机器学习策略的词义标注方法。实验中建立了四个词义排歧模型，并对其测试结果进行了比较.其中实验效果最优的词义排歧模型融合了两种无指导的机器学习策略，并借助依存文法分析手段对上下文特征词进行选择.最终确定的词义标注方法可以使用大规模语料对模型进行训练，较好的解决了数据稀疏问题，并且该方法具有标注正确率高、扩展性能好等优点，适合大规模文本的词义标注工作.     

Parallel growing and training of neural networks using outputparallelism

In order to find an appropriate architecture for a large-scale real-world application automatically and efficiently, a natural method is to divide the original problem into a set of subproblems. In this paper, we propose a simple neural-network task decomposition method based on output parallelism. By using this method, a problem can be divided flexibly into several subproblems as chosen, each of which is composed of the whole input vector and a fraction of the output vector. Each module (for one subproblem) is responsible for producing a fraction of the output vector of the original problem. The hidden structure for the original problem's output units are decoupled. These modules can be grown and trained in parallel on parallel processing elements. Incorporated with a constructive learning algorithm, our method does not require excessive computation and any prior knowledge concerning decomposition. The feasibility of output parallelism is analyzed and proved. Some benchmarks are implemented to test the validity of this method. Their results show that this method can reduce computational time, increase learning speed and improve generalization accuracy for both classification and regression problems.     

一种新的基于粒群优化的BP网络学习算法

标准BP学习算法是多层感知器的一种训练学习算法，是基于无约束极值问题的梯度法而设计的。针对标准算法存在的收敛速度慢、目标函数易陷入局部极小等缺点，该文提出了一种基于粒群优化的全新学习算法——粒群学习算法。该算法采用并行全局寻优策略，使网络以更快的速度收敛至全局最优解，且更易于编程实现。仿真实例证明，该算法是一种简洁高效的BP神经网络学习算法，有着极为广泛的应用前景。     

An accelerated learning algorithm for multilayer perceptrons: optimization layer by layer.

Multilayer perceptrons are successfully used in an increasing number of nonlinear signal processing applications. The backpropagation learning algorithm, or variations hereof, is the standard method applied to the nonlinear optimization problem of adjusting the weights in the network in order to minimize a given cost function. However, backpropagation as a steepest descent approach is too slow for many applications. In this paper a new learning procedure is presented which is based on a linearization of the nonlinear processing elements and the optimization of the multilayer perceptron layer by layer. In order to limit the introduced linearization error a penalty term is added to the cost function. The new learning algorithm is applied to the problem of nonlinear prediction of chaotic time series. The proposed algorithm yields results in both accuracy and convergence rates which are orders of magnitude superior compared to conventional backpropagation learning.     

PSO-optimized modular neural network trained by OWO-HWO algorithm for fault location in analog circuits

Fault diagnosis of analog circuits is a key problem in the theory of circuit networks and has been investigated by many researchers in recent decades. In this paper, an active filter circuit is used as the circuit under test (CUT) and is simulated in both fault-free and faulty conditions. A modular neural network model is proposed in this paper for soft fault diagnosis of the CUT. To optimize the structure of neural network modules in the proposed scheme, particle swarm optimization (PSO) algorithm is used to determine the number of hidden layer nodes of neural network modules. In addition, the output weight optimization–hidden weight optimization (OWO-HWO) training algorithm is employed, instead of conventional output weight optimization–backpropagation (OWO-BP) algorithm, to improve convergence speed in training of the neural network modules in proposed modular model. The performance of the proposed method is compared to that of monolithic multilayer perceptrons (MLPs) trained by OWO-BP and OWO-HWO algorithms, K-nearest neighbor (KNN) classifier and a related system with the same CUT. Experimental results show that the PSO-optimized modular neural network model which is trained by the OWO-HWO algorithm offers higher correct fault location rate in analog circuit fault diagnosis application as compared to the classic and monolithic investigated neural models.     

基于强化学习的壮语词性标注

目前壮语智能信息处理研究处于起步阶段,缺乏自动词性标注方法.针对壮语标注语料匮乏、人工标注费时费力而机器标注性能较差的现状,提出一种基于强化学习的壮语词性标注方法.依据壮语的文法特点和中文宾州树库符号构建标注词典,通过依存句法分析融合语义特征,并以长短期记忆网络为策略网络,利用循环记忆完善部分观测信息.在此基础上,引入强化学习框架,将目标词性作为环境反馈,通过特征学习不断逼近目标真实值.实验结果表明,该方法可缓解词性标注模型对训练语料库的依赖,能够快速扩大壮语标注词典的规模,实现壮语词性的自动标注.     

Inverting feedforward neural networks using linear and nonlinearprogramming

The problem of inverting trained feedforward neural networks is to find the inputs which yield a given output. In general, this problem is an ill-posed problem. We present a method for dealing with the inverse problem by using mathematical programming techniques. The principal idea behind the method is to formulate the inverse problem as a nonlinear programming problem, a separable programming (SP) problem, or a linear programming problem according to the architectures of networks to be inverted or the types of network inversions to be computed. An important advantage of the method over the existing iterative inversion algorithm is that various designated network inversions of multilayer perceptrons and radial basis function neural networks can be obtained by solving the corresponding SP problems, which can be solved by a modified simplex method. We present several examples to demonstrate the proposed method and applications of network inversions to examine and improve the generalization performance of trained networks. The results show the effectiveness of the proposed method.     

A fuzzy neural network model and its hardware implementation

A fuzzy classifier based on a four-layered feedforward neural network model is proposed. This connectionist fuzzy classifier, called CFC, realizes the weighted-Euclidean-distance fuzzy classification concept in a massively parallel manner to recognize input patterns. CFC employs a hybrid supervised/unsupervised learning scheme to organize referenced pattern vectors. This scheme not only overcomes the major drawbacks of multilayer perceptron models using the backpropagation algorithm, i.e., the local minimal problem and long training time, but also avoids the disadvantage of the huge storage space requirement of the probabilistic neural network. According to experimental results, CFC shows better accuracy for speech recognition than several existing methods, even in a noisy environment. Moreover, it has higher stability of recognition rates for different environmental conditions. A massively parallel hardware architecture has been developed to implement CFC. A bus-oriented multiprocessor, systolic processor structure, and pipelining are used to obtain low-cost, high-performance fuzzy classification     

Multilayer Potts Perceptrons With Levenberg–Marquardt Learning

This paper presents learning multilayer Potts perceptrons (MLPotts) for data driven function approximation. A Potts perceptron is composed of a receptive field and a $K$ -state transfer function that is generalized from sigmoid-like transfer functions of traditional perceptrons. An MLPotts network is organized to perform translation from a high-dimensional input to the sum of multiple postnonlinear projections, each with its own postnonlinearity realized by a weighted $K$-state transfer function. MLPotts networks span a function space that theoretically covers network functions of multilayer perceptrons. Compared with traditional perceptrons, weighted Potts perceptrons realize more flexible postnonlinear functions for nonlinear mappings. Numerical simulations show MLPotts learning by the Levenberg–Marquardt (LM) method significantly improves traditional supervised learning of multilayer perceptrons for data driven function approximation.      

基于混合神经网络的文本分类方法

为更好解决卷积神经网络提取特征不充分,难以处理长文本结构信息和捕获句子语义关系等问题,提出一种融合CNN和自注意力BiLSTM的并行神经网络模型TC-ABlstm.对传统的卷积神经网络进行改进,增强对文本局部特征的提取能力;设计结合注意力机制的双向长短期记忆神经网络模型来捕获文本上下文相关的全局特征;结合两个模型提取文...     

Training trajectories by continuous recurrent multilayer networks

This paper addresses the problem of training trajectories by means of continuous recurrent neural networks whose feedforward parts are multilayer perceptrons. Such networks can approximate a general nonlinear dynamic system with arbitrary accuracy. The learning process is transformed into an optimal control framework where the weights are the controls to be determined. A training algorithm based upon a variational formulation of Pontryagin's maximum principle is proposed for such networks. Computer examples demonstrating the efficiency of the given approach are also presented.     

Two-stage neural network regression of eye location in face images

Automatic eye localization is a crucial part of many computer vision algorithms for processing face images. Some of the existing algorithms can be very accurate, albeit at the cost of computational complexity. In this paper, a new solution to the problem of automatic eye localization is proposed. Eye localization is posed as a nonlinear regression problem solved by two feed-forward multilayer perceptrons (MLP) working in a cascade. The input feature vector of the first network is constructed from coefficients of a two dimensional discrete cosine transform(DCT) of a face image. The second network generates corrections based on small image patches. Feature extraction and neural network prediction have known and efficient implementations, thus the entire procedure can be very fast. The paper hints at the neural network structure and the procedure for generating artificial training samples from a low number of face images. In terms of accuracy, the method is comparable to state-of-the-art techniques; however it is based on numerical procedures that could be highly optimized (fast Fourier transform and matrix multiplication).     

面向答疑文本的词类标注方法的研究与实现

针对已有词类标注方法在标注网络答疑文本时存在的不足,文章提出了一种面向自然语言答疑文本的词类标注方法。该方法根据答疑文本的特点和后续关键信息提取的需要,对已有的词类标记集进行了扩展;用统计方法标注答疑真实文本,将其结果与正确结果相比较,从中获取词类排歧规则,使规则具有较强的文本针对性,以提高规则排歧的精度;对规则进行分类和优化,提高了标注的速度;采用先规则后统计的标注方法,较好地解决了答疑文本中规则和统计方法的组合问题。目前,该方法已在基于自然语言的网络答疑系统(NaturalLanguageOrientedWebAnswerSystem,,简称NL_WAS)中实现并得到了初步应用。     

Local linear perceptrons for classification

A structure composed of local linear perceptrons for approximating global class discriminants is investigated. Such local linear models may be combined in a cooperative or competitive way. In the cooperative model, a weighted sum of the outputs of the local perceptrons is computed where the weight is a function of the distance between the input and the position of the local perceptron. In the competitive model, the cost function dictates a mixture model where only one of the local perceptrons give output. Learning of the local models' positions and the linear mappings they implement are coupled and both supervised. We show that this is preferable to the uncoupled case where the positions are trained in an unsupervised manner before the separate, supervised training of mappings. We use goodness criteria based on the cross-entropy and give learning equations for both the cooperative and competitive cases. The coupled and uncoupled versions of cooperative and competitive approaches are compared among themselves and with multilayer perceptrons of sigmoidal hidden units and radial basis functions (RBFs) of Gaussian units on the application of recognition of handwritten digits. The criteria of comparison are the generalization accuracy, learning time, and the number of free parameters. We conclude that even on such a high-dimensional problem, such local models are promising. They generalize much better than RBF's and use much less memory. When compared with multilayer perceptrons, we note that local models learn much faster and generalize as well and sometimes better with comparable number of parameters.     

基于概率分布的图卷积文本分类模型

近年来,图神经网络模型因其对非欧氏数据的建模和对全局依赖关系的捕获能力而广泛应用于文本分类任务。现有的基于图卷积网络的分类模型中的构图方法存在消耗内存过大、难以适应新文本等问题。此外,现有研究中用于描述图节点间的全局依赖关系的方法并不完全适用于分类任务。为解决上述问题,该文设计并提出了基于概率分布的文本分类网络模型,以语料库中的词和标签为节点构建标签-词异构关系图,利用词语在各标签上的概率分布描述节点间的全局依赖关系,并通过图卷积操作进行文本表示学习。在5个公开的文本分类数据集上的实验表明,该文提出的模型在有效缩减图尺寸的同时,相比于其他文本分类网络模型取得了较为先进的结果。