Evolving Multilayer Perceptrons

This paper proposes a new version of a method (G-Prop, genetic backpropagation) that attempts to solve the problem of finding appropriate initial weights and learning parameters for a single hidden layer Multilayer Perceptron (MLP) by combining an evolutionary algorithm (EA) and backpropagation (BP). The EA selects the MLP initial weights, the learning rate and changes the number of neurons in the hidden layer through the application of specific genetic operators, one of which is BP training. The EA works on the initial weights and structure of the MLP, which is then trained using QuickProp; thus G-Prop combines the advantages of the global search performed by the EA over the MLP parameter space and the local search of the BP algorithm. The application of the G-Prop algorithm to several real-world and benchmark problems shows that MLPs evolved using G-Prop are smaller and achieve a higher level of generalization than other perceptron training algorithms, such as QuickPropagation or RPROP, and other evolutive algorithms. It also shows some improvement over previous versions of the algorithm.     

A class-modular feedforward neural network for handwriting recognition

Since the conventional feedforward neural networks for character recognition have been designed to classify a large number of classes with one large network structure, inevitably it poses the very complex problem of determining the optimal decision boundaries for all the classes involved in a high-dimensional feature space. Limitations also exist in several aspects of the training and recognition processes. This paper introduces the class modularity concept to the feedforward neural network classifier to overcome such limitations. In the class-modular concept, the original K-classification problem is decomposed into K 2-classification subproblems. A modular architecture is adopted which consists of K subnetworks, each responsible for discriminating a class from the other K−1 classes. The primary purpose of this paper is to prove the effectiveness of class-modular neural networks in terms of their convergence and recognition power. Several cases have been studied, including the recognition of handwritten numerals (10 classes), English capital letters (26 classes), touching numeral pairs (100 classes), and Korean characters in postal addresses (352 classes). The test results confirmed the superiority of the class-modular neural network and the interesting aspects on further investigations of the class modularity paradigm.     

DESIGN OF A FUZZY MIN-MAX HYPERBOX CLASSIFIER USING A SUPERVISED LEARNING METHOD

ABSTRACT

This investigation proposes a fuzzy min-max hyperbox classifier to solve M-class classification problems. In the proposed fuzzy min-max hyperbox classifier, a supervised learning method is implemented to generate min-max hyperboxes for the training patterns in each class so that the generated fuzzy min-max hyperbox classifier has a perfect classification rate in the training set. However, the 100% correct classification of the training set generally leads to overfitting. In order to improve this drawback, a procedure is employed to decrease the complexity of the input decision boundaries so that the generated fuzzy hyperbox classifier has a good generalization performance. Finally, two benchmark data sets are considered to demonstrate the good performance of the proposed approach for solving this classification problem.     

Fast and Stable Learning Utilizing Singular Regions of Multilayer Perceptron

In the parameter space of MLP(J), multilayer perceptron with J hidden units, there exist flat areas called singular regions created by applying reducibility mappings to the optimal solution of MLP( $J-1$ ). Since such singular regions cause serious stagnation of learning, a learning method to avoid singular regions has been desired. However, such avoiding does not guarantee the quality of the final solutions. This paper proposes a new learning method which does not avoid but makes good use of singular regions to stably and successively find excellent solutions commensurate with MLP(J). The proposed method worked well in our experiments using artificial and real data sets.     

Classification methodologies of multilayer perceptrons with sigmoid activation functions

This paper studies the classification mechanisms of multilayer perceptrons (MLPs) with sigmoid activation functions (SAFs). The viewpoint is presented that in the input space the hyperplanes determined by the hidden basis functions with values 0's do not play the role of decision boundaries, and such hyperplanes do not certainly go through the marginal regions between different classes. For solving an n-class problem, a single-hidden-layer perceptron with at least log2(n-1)?2 hidden nodes is needed. The final number of hidden neurons is still related to the sample distribution shapes and regions, but not to the number of samples and input dimensions. As a result, an empirical formula for optimally selecting the initial number of hidden nodes is proposed. The ranks of response matrixes of hidden layers should be taken as a main basis for pruning or growing the existing hidden neurons. A structure-fixed perceptron ought to learn more than one round from different starting weight points for one classification task, and only the group of weights and biases that has the best generalization performance should be reserved. Finally, three examples are given to verify the above viewpoints.     

A novel training scheme for multilayered perceptrons to realize proper generalization and incremental learning

The response of a multilayered perceptron (MLP) network on points which are far away from the boundary of its training data is generally never reliable. Ideally a network should not respond to data points which lie far away from the boundary of its training data. We propose a new training scheme for MLPs as classifiers, which ensures this. Our training scheme involves training subnets for each class present in the training data. Each subnet can decide whether a data point belongs to a certain class or not. Training each subnet requires data from the class which the subnet represents along with some points outside the boundary of that class. For this purpose we propose an easy but approximate method to generate points outside the boundary of a pattern class. The trained subnets are then merged to solve the multiclass classification problem. We show through simulations that an MLP trained by our method does not respond to points which lies outside the boundary of its training sample. Also, our network can deal with overlapped classes in a better manner. In addition, this scheme enables incremental training of an MLP, i.e., the MLP can learn new knowledge without forgetting the old knowledge.     

Supervised Versus Unsupervised Binary-Learning by Feedforward Neural Networks

Binary classification is typically achieved by supervised learning methods. Nevertheless, it is also possible using unsupervised schemes. This paper describes a connectionist unsupervised approach to binary classification and compares its performance to that of its supervised counterpart. The approach consists of training an autoassociator to reconstruct the positive class of a domain at the output layer. After training, the autoassociator is used for classification, relying on the idea that if the network generalizes to a novel instance, then this instance must be positive, but that if generalization fails, then the instance must be negative. When tested on three real-world domains, the autoassociator proved more accurate at classification than its supervised counterpart, MLP, on two of these domains and as accurate on the third (Japkowicz, Myers, & Gluck, 1995). The paper seeks to generalize these results and concludes that, in addition to learning aconcept in the absence of negative examples, 1) autoassociation is more efficient than MLP in multi-modal domains, and 2) it is more accurate than MLP in multi-modal domains for which the negative class creates a particularly strong need for specialization or the positive class creates a particularly weak need for specialization. In multi-modal domains for which the positive class creates a particularly strong need for specialization, on the other hand, MLP is more accurate than autoassociation.     

Neural Networks in Non-Euclidean Spaces

Multilayer Perceptrons (MLPs) use scalar products to compute weighted activation of neurons providing decision borders using combinations of soft hyperplanes. The weighted fun-in activation function may be replaced by a distance function between the inputs and the weights, offering a natural generalization of the standard MLP model. Non-Euclidean distance functions may also be introduced by normalization of the input vectors into an extended feature space. Both approaches influence the shapes of decision borders dramatically. An illustrative example showing these changes is provided.     

Application of multi-layer perceptron neural networks to vision problems

This paper discusses the application of a class of feed-forward Artificial Neural Networks (ANNs) known as Multi-Layer Perceptrons(MLPs) to two vision problems: recognition and pose estimation of 3D objects from a single 2D perspective view; and handwritten digit recognition. In both cases, a multi-MLP classification scheme is developed that combines the decisions of several classifiers. These classifiers operate on the same feature set for the 3D recognition problem whereas different feature types are used for the handwritten digit recognition. The backpropagationlearning rule is used to train the MLPs. Application of the MLP architecture to other vision problems is also briefly discussed.     

面向分布式数据流大数据分类的多变量决策树

分布式数据流大数据中的类别边界不规则且易变,因此基于单变量决策树的集成分类器需要较大数量的基分类器才能准确地近似表达类别边界,这将降低集成分类器的学习与分类性能.因而,本文提出了基于几何轮廓相似度的多变量决策树.在最优基准向量的引导下将n维空间样本点投影到一维空间以建立有序投影点集合,然后通过类别投影边界将有序投影点集合划分为多个子集,接着分别对不同类别集合的交集递归投影分裂,最终生成决策树.实验表明,本文提出的多变量决策树GODT具有很高的分类精度和较低的训练时间,有效结合了单变量决策树学习效率高与多变量决策树表示能力强的优点.     

Distributed learning with bagging-like performance

Bagging forms a committee of classifiers by bootstrap aggregation of training sets from a pool of training data. A simple alternative to bagging is to partition the data into disjoint subsets. Experiments with decision tree and neural network classifiers on various datasets show that, given the same size partitions and bags, disjoint partitions result in performance equivalent to, or better than, bootstrap aggregates (bags). Many applications (e.g., protein structure prediction) involve use of datasets that are too large to handle in the memory of the typical computer. Hence, bagging with samples the size of the data is impractical. Our results indicate that, in such applications, the simple approach of creating a committee of n classifiers from disjoint partitions each of size 1/n (which will be memory resident during learning) in a distributed way results in a classifier which has a bagging-like performance gain. The use of distributed disjoint partitions in learning is significantly less complex and faster than bagging.     

Discriminative power for ensembles of linear decision rules

A novel class of ensembles of linear decision rules is introduced which includes majority voting-based ensembles as a particular case. Based on this general framework, new results are given that state the ability of a subclass to discriminate between two infinite subsets A and B in R ⁿ , thus generalizing Mazurov’s theorem for two finite sets.     

动态置信度的序列选择增量学习方法

贝叶斯在训练样本不完备的情况下,对未知类别新增训练集进行增量学习时,会将分类错误的训练样本过早地加入到分类器中而降低其性能,另外增量学习采用固定的置信度评估参数会使其效率低下,泛化性能不稳定.为解决上述问题,提出一种动态置信度的序列选择增量学习方法.首先,在现有的分类器基础上选出分类正确的文本组成新增训练子集.其次,利用置信度动态监控分类器性能来对新增训练子集进行批量实例选择.最后,通过选择合理的学习序列来强化完备数据的积极影响,弱化噪声数据的消极影响,并实现对测试文本的分类.实验结果表明,本文提出的方法在有效提高分类精度的同时也能明显改善增量学习效率.     

Neural networks with enhanced outlier rejection ability for off-line handwritten word recognition

For a segmentation and dynamic programming-based handwritten word recognition system, outlier rejection at the character level can improve word recognition performance because it reduces the chances that erroneous combinations of segments result in high word confidence values. We studied the multilayer perceptron (MLP) and a variant of radial basis function network (RBF) with the goal to use them as character level classifiers that have enhanced outlier rejection ability. The variant of the RBF uses principal component analysis (PCA) on the clusters defined by the nodes in the hidden layer. It was also trained with and without a regularization term that was aimed at minimizing the variances of the nodes in the hidden layer. Our experiments on handwritten word recognition showed: (1) In the case of MLPs, using more hidden nodes than that required for classification and including outliers in the training data can improve outlier rejection performance; (2) in the case of PCA-RBFs, training with the regularization term and no outlier can achieve performance very close to training with outliers. These results are both interesting. Result (1) is of interest because it is well known that minimizing the number of parameters, and therefore keeping the number of hidden units low, should increase the generalization capability. On the other hand, using more hidden units increases the chances of creating closed decision regions, as predicted by the theory in Gori and Scarselli (IEEE Trans. PAMI 20 (11) (1998) 1121). Result (2) is a strong statement in support of the use of regularization terms for the training of RBF-type neural networks in problems such as handwriting recognition for which outlier rejection is important. Additional tests on combining MLPs and PCA-RBF networks showed the potential to improve word recognition performance by exploiting the complementarity of these two kinds of neural networks.     

A Self-Tester for Linear Functions over the Integers with an Elementary Proof of Correctness

We present simple, self-contained proofs of correctness for algorithms for linearity testing and program checking of linear functions on finite subsets of integers represented as n-bit numbers. In addition we explore a generalization of self-testing to homomorphisms on a multidimensional vector space. We show that our self-testing algorithm for the univariate case can be directly generalized to vector space domains. The number of queries made by our algorithms is independent of domain size.     

Representing Hyper-arithmetical Sets by Equations over Sets of Integers

Systems of equations with sets of integers as unknowns are considered. It is shown that the class of sets representable by unique solutions of equations using the operations of union and addition, defined as S+T={m+n?Om??S,n??T}, and with ultimately periodic constants is exactly the class of hyper-arithmetical sets. Equations using addition only can represent every hyper-arithmetical set under a simple encoding. All hyper-arithmetical sets can also be represented by equations over sets of natural numbers equipped with union, addition and subtraction $S \mathop {\mbox {$-^{\hspace {-.5em}\cdot }\,\,$}}T=\{m-n \mid m \in S, n \in T, m \geq n\}$ . Testing whether a given system has a solution is $\varSigma ^{1}_{1}$ -complete for each model. These results, in particular, settle the expressive power of the most general types of language equations, as well as equations over subsets of free groups.     

Genetic algorithms for generation of class boundaries.

A method is described for finding decision boundaries, approximated by piecewise linear segments, for classifying patterns in R(N),N>/=2, using an elitist model of genetic algorithms. It involves generation and placement of a set of hyperplanes (represented by strings) in the feature space that yields minimum misclassification. A scheme for the automatic deletion of redundant hyperplanes is also developed in case the algorithm starts with an initial conservative estimate of the number of hyperplanes required for modeling the decision boundary. The effectiveness of the classification methodology, along with the generalization ability of the decision boundary, is demonstrated for different parameter values on both artificial data and real life data sets having nonlinear/overlapping class boundaries. Results are compared extensively with those of the Bayes classifier, k-NN rule and multilayer perceptron.     

Binary labelling and decision-level fusion

Two binary labelling techniques for decision-level fusion are considered for reducing correlation in the context of multiple classifier systems. First, we describe a method based on error correcting coding that uses binary code words to decompose a multi-class problem into a set of complementary two-class problems. We look at the conditions necessary for reduction of error and introduce a modified version that is less sensitive to code word selection. Second, we describe a partitioning method for two-class problems that transforms each training pattern into a vertex of the binary hypercube. A constructive algorithm for binary-to-binary mappings identifies a set of inconsistently classified patterns, random subsets of which are used to perturb base classifier training sets. Experimental results on artificial and real data, using a combination of simple neural network classifiers, demonstrate improvement in performance for these techniques, the first suitable for k-class problems, k>2 and the second for k=2.     

改进的基于BiLSTM的网络入侵检测方法

针对传统的基于机器学习的网络入侵检测技术存在准确率偏低和泛化能力较差的问题,提出一种改进的基于BiLSTM的网络入侵检测方法,融合BiLSTM和Batch Normalization机制的优点,更好解析数据之间潜在的联系。在NSLKDD的两个数据集上的实验结果表明,与循环神经网络入侵检测方法相比,在二分类实验的两个测试集中的准确率分别提高了2.9%和8.41%,五分类实验中的准确率也有所提高。