Saturated Perceptrons for Maximum Margin and Minimum Misclassification Error

This Letter discusses the application of gradient-based methods to train a single layer perceptron subject to the constraint that the saturation degree of the sigmoid activation function (measured as its maximum slope in the sample space) is fixed to a given value. From a theoretical standpoint, we show that, if the training set is not linearly separable, the minimization of an L _p error norm provides an approximation to the minimum error classifier, provided that the perceptron is highly saturated. Moreover, if data are linearly separable, the perceptron approximates the maximum margin classifier     

Classification of linearly nonseparable patterns by linearthreshold elements

Learning and convergence properties of linear threshold elements or perceptrons are well understood for the case where the input vectors (or the training sets) to the perceptron are linearly separable. Little is known, however, about the behavior of the perceptron learning algorithm when the training sets are linearly nonseparable. We present the first known results on the structure of linearly nonseparable training sets and on the behavior of perceptrons when the set of input vectors is linearly nonseparable. More precisely, we show that using the well known perceptron learning algorithm, a linear threshold element can learn the input vectors that are provably learnable, and identify those vectors that cannot be learned without committing errors. We also show how a linear threshold element can be used to learn large linearly separable subsets of any given nonseparable training set. In order to develop our results, we first establish formal characterizations of linearly nonseparable training sets and define learnable structures for such patterns. We also prove computational complexity results for the related learning problems. Next, based on such characterizations, we show that a perceptron does the best one can expect for linearly nonseparable sets of input vectors and learns as much as is theoretically possible.     

一种基于核函数的非线性感知器算法

为了提高经典Rosenblatt感知器算法的分类能力，该文提出一种基于核函数的非线性感知器算法，简称核感知器算法，其特点是用简单的迭代过程和核函数来实现非线性分类器的一种设计，核感知器算法能够处理原始属性空间中线性不可分问题和高维特征空间中线性可分问题。同时，文中详细分析了其算法与径向基函数神经网络、势函数方法和支持向量机等非线性算法的关系。人工和实际数据的计算结果表明：与线性感知器算法相比，核感知器算法可以有效地提高分类精度。     

Adaptive Ho-Kashyap rules for perceptron training

Three adaptive versions of the Ho-Kashyap perceptron training algorithm are derived based on gradient descent strategies. These adaptive Ho-Kashyap (AHK) training rules are comparable in their complexity to the LMS and perceptron training rules and are capable of adaptively forming linear discriminant surfaces that guarantee linear separability and of positioning such surfaces for maximal classification robustness. In particular, a derived version called AHK II is capable of adaptively identifying critical input vectors lying close to class boundaries in linearly separable problems. The authors extend this algorithm as AHK III, which adds the capability of fast convergence to linear discriminant surfaces which are good approximations for nonlinearly separable problems. This is achieved by a simple built-in unsupervised strategy which allows for the adaptive grading and discarding of input vectors causing nonseparability. Performance comparisons with LMS and perceptron training are presented.     

Global convergence and limit cycle behavior of weights of perceptron.

In this paper, it is found that the weights of a perceptron are bounded for all initial weights if there exists a nonempty set of initial weights that the weights of the perceptron are bounded. Hence, the boundedness condition of the weights of the perceptron is independent of the initial weights. Also, a necessary and sufficient condition for the weights of the perceptron exhibiting a limit cycle behavior is derived. The range of the number of updates for the weights of the perceptron required to reach the limit cycle is estimated. Finally, it is suggested that the perceptron exhibiting the limit cycle behavior can be employed for solving a recognition problem when downsampled sets of bounded training feature vectors are linearly separable. Numerical computer simulation results show that the perceptron exhibiting the limit cycle behavior can achieve a better recognition performance compared to a multilayer perceptron.     

感知器学习算法研究

介绍感知器学习算法及其变种,给出各种感知器算法的伪代码,指出各种算法的优点。给出感知器算法在线性可分和线性不可分情况下的误差界定理,讨论各种感知器学习算法的误差界理论,给出各种算法的误差界。介绍感知器学习算法在在线优化场景、强化学习场景和赌博机算法中的应用,并对未解决的问题进行讨论。     

A constructive design method for two-layer perceptrons and its applicatio to the design of modular neural networks

Abstract: A multilayer perceptron is known to be capable of approximating any smooth function to any desired accuracy if it has a sufficient number of hidden neurons. But its training, based on the gradient method, is usually a time consuming procedure that may converge toward a local minimum, and furthermore its performance is greatly influenced by the number of hidden neurons and their initial weights. Usually these crucial parameters are determined based on the trial and error procedure, requiring much experience on the designer's part.
In this paper, a constructive design method (CDM) has been proposed for a two-layer perceptron that can approximate a class of smooth functions whose feature vector classes are linearly separable. Based on the analysis of a given data set sampled from the target function, feature vectors that can characterize the function'well'are extracted and used to determine the number of hidden neurons and the initial weights of the network. But when the classes of the feature vectors are not linearly separable, the network may not be trained easily, mainly due to the interference among the hyperplanes generated by hidden neurons. Next, to compensate for this interference, a refined version of the modular neural network (MNN) has been proposed where each network module is created by CDM. After the input space has been partitioned into many local regions, a two-layer perceptron constructed by CDM is assigned to each local region. By doing this, the feature vector classes are more likely to become linearly separable in each local region and as a result, the function may be approximated with greatly improved accuracy by MNN. An example simulation illustrates the improvements in learning speed using a smaller number of neurons.     

Performance optimization of integrated resilience engineering and lean production principles

This paper conducts performance assessment from integrated resilience engineering (IRE) and lean production points of view. This is the first study that evaluates the impact of integrated resilience engineering (IRE) on lean production principles. Second, this study considers integrated impact of lean production by a unique intelligent algorithm. The proposed algorithm is composed of radial basis function (RBF), multi-layer perceptron (MLP) and adaptive neuro-fuzzy inference system (ANFIS). Moreover, the algorithm is capable of handling both crisp and fuzzy data due to the existence of intelligent approach. The proposed algorithm is equipped with verification and validation mechanism through conventional regression, statistical methods and data envelopment analysis. To demonstrate the applicability of the study, a real-world pipe manufacturer is considered as our case study. The results showed that “pull system” and “fault tolerant” among lean and IRE factors, respectively have been implemented inappropriately, while other factors are either suitably executed or ineffective.     

Uncertainty of data, fuzzy membership functions, and multilayer perceptrons

Probability that a crisp logical rule applied to imprecise input data is true may be computed using fuzzy membership function (MF). All reasonable assumptions about input uncertainty distributions lead to MFs of sigmoidal shape. Convolution of several inputs with uniform uncertainty leads to bell-shaped Gaussian-like uncertainty functions. Relations between input uncertainties and fuzzy rules are systematically explored and several new types of MFs discovered. Multilayered perceptron (MLP) networks are shown to be a particular implementation of hierarchical sets of fuzzy threshold logic rules based on sigmoidal MFs. They are equivalent to crisp logical networks applied to input data with uncertainty. Leaving fuzziness on the input side makes the networks or the rule systems easier to understand. Practical applications of these ideas are presented for analysis of questionnaire data and gene expression data.     

Fuzzy multi-layer perceptron, inferencing and rule generation.

A connectionist expert system model, based on a fuzzy version of the multilayer perceptron developed by the authors, is proposed. It infers the output class membership value(s) of an input pattern and also generates a measure of certainty expressing confidence in the decision. The model is capable of querying the user for the more important input feature information, if and when required, in case of partial inputs. Justification for an inferred decision may be produced in rule form, when so desired by the user. The magnitudes of the connection weights of the trained neural network are utilized in every stage of the proposed inferencing procedure. The antecedent and consequent parts of the justificatory rules are provided in natural forms. The effectiveness of the algorithm is tested on the speech recognition problem, on some medical data and on artificially generated intractable (linearly nonseparable) pattern classes.     

一种基于L2范数的软核感知机

感知机只能解决线性可分问题。支持向量机中的L2范数软边缘算法可以将线性不可分问题转化为线性可分问题。基于这一事实,提出一种基于L2范数的软核感知机(SoftKernelPerceptron,SKP),将感知机算法直接用于求解L2范数软边缘算法决定的线性可分问题。通过使用核技巧,得到一种普适的非线性分类方法。实际数据库的测试结果表明,SKP算法能够有效地解决非线性问题,并且继承了感知机运算简单速度快的优点。     

Large Margin Classification Using the Perceptron Algorithm

We introduce and analyze a new algorithm for linear classification which combines Rosenblatt's perceptron algorithm with Helmbold and Warmuth's leave-one-out method. Like Vapnik's maximal-margin classifier, our algorithm takes advantage of data that are linearly separable with large margins. Compared to Vapnik's algorithm, however, ours is much simpler to implement, and much more efficient in terms of computation time. We also show that our algorithm can be efficiently used in very high dimensional spaces using kernel functions. We performed some experiments using our algorithm, and some variants of it, for classifying images of handwritten digits. The performance of our algorithm is close to, but not as good as, the performance of maximal-margin classifiers on the same problem, while saving significantly on computation time and programming effort.     

Fuzzy perceptron neural networks for classifiers with numericaldata and linguistic rules as inputs

This paper presents a novel learning algorithm of fuzzy perceptron neural networks (FPNNs) for classifiers that utilize expert knowledge represented by fuzzy IF-THEN rules as well as numerical data as inputs. The conventional linear perceptron network is extended to a second-order one, which is much more flexible for defining a discriminant function. In order to handle fuzzy numbers in neural networks, level sets of fuzzy input vectors are incorporated into perceptron neural learning. At different levels of the input fuzzy numbers, updating the weight vector depends on the minimum of the output of the fuzzy perceptron neural network and the corresponding nonfuzzy target output that indicates the correct class of the fuzzy input vector. This minimum is computed efficiently by employing the modified vertex method. Moreover, the fuzzy pocket algorithm is introduced into our fuzzy perceptron learning scheme to solve the nonseparable problems. Simulation results demonstrate the effectiveness of the proposed FPNN model     

A characteristic vector matrix method for multicategory pattern classification†

An algorithm of classifying multicategory patterns using a characteristic vector matrix is described. It is shown that, if the training patterns are linearly separable, the algorithm will guarantee convergence. Furthermore, if the training patterns are not linearly separable, then a method for testing the existence of a limit cycle in the iteration procedure is developed. Examples indicate that the algorithm has rapid convergence and its usefulness is promising. Also a comparison of the convergence condition of the algorithm with that of the fractional correction rule is made.     

Perceptron Learning Revisited: The Sonar Targets Problem

Recently it was pointed out that a well-known benchmark data set, the sonar target data, indeed is linearly separable. This fact comes somewhat surprising, since earlier studies involving delta rule trained perceptrons did not achieve the separation of the training data. These results immediately raise the question of why a perceptron with a continuous activation function may fail to recognize linear separability and how to remedy this failure. The study of these issues directly leads to a performance comparison of a wide variety of different perceptron training procedures on real world data.     

两种模糊决策树算法的对比研究

模糊决策树归纳是从具有模糊表示的示例中学习规则的一种重要方法,从符号值属性类分明的数据中提取规则可视为模糊决策树归纳的一种特殊情况。由于构建最优的模糊决策树是NP-hard,因此,针对启发式算法的研究是非常必要的。该文主要对两种启发式算法即FuzzyID3和Min-Ambiguity算法应用于符号值属性并且类分明情况所作的分析比较。通过实验与理论分析,发现FuzzyID3算法应用于符号值属性类分明的数据库时从训练准确度、测试准确度和树的规模等方面都要优于Min-Ambiguity算法。     

Fuzzy kernel perceptron

A new learning method, the fuzzy kernel perceptron (FKP), in which the fuzzy perceptron (FP) and the Mercer kernels are incorporated, is proposed in this paper. The proposed method first maps the input data into a high-dimensional feature space using some implicit mapping functions. Then, the FP is adopted to find a linear separating hyperplane in the high-dimensional feature space. Compared with the FP, the FKP is more suitable for solving the linearly nonseparable problems. In addition, it is also more efficient than the kernel perceptron (KP). Experimental results show that the FKP has better classification performance than FP, KP, and the support vector machine.     

基于混合核函数的可能性C-均值聚类算法

针对传统的模糊C-均值算法对于非球形分布的数据聚类效果不理想且易受到噪声数据的影响,利用可能性C-均值算法具有良好的抗噪声性能,将混合核函数引入到该算法中,提出了一种基于混合核函数的可能性C-均值(HKPCM)聚类算法。该算法将原空间的待分类样本映射到一个高维的特征空间(核空间)中,使得样本变得线性可分,然后在核空间中进行聚类。实验结果证实了HKPCM算法的可行性和有效性。     

Selection of optimal set of weights in a layered network using genetic algorithms

Genetic algorithms represent a class of highly parallel robust adaptive search processes for solving a wide range of optimization and machine learning problems. The present work is an attempt to demonstrate their effectiveness to search a global optimal solution to select a decision boundary for a pattern recognition problem using a multilayer perceptron. The proposed method incorporates a new concept of nonlinear selection for creating mating pools and a weighted error as a fitness function. Since there is no need for the backpropagation technique, the algorithm is computationally efficient and avoids all the drawbacks of the backpropagation algorithm. Moreover, it does not depend on the sequence of the training data. The performance of the method along with the convergence has been experimentally demonstrated for both linearly separable and nonseparable pattern classes.     

Parameterized attribute reduction with Gaussian kernel based fuzzy rough sets

Fuzzy rough sets are considered as an effective tool to deal with uncertainty in data analysis, and fuzzy similarity relations are used in fuzzy rough sets to calculate similarity between objects. On the other hand in kernel tricks, a kernel maps data into a higher dimensional feature space where the resulting structure of the learning task is linearly separable, while the kernel is the inner product of this feature space and can also be viewed as a similarity function. It has been reported there is an overlap between family of kernels and collection of fuzzy similarity relations. This fact motivates the idea in this paper to use some kernels as fuzzy similarity relations and develop kernel based fuzzy rough sets. First, we consider Gaussian kernel and propose Gaussian kernel based fuzzy rough sets. Second we introduce parameterized attribute reduction with the derived model of fuzzy rough sets. Structures of attribute reduction are investigated and an algorithm with discernibility matrix to find all reducts is developed. Finally, a heuristic algorithm is designed to compute reducts with Gaussian kernel fuzzy rough sets. Several experiments are provided to demonstrate the effectiveness of the idea.