Support-vector-based fuzzy neural network for pattern classification

Fuzzy neural networks (FNNs) for pattern classification usually use the backpropagation or C-cluster type learning algorithms to learn the parameters of the fuzzy rules and membership functions from the training data. However, such kinds of learning algorithms usually cannot minimize the empirical risk (training error) and expected risk (testing error) simultaneously, and thus cannot reach a good classification performance in the testing phase. To tackle this drawback, a support-vector-based fuzzy neural network (SVFNN) is proposed for pattern classification in this paper. The SVFNN combines the superior classification power of support vector machine (SVM) in high dimensional data spaces and the efficient human-like reasoning of FNN in handling uncertainty information. A learning algorithm consisting of three learning phases is developed to construct the SVFNN and train its parameters. In the first phase, the fuzzy rules and membership functions are automatically determined by the clustering principle. In the second phase, the parameters of FNN are calculated by the SVM with the proposed adaptive fuzzy kernel function. In the third phase, the relevant fuzzy rules are selected by the proposed reducing fuzzy rule method. To investigate the effectiveness of the proposed SVFNN classification, it is applied to the Iris, Vehicle, Dna, Satimage, Ijcnn1 datasets from the UCI Repository, Statlog collection and IJCNN challenge 2001, respectively. Experimental results show that the proposed SVFNN for pattern classification can achieve good classification performance with drastically reduced number of fuzzy kernel functions.     

POPFNN-CRI(S): pseudo outer product based fuzzy neural network using the compositional rule of inference and singleton fuzzifier

A pseudo-outer product based fuzzy neural network using the compositional rule of inference and singleton fuzzifier [POPFNN-CRI(S)] is proposed in this paper. The correspondence of each layer in the proposed POPFNN-CRI(S) to the compositional rule of inference using standard T-norm and fuzzy relation gives it a strong theoretical foundation. The proposed POPFNN-CRI(S) training consists of two phases; namely: the fuzzy membership derivation phase using the novel fuzzy Kohonen partition (FKP) and pseudo Kohonen partition (PFKP) algorithms, and the rule identification phase using the novel one-pass POP learning algorithm. The proposed two-phase learning process effectively constructs the membership functions and identifies the fuzzy rules. Extensive experimental results based on the classification performance of the POPFNN-CRI(S) using the Anderson's Iris data are presented for discussion. Results show that the POPFNN-CRI(S) has taken only 15 training iterations and misclassify only three out of all the 150 patterns in the Anderson's Iris data.     

Genetic tuning of fuzzy inference within fuzzy classifier systems

Abstract: In generating a suitable fuzzy classifier system, significant effort is often placed on the determination and the fine tuning of the fuzzy sets. However, in such systems little thought is given to the way in which membership functions are combined within the fuzzy rules. Often traditional fuzzy inference strategies are used which consequently provide no control over how strongly or weakly the inference is applied within these rules. Furthermore such strategies will allow no interaction between grades of membership. A number of theoretical fuzzy inference operators have been proposed for both regression and classification problems but they have not been investigated in the context of real-world applications. In this paper we propose a novel genetic algorithm framework for optimizing the strength of fuzzy inference operators concurrently with the tuning of membership functions for a given fuzzy classifier system. Each fuzzy system is generated using two well-established decision tree algorithms: C4.5 and CHAID. This will enable both classification and regression problems to be addressed within the framework. Each solution generated by the genetic algorithm will produce a set of fuzzy membership functions and also determine how strongly the inference will be applied within each fuzzy rule. We investigate several theoretical proven fuzzy inference techniques (T-norms) in the context of both classification and regression problems. The methodology proposed is applied to a number of real-world data sets in order to determine the effects of the simultaneous tuning of membership functions and inference parameters on the accuracy and robustness of fuzzy classifiers.     

AUTOMATICALLY CONSTRUCTING MEMBERSHIP FUNCTIONS AND GENERATING FUZZY RULES USING GENETIC ALGORITHMS

The most important task in designing a fuzzy classification system is to find a set of fuzzy rules from training data to deal with a specific classification problem. In recent years, many methods have been proposed to construct membership functions and generate fuzzy rules from training data for handling fuzzy classification problems. We propose a new method to generate fuzzy rules from training data by using genetic algorithms (GAs). First, we divide the training data into several clusters by using the weighted distance clustering method and generate a fuzzy rule for each cluster. Then, we use GAs to tune the membership functions of the generated fuzzy rules. The proposed method attains a higher average classification accuracy rate than the existing methods.     

Construction of a neuron-fuzzy classification model based on feature-extraction approach

In this paper, a Feature-Extraction Neuron-Fuzzy Classification Model (FENFCM) is proposed that enables the extraction of feature variables and provides the classification results. The proposed classification model synergistically integrates a standard fuzzy inference system and a neural network with supervised learning. The FENFCM automatically generates the fuzzy rules from the numerical data and triangular functions that are used as membership functions both in the feature extraction unit and in the inference unit. To adapt the proposed FENFCM, two modificatory algorithms are applied. First, we utilize Evolutionary Programming (EP) to determine the distribution of fuzzy sets for each feature variable of the feature extraction unit. Second, the Weight Revised Algorithm (WRA) is used to regulate the weight grade of the principal output node of the inference unit. Finally, the proposed FENFCM is validated using two benchmark data sets: the Wine database and the Iris database. Computer simulation results demonstrate that the proposed classification model can provide a sufficiently high classification rate in comparison with that of other models proposed in the literature.     

Evolutionary design of a fuzzy classifier from data

Genetic algorithms show powerful capabilities for automatically designing fuzzy systems from data, but many proposed methods must be subjected to some minimal structure assumptions, such as rule base size. In this paper, we also address the design of fuzzy systems from data. A new evolutionary approach is proposed for deriving a compact fuzzy classification system directly from data without any a priori knowledge or assumptions on the distribution of the data. At the beginning of the algorithm, the fuzzy classifier is empty with no rules in the rule base and no membership functions assigned to fuzzy variables. Then, rules and membership functions are automatically created and optimized in an evolutionary process. To accomplish this, parameters of the variable input spread inference training (VISIT) algorithm are used to code fuzzy systems on the training data set. Therefore, we can derive each individual fuzzy system via the VISIT algorithm, and then search the best one via genetic operations. To evaluate the fuzzy classifier, a fuzzy expert system acts as the fitness function. This fuzzy expert system can effectively evaluate the accuracy and compactness at the same time. In the application section, we consider four benchmark classification problems: the iris data, wine data, Wisconsin breast cancer data, and Pima Indian diabetes data. Comparisons of our method with others in the literature show the effectiveness of the proposed method.     

The MSFAM: a modified fuzzy ARTMAP system

A fuzzy ARTMAP system is a system for incremental supervised learning of recognition categories and multi-dimensional maps in response to an arbitrary sequence of analog or binary input vectors. Fuzzy ARTMAP systems have been benchmarked against a variety of machine learning, neural networks, and genetic algorithms with considerable success. Owing to many appealing properties, fuzzy ARTMAP systems provide a natural basis for many researchers. Many different approaches have been proposed to modify fuzzy ARTMAP systems. In this paper, we propose a new approach to modifying a fuzzy ARTMAP system. We refer to the new system as the modified and simplified fuzzy ARTMAP (MSFAM) system. The aims of MSFAM systems are not only to reduce the architectural redundancy of the fuzzy ARTMAP system, but also to make extracted rules more comprehensible and concise. Four data sets were used for demonstrating the performance of the proposed MSFAM system.     

Effect of merging order on performance of fuzzy induction

Most fuzzy controllers must predefine membership functions and fuzzy inference rules to map numeric data into fuzzy linguistic values and make fuzzy reasoning work. In T.P. Hong, C.Y. Lee, Fuzzy Sets and Systems 84 (1996) 33–47, we proposed a general learning method for automatically deriving fuzzy-if-then rules and membership functions from a set of given training examples by merging the decision tables and membership functions. The merging order of the attributes, however, has great consequences on the accuracy of the final learning results. In this paper, we present appropriate heuristics to determine the merging order. Less relevant attributes will be processed earlier to reduce the complexity of the decision table. Experiments were also made, showing that our proposed heuristics demonstrate good performance.     

Hybrid identification of fuzzy rule‐based models

In this study, we propose a hybrid identification algorithm for a class of fuzzy rule‐based systems. The rule‐based fuzzy modeling concerns structure optimization and parameter identification using the fuzzy inference methods and hybrid structure combined with two methods of optimization theories for nonlinear systems. Two types of inference methods of a fuzzy model concern a simplified and linear type of inference. The proposed hybrid optimal identification algorithm is carried out using a combination of genetic algorithms and an improved complex method. The genetic algorithms determine initial parameters of the membership function of the premise part of the fuzzy rules. In the sequel, the improved complex method (being in essence a powerful auto‐tuning algorithm) leads to fine‐tuning of the parameters of the respective membership functions. An aggregate performance index with a weighting factor is proposed in order to achieve a balance between performance of the fuzzy model obtained for the training and testing data. Numerical examples are included to evaluate the performance of the proposed model. They are also contrasted with the performance of the fuzzy models existing in the literature. © 2002 John Wiley & Sons, Inc.     

DCT-Yager FNN: a novel Yager-based fuzzy neural network with the discrete clustering technique.

Earlier clustering techniques such as the modified learning vector quantization (MLVQ) and the fuzzy Kohonen partitioning (FKP) techniques have focused on the derivation of a certain set of parameters so as to define the fuzzy sets in terms of an algebraic function. The fuzzy membership functions thus generated are uniform, normal, and convex. Since any irregular training data is clustered into uniform fuzzy sets (Gaussian, triangular, or trapezoidal), the clustering may not be exact and some amount of information may be lost. In this paper, two clustering techniques using a Kohonen-like self-organizing neural network architecture, namely, the unsupervised discrete clustering technique (UDCT) and the supervised discrete clustering technique (SDCT), are proposed. The UDCT and SDCT algorithms reduce this data loss by introducing nonuniform, normal fuzzy sets that are not necessarily convex. The training data range is divided into discrete points at equal intervals, and the membership value corresponding to each discrete point is generated. Hence, the fuzzy sets obtained contain pairs of values, each pair corresponding to a discrete point and its membership grade. Thus, it can be argued that fuzzy membership functions generated using this kind of a discrete methodology provide a more accurate representation of the actual input data. This fact has been demonstrated by comparing the membership functions generated by the UDCT and SDCT algorithms against those generated by the MLVQ, FKP, and pseudofuzzy Kohonen partitioning (PFKP) algorithms. In addition to these clustering techniques, a novel pattern classifying network called the Yager fuzzy neural network (FNN) is proposed in this paper. This network corresponds completely to the Yager inference rule and exhibits remarkable generalization abilities. A modified version of the pseudo-outer product (POP)-Yager FNN called the modified Yager FNN is introduced that eliminates the drawbacks of the earlier network and yi- elds superior performance. Extensive experiments have been conducted to test the effectiveness of these two networks, using various clustering algorithms. It follows that the SDCT and UDCT clustering algorithms are particularly suited to networks based on the Yager inference rule.     

On the design of neural-fuzzy control system

This article presents a neural–network-based fuzzy logic control (NN–FLC) system. The NN–FLC model has the learning capabilities for constructing membership functions and extracting fuzzy rules from training examples. Both unsupervised and supervised training algorithms are used to find the membership functions of the FLC. Competitive learning algorithms are employed to evaluate fuzzy logic rules. Matlab programs using both neural and fuzzy toolboxes are developed to implement the NN–FLC model. Computer simulations of the inverted pendulum controlled by NN–FLC system were conducted to illustrate the self-learning ability of the network. © 1998 John Wiley & Sons, Inc.13: 11–26, 1998     

Research and Design of a Fuzzy Neural Expert System

We have developed a fuzzy neural expert system that has the precision and learning ability of a neural network.Knowledge is acquired from domain experts as fuzzy rules and membership functions.Then,they are converted into a neural network which implements fuzzy inference without rule matching.The neural network is applied to problem-solving and learns from the data obtained during operation to enhance the accuracy.The learning ability of the neural network makes it easy to modify the membership functions defined by domain experts.Also,by modifying the weights of neural networks adaptively,the problem of belief propagation in conventional expert systems can be solved easily.Converting the neural network back into fuzzy rules and membership functions helps explain the inner representation and operation of the neural network.     

Cascade ARTMAP: integrating neural computation and symbolic knowledge processing

This paper introduces a hybrid system termed cascade adaptive resonance theory mapping (ARTMAP) that incorporates symbolic knowledge into neural-network learning and recognition. Cascade ARTMAP, a generalization of fuzzy ARTMAP, represents intermediate attributes and rule cascades of rule-based knowledge explicitly and performs multistep inferencing. A rule insertion algorithm translates if-then symbolic rules into cascade ARTMAP architecture. Besides that initializing networks with prior knowledge can improve predictive accuracy and learning efficiency, the inserted symbolic knowledge can be refined and enhanced by the cascade ARTMAP learning algorithm. By preserving symbolic rule form during learning, the rules extracted from cascade ARTMAP can be compared directly with the originally inserted rules. Simulations on an animal identification problem indicate that a priori symbolic knowledge always improves system performance, especially with a small training set. Benchmark study on a DNA promoter recognition problem shows that with the added advantage of fast learning, cascade ARTMAP rule insertion and refinement algorithms produce performance superior to those of other machine learning systems and an alternative hybrid system known as knowledge-based artificial neural network (KBANN). Also, the rules extracted from cascade ARTMAP are more accurate and much cleaner than the NofM rules extracted from KBANN.     

一种模糊逻辑推理神经网络的结构及算法设计

建立了一种基于模糊逻辑推理的神经网络．由样本获取的初始规则确定规则层神经元个数，并确立模糊化层与规则层之间的连接．利用黄金分割法确定模糊化层隶属度函数的初始中心和宽度；根据初始规则的结论确定清晰化层的初始权值；针对网络结构提出了改进的BP算法．仿真实例表明，网络结构合理。具有较好的非线性映射能力，改进的BP算法适合于此网络，与另一种模糊神经网络相比较具有较快的训练速度和较好的泛化能力．     

A new method for constructing membership functions and fuzzy rulesfrom training examples

To extract knowledge from a set of numerical data and build up a rule-based system is an important research topic in knowledge acquisition and expert systems. In recent years, many fuzzy systems that automatically generate fuzzy rules from numerical data have been proposed. In this paper, we propose a new fuzzy learning algorithm based on the alpha-cuts of equivalence relations and the alpha-cuts of fuzzy sets to construct the membership functions of the input variables and the output variables of fuzzy rules and to induce the fuzzy rules from the numerical training data set. Based on the proposed fuzzy learning algorithm, we also implemented a program on a Pentium PC using the MATLAB development tool to deal with the Iris data classification problem. The experimental results show that the proposed fuzzy learning algorithm has a higher average classification ratio and can generate fewer rules than the existing algorithm.     

A neural fuzzy system with fuzzy supervised learning

A neural fuzzy system learning with fuzzy training data (fuzzy if-then rules) is proposed in this paper. This system is able to process and learn numerical information as well as linguistic information. At first, we propose a five-layered neural network for the connectionist realization of a fuzzy inference system. The connectionist structure can house fuzzy logic rules and membership functions for fuzzy inference. We use alpha-level sets of fuzzy numbers to represent linguistic information. The inputs, outputs, and weights of the proposed network can be fuzzy numbers of any shape. Furthermore, they can be hybrid of fuzzy numbers and numerical numbers through the use of fuzzy singletons. Based on interval arithmetics, a fuzzy supervised learning algorithm is developed for the proposed system. It extends the normal supervised learning techniques to the learning problems where only linguistic teaching signals are available. The fuzzy supervised learning scheme can train the proposed system with desired fuzzy input-output pairs which are fuzzy numbers instead of the normal numerical values. With fuzzy supervised learning, the proposed system can be used for rule base concentration to reduce the number of rules in a fuzzy rule base. Simulation results are presented to illustrate the performance and applicability of the proposed system.     

Neuro-fuzzy learning with symbolic and numeric data

In real world datasets we often have to deal with different kinds of variables. The data can be, for example, symbolic or numeric. Data analysis methods can often deal with only one kind of data. Even when fuzzy systems are applied – which are not dependent on the scales of variables – usually only numeric data is considered. In this paper we present learning algorithms for creating fuzzy rules and training membership functions from data with symbolic and numeric variables. The algorithms are exentions to our neuro-fuzzy classification approach NEFCLASS. We also demonstrate the applicability of the algorithms on two real-world datasets.     

A characteristic-point-based fuzzy inference system aimed to minimize the number of fuzzy rules

This paper presents a characteristic-point-based fuzzy inference system (CPFIS) for fuzzy modeling from training data. The aim of the CPFIS is not only satisfactory precision performance, but also to employ as few purely linguistic fuzzy rules as possible by using a minimization-based systematic training method. Characteristic points (CPs) are defined as the few data points among the original training data which, when they are directly mapped to fuzzy rules and thus form the entire rule base, allow the underlying system to be effectively modeled. Three minimization-based algorithms in a sequence are proposed to train the CPFIS: a gradient-projection method, a Gauss-Jordan-elimination-based column elimination, and back-propagation. The CPs are determined by iterative computations of the first two minimization algorithms, after which the resulting fuzzy sets are further fine-tuned by the third algorithm. Experiments conducted on three benchmark problems showed that the CPFIS used one of the smallest number of fuzzy rules among the reported results for other methods. The Gaussian membership functions in both the input and output fuzzy sets and the small number of fuzzy rules make the rule interpretation of the CPFIS much easier than that of other methods, thus enhancing human-computer cooperation in knowledge discovery.     

Accurate and Fast Off and Online Fuzzy ARTMAP-Based Image Classification With Application to Genetic Abnormality Diagnosis

We propose and investigate the fuzzy ARTMAP neural network in off and online classification of fluorescence in situ hybridization image signals enabling clinical diagnosis of numerical genetic abnormalities. We evaluate the classification task (detecting a several abnormalities separately or simultaneously), classifier paradigm (monolithic or hierarchical), ordering strategy for the training patterns (averaging or voting), training mode (for one epoch, with validation or until completion) and model sensitivity to parameters. We find the fuzzy ARTMAP accurate in accomplishing both tasks requiring only very few training epochs. Also, selecting a training ordering by voting is more precise than if averaging over orderings. If trained for only one epoch, the fuzzy ARTMAP provides fast, yet stable and accurate learning as well as insensitivity to model complexity. Early stop of training using a validation set reduces the fuzzy ARTMAP complexity as for other machine learning models but cannot improve accuracy beyond that achieved when training is completed. Compared to other machine learning models, the fuzzy ARTMAP does not loose but gain accuracy when overtrained, although increasing its number of categories. Learned incrementally, the fuzzy ARTMAP reaches its ultimate accuracy very fast obtaining most of its data representation capability and accuracy by using only a few examples. Finally, the fuzzy ARTMAP accuracy for this domain is comparable with those of the multilayer perceptron and support vector machine and superior to those of the naive Bayesian and linear classifiers.     

Performance evaluation of fuzzy classifier systems formultidimensional pattern classification problems

We examine the performance of a fuzzy genetics-based machine learning method for multidimensional pattern classification problems with continuous attributes. In our method, each fuzzy if-then rule is handled as an individual, and a fitness value is assigned to each rule. Thus, our method can be viewed as a classifier system. In this paper, we first describe fuzzy if-then rules and fuzzy reasoning for pattern classification problems. Then we explain a genetics-based machine learning method that automatically generates fuzzy if-then rules for pattern classification problems from numerical data. Because our method uses linguistic values with fixed membership functions as antecedent fuzzy sets, a linguistic interpretation of each fuzzy if-then rule is easily obtained. The fixed membership functions also lead to a simple implementation of our method as a computer program. The simplicity of implementation and the linguistic interpretation of the generated fuzzy if-then rules are the main characteristic features of our method. The performance of our method is evaluated by computer simulations on some well-known test problems. While our method involves no tuning mechanism of membership functions, it works very well in comparison with other classification methods such as nonfuzzy machine learning techniques and neural networks.