Dynamically adding symbolically meaningful nodes to knowledge-based neural networks

Traditional connectionist theory-refinement systems map the dependencies of a domain-specific rule base into a neural network, and then refine this network using neural learning techniques. Most of these systems, however, lack the ability to refine their network's topology and are thus unable to add new rules to the (reformulated) rule base. Therefore, with domain theories that lack rules, generalization is poor, and training can corrupt the original rules — even those that were initially correct. The paper presents TopGen, an extension to the KBANN algorithm, which heuristically searches for possible expansions to the KBANN network. TopGen does this by dynamically adding hidden nodes to the neural representation of the domain theory, in a manner that is analogous to the adding of rules and conjuncts to the symbolic rule base. Experiments indicate that the method is able to heuristically find effective places to add nodes to the knowledge bases of four real-world problems, as well as an artificial chess domain. The experiments also verify that new nodes must be added in an intelligent manner. The algorithm showed statistically significant improvements over the KBANN algorithm in all five domains.     

Heuristic constraints enforcement for training of and knowledgeextraction from a fuzzy/neural architecture. I. Foundation

Using fuzzy/neural architectures to extract heuristic information from systems has received increasing attention. A number of fuzzy/neural architectures and knowledge extraction methods have been proposed. Knowledge extraction from systems where the existing knowledge limited is a difficult task. One of the reasons is that there is no ideal rulebase, which can be used to validate the extracted rules. In most of the cases, using output error measures to validate extracted rules is not sufficient as extracted knowledge may not make heuristic sense, even if the output error may meet the specified criteria. The paper proposes a novel method for enforcing heuristic constraints on membership functions for rule extraction from a fuzzy/neural architecture. The proposed method not only ensures that the final membership functions conform to a priori heuristic knowledge, but also reduces the domain of search of the training and improves convergence speed. Although the method is described on a specific fuzzy/neural architecture, it is applicable to other realizations, including adaptive or static fuzzy inference systems. The foundations of the proposed method are given in Part I. The techniques for implementation and integration into the training are given in Part II, together with applications     

Extracting Refined Rules from Knowledge-Based Neural Networks

Neural networks, despite their empirically proven abilities, have been little used for the refinement of existing knowledge because this task requires a three-step process. First, knowledge must be inserted into a neural network. Second, the network must be refined. Third, the refined knowledge must be extracted from the network. We have previously described a method for the first step of this process. Standard neural learning techniques can accomplish the second step. In this article, we propose and empirically evaluate a method for the final, and possibly most difficult, step. Our method efficiently extracts symbolic rules from trained neural networks. The four major results of empirical tests of this method are that the extracted rules 1) closely reproduce the accuracy of the network from which they are extracted; 2) are superior to the rules produced by methods that directly refine symbolic rules; 3) are superior to those produced by previous techniques for extracting rules from trained neural networks; and 4) are human comprehensible. Thus, this method demonstrates that neural networks can be used to effectively refine symbolic knowledge. Moreover, the rule-extraction technique developed herein contributes to the understanding of how symbolic and connectionist approaches to artificial intelligence can be profitably integrated.     

ANN-DT: an algorithm for extraction of decision trees fromartificial neural networks

Although artificial neural networks can represent a variety of complex systems with a high degree of accuracy, these connectionist models are difficult to interpret. This significantly limits the applicability of neural networks in practice, especially where a premium is placed on the comprehensibility or reliability of systems. A novel artificial neural-network decision tree algorithm (ANN-DT) is therefore proposed, which extracts binary decision trees from a trained neural network. The ANN-DT algorithm uses the neural network to generate outputs for samples interpolated from the training data set. In contrast to existing techniques, ANN-DT can extract rules from feedforward neural networks with continuous outputs. These rules are extracted from the neural network without making assumptions about the internal structure of the neural network or the features of the data. A novel attribute selection criterion based on a significance analysis of the variables on the neural-network output is examined. It is shown to have significant benefits in certain cases when compared with the standard criteria of minimum weighted variance over the branches. In three case studies the ANN-DT algorithm compared favorably with CART, a standard decision tree algorithm.     

Reverse Engineering the Neural Networks for Rule Extraction in Classification Problems

Artificial neural networks often achieve high classification accuracy rates, but they are considered as black boxes due to their lack of explanation capability. This paper proposes the new rule extraction algorithm RxREN to overcome this drawback. In pedagogical approach the proposed algorithm extracts the rules from trained neural networks for datasets with mixed mode attributes. The algorithm relies on reverse engineering technique to prune the insignificant input neurons and to discover the technological principles of each significant input neuron of neural network in classification. The novelty of this algorithm lies in the simplicity of the extracted rules and conditions in rule are involving both discrete and continuous mode of attributes. Experimentation using six different real datasets namely iris, wbc, hepatitis, pid, ionosphere and creditg show that the proposed algorithm is quite efficient in extracting smallest set of rules with high classification accuracy than those generated by other neural network rule extraction methods.     

Learning in artificial neural systems

This paper presents an overview and analysis of teaming in artificial neural systems (ANSs). It begins with a general introduction to neural networks and connectionist approaches to information processing. The basis for learning in ANSs is then described and compared with classical machine learning. While similar in some ways, ANS learning deviates from tradition in its dependence on the modification of individual weights to bring about changes in a knowledge representation distributed across connections in a network. This unique form of learning is analyzed from two aspects: the selection of an appropriate network architecture for representing the problem, and the choice of a suitable learning rule capable of reproducing the desired function within the given network. The various network architectures are classified, and then identified with explicit restrictions on the types of functions they are capable of representing. The learning rules, i.e., algorithms that specify how the network weights are modified, are similarly taxonomized and, where possible, the limitations inherent to specific classes of rules are outlined.     

Combining Symbolic and Neural Learning

Conclusion Connectionist machine learning has proven to be a fruitful approach, and it makes sense to investigate systems that combine the strengths of the symbolic and connectionist approaches to AI. Over the past few years, researchers have successfully developed a number of such systems. This article summarizes one view of this endeavor, a framework that encompasses the approaches of several different research groups. This framework (see Figure 1) views the combination of symbolic and neural learning as a three-stage process: (1) the insertion of symbolic information into a neural network, thereby (partially) determining the topology and initial weight settings of a network, (2) the refinement of this network using a numeric optimization method such as backpropagation, possibly under the guidance of symbolic knowledge, and (3) the extraction of symbolic rules that accurately represent the knowledge contained in a trained network. These three components form an appealing, complete picture—approximately-correct symbolic information in, more-accurate symbolic information out—however, these three stages can be independently studied. In conclusion, the research summarized in this paper demonstrates that combining symbolic and connectionist methods is a promising approach to machine learning.     

Characterizing network traffic by means of the NetMine framework

The NetMine framework allows the characterization of traffic data by means of data mining techniques. NetMine performs generalized association rule extraction to profile communications, detect anomalies, and identify recurrent patterns. Association rule extraction is a widely used exploratory technique to discover hidden correlations among data. However, it is usually driven by frequency constraints on the extracted correlations. Hence, it entails (i) generating a huge number of rules which are difficult to analyze, or (ii) pruning rare itemsets even if their hidden knowledge might be relevant. To overcome these issues NetMine exploits a novel algorithm to efficiently extract generalized association rules, which provide a high level abstraction of the network traffic and allows the discovery of unexpected and more interesting traffic rules. The proposed technique exploits (user provided) taxonomies to drive the pruning phase of the extraction process. Extracted correlations are automatically aggregated in more general association rules according to a frequency threshold. Eventually, extracted rules are classified into groups according to their semantic meaning, thus allowing a domain expert to focus on the most relevant patterns. Experiments performed on different network dumps showed the efficiency and effectiveness of the NetMine framework to characterize traffic data.     

Cascade ARTMAP: integrating neural computation and symbolic knowledge processing

This paper introduces a hybrid system termed cascade adaptive resonance theory mapping (ARTMAP) that incorporates symbolic knowledge into neural-network learning and recognition. Cascade ARTMAP, a generalization of fuzzy ARTMAP, represents intermediate attributes and rule cascades of rule-based knowledge explicitly and performs multistep inferencing. A rule insertion algorithm translates if-then symbolic rules into cascade ARTMAP architecture. Besides that initializing networks with prior knowledge can improve predictive accuracy and learning efficiency, the inserted symbolic knowledge can be refined and enhanced by the cascade ARTMAP learning algorithm. By preserving symbolic rule form during learning, the rules extracted from cascade ARTMAP can be compared directly with the originally inserted rules. Simulations on an animal identification problem indicate that a priori symbolic knowledge always improves system performance, especially with a small training set. Benchmark study on a DNA promoter recognition problem shows that with the added advantage of fast learning, cascade ARTMAP rule insertion and refinement algorithms produce performance superior to those of other machine learning systems and an alternative hybrid system known as knowledge-based artificial neural network (KBANN). Also, the rules extracted from cascade ARTMAP are more accurate and much cleaner than the NofM rules extracted from KBANN.     

On connectionism, rule extraction, and brain-like learning

There is a growing body of work that shows that both fuzzy and symbolic rule systems can be implemented using neural networks. This body of work also shows that these fuzzy and symbolic rules can be retrieved from these networks, once they have been learned by procedures that generally fall under the category of rule extraction. The paper argues that the idea of rule extraction from a neural network involves certain procedures, specifically the reading of parameters from a network, that are not allowed by the connectionist framework that these neural networks are based on. It argues that such rule extraction procedures imply a greater freedom and latitude about the internal mechanisms of the brain than is permitted by connectionism, but that such latitude is permitted by the recently proposed control theoretic paradigm for the brain. The control theoretic paradigm basically suggests that there are parts of the brain that control other parts and has far less restrictions on the kind of procedures that can be called “brain like”. The paper shows that this control theoretic paradigm is supported by new evidence from neuroscience about the role of neuromodulators and neurotransmitters in the brain. In addition, it shows that the control theoretic paradigm is also used in connectionist algorithms, although never acknowledged explicitly. The paper suggests that far better learning and rule extraction algorithms can be developed using these control theoretic notions and they would be consistent with the more recent understanding of how the brain works and learns     

A novel fuzzy logic system based on N-version programming

For the consideration of different application systems, modeling the fuzzy logic rule, and deciding the shape of membership functions are very critical issues due to they play key roles in the design of fuzzy logic control system. This paper proposes a novel design methodology of fuzzy logic control system using the neural network and fault-tolerant approaches. The connectionist architecture with the learning capability of neural network and N-version programming development of a fault-tolerant technique are implemented in the proposed fuzzy logic control system. In other words, this research involves the modeling of parameterized membership functions and the partition of fuzzy linguistic variables using neural networks trained by the unsupervised learning algorithms. Based on the self-organizing algorithm, the membership function and partition of fuzzy class are not only derived automatically, but also the preconditions of fuzzy IF-THEN rules are organized. We also provide two examples, pattern recognition and tendency prediction, to demonstrate that the proposed system has a higher computational performance and its parallel architecture supports noise-tolerant capability. This generalized scheme is very satisfactory for pattern recognition and tendency prediction problems     

Combining models from neural networks and inductive learning algorithms

The knowledge-based artificial neural network (KBANN) is composed of phases involving the expression of domain knowledge, the abstraction of domain knowledge at neural networks, the training of neural networks, and finally, the extraction of rules from trained neural networks. The KBANN attempts to open up the neural network black box and generates symbolic rules with (approximately) the same predictive power as the neural network itself. An advantage of using KBANN is that the neural network considers the contribution of the inputs towards classification as a group, while rule-based algorithms like C5.0 measure the individual contribution of the inputs one at a time as the tree is grown. The knowledge consolidation model (KCM) combines the rules extracted using KBANN (NeuroRule), frequency matrix (which is similar to the Naïve Bayesian technique), and C5.0 algorithm. The KCM can effectively integrate multiple rule sets into one centralized knowledge base. The cumulative rules from other single models can improve overall performance as it can reduce error-term and increase R-square. The key idea in the KCM is to combine a number of classifiers such that the resulting combined system achieves higher classification accuracy and efficiency than the original single classifiers. The aim of KCM is to design a composite system that outperforms any individual classifier by pooling together the decisions of all classifiers. Another advantage of KCM is that it does not need the memory space to store the dataset as only extracted knowledge is necessary in build this integrated model. It can also reduce the costs from storage allocation, memory, and time schedule. In order to verify the feasibility and effectiveness of KCM, personal credit rating dataset provided by a local bank in Seoul, Republic of Korea is used in this study. The results from the tests show that the performance of KCM is superior to that of the other single models such as multiple discriminant analysis, logistic regression, frequency matrix, neural networks, decision trees, and NeuroRule. Moreover, our model is superior to a previous algorithm for the extraction of rules from general neural networks.     

基于神经网络与领域知识的外交国际合作元素抽取

为了能够实时了解国际双边合作中有价值的信息,高效地智能提取Web外交新闻中的国际合作元素就显得至关重要。将国际合作元素抽取抽象为类似命名实体识别的问题,首先,界定国际合作元素的内涵;其次,提取了蕴涵领域知识的规则;再次,结合神经网络与领域知识提出了面向外交新闻文本的国际合作元素抽取方法;最后在相同语料库中与神经网络方法以及自身规则组合进行了比较,实验结果表明该方法具有更好的效果。     

Using single-layered neural networks for the extraction of conjunctive rules and hierarchical classifications

Machine Learning is an area concerned with the automation of the process of knowledge acquisition. Neural networks generally represent their knowledge at the lower level, while knowledge based systems use higher level knowledge representations. The method we propose here, provides a technique which automatically allows us to extract production rules from the lower level representation used by a single-layered neural networks trained by Hebb's rule. Even though a single-layered neural network can not model complex, nonlinear domains, their strength in dealing with noise has enabled us to produce correct rules in a noisy domain.     

一种基于统计的神经网络规则抽取方法

从功能性观点出发,提出了一种基于统计的神经网络规则抽取方法.该方法利用统计技术对抽取出的规则进行评价,使其可以较好地覆盖示例空间.采用独特的连续属性处理方式,降低了离散化处理的主观性和复杂度.采用优先级规则形式,不仅使得规则表示简洁、紧凑,而且还免除了规则应用时所需要的一致性处理.该方法不依赖于具体的网络结构和训练算法,可以方便地应用于各种分类器型神经网络.实验表明,利用该方法可以抽取出可理解性好,简洁、紧凑,保真度高的符号规则.     

Application of a Neural Fuzzy System with Rule Extraction to Fault Detection and Diagnosis

In this paper, the fuzzy min–max (FMM) neural network is integrated with a rule extraction algorithm, and the resulting network is applied to a real-world fault detection and diagnosis task in complex industrial processes. With the rule extraction capability, the FMM network is able to overcome the “black-box” phenomenon by justifying its predictions using fuzzy if–then rules that are comprehensible to domain users. To assess the effectiveness of the FMM network, real sensor measurements are collected and used for detecting and diagnosing the heat transfer and tube blockage conditions of a circulating water (CW) system in a power generation plant. The FMM network parameters are systematically varied and tested, with the results explained. Bootstrapping is employed to quantify stability of the network performance statistically. The extracted rules are found to be compatible with the domain information as well as the opinions of domain experts who are involved in the maintenance of the CW system. Implications of the FMM network with the rule extraction facility as an intelligent and useful fault detection and diagnosis tool are discussed.     

Generic form feature recognition and operation selection using connectionist modelling

For the purpose of enhancing the adaptability of computer-aided process planning systems, two connectionist modelling methods, namely neocognitron (i.e. neural network modelling for pattern recognition) and fuzzy associative memories (FAM), are applied to the phases of feature recognition and operation selection respectively in order to provide the system with the ability of self-learning and the ability to integrate traditional expert planning systems with connectionism-based models. In this paper, the attributed adjacency graph (AAG) extracted from a (B-Rep) solid model is converted to attributed adjacency matrices (AAM) that can be used as input data for the neocognitron to train and recognize feature patterns. With this technique, the system can not only self-reconstruct its recognition abilities for new features by learning without a priori knowledge but can also recognize and decompose intersection features. A fuzzy connectionist model, which is created using the Hebbian fuzzy learning algorithm, is employed subsequently to map the features to the appropriate operations. As the algorithm is capable of learning from rules, it is much easier to integrate the proposed model with conventional expert CAPP systems so that they become more generic in dealing with uncertain information processing and perform knowledge updating. mg]Keywords mw]Computer-aided process planning mw]feature recognition mw]neural networks mw]fuzzy neural networks mw]operation selection mw]connectionist model mw]fuzzy associative memories     

Fuzzy neural networks for estimation and fuzzy controller design: simulation study for a pulp batch digester

A structural implementation of a fuzzy inference system through connectionist network based on MLP with logical neurons connected through binary and numerical weights is considered. The resulting fuzzy neural network is trained using classical backpropagation to learn the rules of inference of a fuzzy system, by adjustment of the numerical weights. For controller design, training is carried out off line in a closed loop simulation. Rules for the fuzzy logic controller are extracted from the network by interpreting the consequence weights as measure of confidence of the underlying rule. The framework is used in a simulation study for estimation and control of a pulp batch digester. The controlled variable, the Kappa number, a measure of lignin content in the pulp, which is not measurable is estimated through temperature and liquor concentration using the fuzzy neural network. On the other hand a fuzzy neural network is trained to control the Kappa number and rules are extracted from the trained network to construct a fuzzy logic controller.     

A hybrid expert system for investment advising

Abstract: Investor is an investment advising hybrid expert system which incorporates two of the most powerful techniques developed in artificial intelligence, the connectionist and the logic programming paradigms, in a unified model. Investor employs a neural network model to detect the interest rate trends and a rule-based component to classify the user into the appropriate investor model. The investor model and the interest rate information are then used by the knowledge base to infer the appropriate investment strategy. Genetic algorithms and classifier systems were employed to train and configure the architecture of Investor's neural network component.     

SYMCON—A Hybrid Symbolic/Connectionist System for Word Sense Disambiguation

Connectionist methods and knowledge-based techniques are two largely complementary approaches to natural language processing (NLP). However, they both have some potential problems which preclude their being a general purpose processing method. Research reveals that a hybrid processing approach that combines connectionist with symbolic techniques may be able to use the strengths of one processing paradigm to address the weakness of the other one. Hence, a system that effectively combines the two different approaches can be superior to either one in isolation. This paper describes a hybrid system—SYMCON (SYMbolic and CONnectionist) which integrates symbolic and connectionist techniques in an attempt to solve the problem of word sense disambiguation (WSD), which is arguably one of the most fundamental and difficult issues in NLP. It consists of three sub-systems: first, a distributed simple recurrent network (SRN) is trained by using the standard back-propagation algorithm to learn the semantic relationships among concepts, thereby generating categorical constraints that are supplied to the other two sub-systems as the initial results of pre-processing. The second sub-system of SYMCON is a knowledge-based symbolic component consisting of a knowledge base containing general inferencing rules in a certain application domain. Third, a localist network is used to select the best interpretation among multiple alternatives and potentially ambiguous inference paths by spreading activation throughout the network. The structure, initial states, and connection weights of the network are determined by the processing outcome in the other two sub-systems. This localist network can be viewed as a medium between the distributed network and the symbolic sub-system. Such a hybrid symbolic/connectionist system combines information from all three sources to select the most plausible interpretation for ambiguous words.