Using single-layered neural networks for the extraction of conjunctive rules and hierarchical classifications

Machine Learning is an area concerned with the automation of the process of knowledge acquisition. Neural networks generally represent their knowledge at the lower level, while knowledge based systems use higher level knowledge representations. The method we propose here, provides a technique which automatically allows us to extract production rules from the lower level representation used by a single-layered neural networks trained by Hebb's rule. Even though a single-layered neural network can not model complex, nonlinear domains, their strength in dealing with noise has enabled us to produce correct rules in a noisy domain.     

Identification of nonlinear dynamical systems using multilayered neural networks

A novel multilayer discrete-time neural net paradigm is presented for the identification of multi-input multi-output (MIMO) nonlinear dynamical systems. The major novelty of this approach is a rigorous proof of identification error convergence that reveals a requirement for a new identifier structure and nonstandard weight tuning algorithms. The NN identifier includes modified delta rule weight tuning and exhibits a learning-while-functioning feature instead of learning-then-functioning, so that the identification is on-line with no explicit off-line learning phase needed. The structure of the neural net (NN) identifier is derived using a passivity aproach. Linearity in the parameters is not required and certainty equivalence is not used. The notion of persistency of excitation (PE) and passivity properties of the multilayer NN are defined and used in the convergence analysis of both the identification error and the weight estimates.     

An acquisition of operator's rules for collision avoidance usingfuzzy neural networks

The procedure for acquiring control rules to improve the performance of control systems has received considerable attention previously. This paper deals with a collision avoidance problem in which the controlled object is a ship with inertia which must avoid collision with a moving object. It has proven to be difficult to obtain collision avoidance rules, i.e., steering rules and speed control rules, which coincide with the operator's knowledge. This paper shows that rules of this type can be acquired directly from observational data using fuzzy neural networks (FNNs). This paper also shows that the FNN can obtain portions of the fuzzy rules for the inferences of the static and dynamic degrees of danger and the decision table based on the degrees of danger to avoid the moving obstacle     

Extracting rules from trained neural networks

Presents an algorithm for extracting rules from trained neural networks. The algorithm is a decompositional approach which can be applied to any neural network whose output function is monotone such as a sigmoid function. Therefore, the algorithm can be applied to multilayer neural networks, recurrent neural networks and so on. It does not depend on training algorithms, and its computational complexity is polynomial. The basic idea is that the units of neural networks are approximated by Boolean functions. But the computational complexity of the approximation is exponential, and so a polynomial algorithm is presented. The author has applied the algorithm to several problems to extract understandable and accurate rules. The paper shows the results for the votes data, mushroom data, and others. The algorithm is extended to the continuous domain, where extracted rules are continuous Boolean functions. Roughly speaking, the representation by continuous Boolean functions means the representation using conjunction, disjunction, direct proportion, and reverse proportion. This paper shows the results for iris data.     

Knowledge acquisition using hypertext

This paper addresses the issue of supporting knowledge acquisition using hypertext. We propose a way of tightly integrating hypertext and structured object representation, using Artificial Intelligence (AI) frames for the basic representation of hypertext nodes. Epistemologically, a dual view of the resulting space is of interest. One view is that of hypertext which emphasizes nodes containg g text, including formal knowledge representation. The other view focuses on objects with certain relationships, which define a semantic network. Both in hypertext and in semantic networks the relations between chunks of knowledge are explicitly represented by links. However, in today's hypertext systems a node typically contains just informal text and references to other nodes. Our approach additionally facilitates the explicit representation of structure “inside” hypertext nodes using partitions. We show the usefulness of such a tight integration for knowledge acquisition, providing several features useful for supporting it based on a level of basic hypertext functionality. In particular, we sketch a method for doing knowledge acquisition in such an environment. Hypertext is used as a mediating “semiformal” representation, which allows experts to directly represent knowledge without the immediate support of knowledge engineers. These help then to make this knowledge operational, supported by the system's facility to provide templates as well as their links to the semiformal representation. As an example of our results of using this method of knowledge acquisition, we illustrate the strategic knowledge in our application domain. More generally, our approach supports important aspects of (software) engineering knowledge-based systems and their maintenance. Also their user interface can be improved this way.     

Constructing neural networks from expert system rules

The implementation of a PC-based expert system rules-to-neural network translator is described. Knowledge expressed as rules is translated to a neural network representation. The generated structure simulates a neural network which is able to perform as the original expert system—conclusions are drawn from user-supplied facts based on the inherent knowledge.     

Extracting M-of-N rules from trained neural networks

An effective algorithm for extracting M-of-N rules from trained feedforward neural networks is proposed. First, we train a network where each input of the data can only have one of the two possible values, -1 or one. Next, we apply the hyperbolic tangent function to each connection from the input layer to the hidden layer of the network. By applying this squashing function, the activation values at the hidden units are effectively computed as the hyperbolic tangent (or the sigmoid) of the weighted inputs, where the weights have magnitudes that are equal one. By restricting the inputs and the weights to binary values either -1 or one, the extraction of M-of-N rules from the networks becomes trivial. We demonstrate the effectiveness of the proposed algorithm on several widely tested datasets. For datasets consisting of thousands of patterns with many attributes, the rules extracted by the algorithm are simple and accurate.     

Knowledge acquisition using a neural network for a weather forecasting knowledge-based system

Neural network technology is experiencing rapid growth and is receiving considerable attention from almost every field of science and engineering. The attraction is due to the successful application of neural network techniques to several real world problems. Neural networks have not yet found widespread application in weather forecasting. The reason for this has been the difficulty in obtaining suitable weather forecasting data sets. In this paper we describe our experience in applying neural network techniques for acquiring the necessary knowledge to predict the weather conditions of Melbourne City and its suburbs in Australia during a 24 hour period beginning at 9 am local time. The accuracy of forecasts produced by a given forecasting procedure typically varies with factors such as geographical location, season, categories of weather, quality of input data, lead time and validity time. Two types of weather data sets assembled from the archives of the Australian Commonwealth Bureau of Meteorology are used for training the neural network. The results of the experiments are competitive and are discussed.     

On the emergence of rules in neural networks

A simple associationist neural network learns to factor abstract rules (i.e., grammars) from sequences of arbitrary input symbols by inventing abstract representations that accommodate unseen symbol sets as well as unseen but similar grammars. The neural network is shown to have the ability to transfer grammatical knowledge to both new symbol vocabularies and new grammars. Analysis of the state-space shows that the network learns generalized abstract structures of the input and is not simply memorizing the input strings. These representations are context sensitive, hierarchical, and based on the state variable of the finite-state machines that the neural network has learned. Generalization to new symbol sets or grammars arises from the spatial nature of the internal representations used by the network, allowing new symbol sets to be encoded close to symbol sets that have already been learned in the hidden unit space of the network. The results are counter to the arguments that learning algorithms based on weight adaptation after each exemplar presentation (such as the long term potentiation found in the mammalian nervous system) cannot in principle extract symbolic knowledge from positive examples as prescribed by prevailing human linguistic theory and evolutionary psychology.     

Perception control with improved expectation learning through multilayered neural networks

In this paper, we investigate the viability of multilayered neural network (NN)-based extension of a conventional "perception" control concept. The perception process selects and completes the information from the system to be controlled before passing it to the controlling agent so that control is not lost when sensory information from the system is incomplete. The perception process produces an expectation of the next set of information to be received from the system. The expectation is used to replace missing parts of the information received and it also influences the next perception. In the existing work, each of the expectation elements is linearly acquired such that the expectation tells only the dominant information in the recent past, i.e., this approach has no capability to sense the trend and the dynamics in the information. This handicap could become a serious problem when the perception process is applied to real physical systems. Here, we introduce an extension of the perception control process by using a radial basis function (RBF) feedforward NN to learn the trend and the dynamics in the information and produce the expectation of the next observation. Through some simulation comparisons, we show that the proposed RBFNN-based method is better than the existing one.     

An improved synthesis method for multilayered neural networks usingqualitative knowledge

An improved synthesis method for the multilayered neural network (NN) as function approximator is proposed. The method offers a translation mechanism that maps the qualitative knowledge into a multilayered NN structure. Qualitative knowledge is expressed in the form of representative points, which can be linguistically described as, `when x is around x_i, then y_i is around y'. Synthesis equations for the translation mechanism are provided. After the direct synthesis of the initial NN, the NN is tuned by backpropagation (BP), using the training data. The direct synthesis decreases the burden on BP and contributes to improved learning efficiency, accuracy, and stability. It is demonstrated that the translation mechanism is also useful for incremental modeling, i.e., increasing the number of neurons, or representative points, based on the results of BP     

Knowledge discovery by inductive neural networks

A new neural network model for inducing symbolic knowledge from empirical data is presented. This model capitalizes on the fact that the certainty factor-based activation function can improve the network generalization performance from a limited amount of training data. The formal properties of the procedure for extracting symbolic knowledge from such a trained neural network are investigated. In the domain of molecular genetics, a case study demonstrated that the described learning system effectively discovered the prior domain knowledge with some degree of refinement. Also, in cross-validation experiments, the system outperformed C4.5, a commonly used rule learning system     

Fuzzy neural networks for learning fuzzy if-then rules

This study is dedicated to developing a fuzzy neural network with linguistic teaching signals. The proposed network, which can be applied either as a fuzzy expert system or a fuzzy controller, is able to process and learn the numerical information as well as the linguistic information. The network consists of two parts: (1) initial weights generation and (2) error back-propagation (EBP)-type learning algorithm. In the first part, a genetic algorithm (GA) generates the initial weights for a fuzzy neural network in order to prevent the network getting stuck to the local minimum. The second part employs the EBP-type learning algorithm for fine-tuning. In addition, the unimportant weights are eliminated during the training process. The simulated results do not only indicate that the proposed network can accurately learn the relations of fuzzy inputs and fuzzy outputs, but also show that the initial weights from the GA can coverage better and weight elimination really can reduce the training error. Moreover, real-world problem results show that the proposed network is able to learn the fuzzy IF-THEN rules captured from the retailing experts regarding the promotion effect on the sales.     

Extraction of rules from artificial neural networks for nonlinearregression

Neural networks (NNs) have been successfully applied to solve a variety of application problems including classification and function approximation. They are especially useful as function approximators because they do not require prior knowledge of the input data distribution and they have been shown to be universal approximators. In many applications, it is desirable to extract knowledge that can explain how Me problems are solved by the networks. Most existing approaches have focused on extracting symbolic rules for classification. Few methods have been devised to extract rules from trained NNs for regression. This article presents an approach for extracting rules from trained NNs for regression. Each rule in the extracted rule set corresponds to a subregion of the input space and a linear function involving the relevant input attributes of the data approximates the network output for all data samples in this subregion. Extensive experimental results on 32 benchmark data sets demonstrate the effectiveness of the proposed approach in generating accurate regression rules.     

An improved SPSA algorithm for system identification using fuzzy rules for training neural networks

Simultaneous perturbation stochastic approximation (SPSA) belongs to the class of gradient-free optimization methods that extract gradient information from successive objective function evaluation. This paper describes an improved SPSA algorithm, which entails fuzzy adaptive gain sequences, gradient smoothing, and a step rejection procedure to enhance convergence and stability. The proposed fuzzy adaptive simultaneous perturbation approximation (FASPA) algorithm is particularly well suited to problems involving a large number of parameters such as those encountered in nonlinear system identification using neural networks (NNs). Accordingly, a multilayer perceptron (MLP) network with popular training algorithms was used to predicate the system response. We found that an MLP trained by FASPSA had the desired accuracy that was comparable to results obtained by traditional system identification algorithms. Simulation results for typical nonlinear systems demonstrate that the proposed NN architecture trained with FASPSA yields improved system identification as measured by reduced time of convergence and a smaller identification error.     

Two regularizers for recursive least squared algorithms infeedforward multilayered neural networks

Recursive least squares (RLS)-based algorithms are a class of fast online training algorithms for feedforward multilayered neural networks (FMNNs). Though the standard RLS algorithm has an implicit weight decay term in its energy function, the weight decay effect decreases linearly as the number of learning epochs increases, thus rendering a diminishing weight decay effect as training progresses. In this paper, we derive two modified RLS algorithms to tackle this problem. In the first algorithm, namely, the true weight decay RLS (TWDRLS) algorithm, we consider a modified energy function whereby the weight decay effect remains constant, irrespective of the number of learning epochs. The second version, the input perturbation RLS (IPRLS) algorithm, is derived by requiring robustness in its prediction performance to input perturbations. Simulation results show that both algorithms improve the generalization capability of the trained network.     

Real-time acquisition of fuzzy rules using genetic algorithms

The paper presents a Genetic Algorithm(GA)-based system for online acquisition and modification of rules for a fuzzy logic controller. This uses a version of the rule competition production systems, called Classifier Systems, but in which the rules are matched in the fuzzy domain rather than as binary patterns. The GA is operated in the incremental mode whereby only one structure from a population is evaluated in each time interval. To hasten the learning process, the payoff received is used to assign estimates of new strengths to the other classifiers, dependent on the degree of matching with the evaluated classifier. The rule learning is initialized with randomly generated structures to which fairly general heuristic knowledge has been added. The interacting environment has been modelled by a real time simulation of closed loop administration of an anaesthetic drug, but the characteristics of the environment are not known to the GA.     

Knowledge transfer in SVM and neural networks

The paper considers general machine learning models, where knowledge transfer is positioned as the main method to improve their convergence properties. Previous research was focused on mechanisms of knowledge transfer in the context of SVM framework; the paper shows that this mechanism is applicable to neural network framework as well. The paper describes several general approaches for knowledge transfer in both SVM and ANN frameworks and illustrates algorithmic implementations and performance of one of these approaches for several synthetic examples.